Changes

Jump to navigation Jump to search

Infrastructure (view source)

Revision as of 14:59, 23 February 2023

14,598 bytes added ,  14:59, 23 February 2023
no edit summary
Line 1: Line 1: −
= Infrastructure =
+
[[Category:ProductOpener]]
 +
[[Category:Developer]]
 +
[[Category:Roadmap]]
 +
[[Category:Infrastructure]]
    
This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects.
 
This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects.
   −
== High level infrastructure design ==
+
== Infrastructure planning ==
   −
1 dedicated OVH server running:
+
* Hardware needs: [[Infrastructure Planning]]
−
* 1 nginx reverse proxy to serve static files (installed with apt-get)
+
 
−
* 1 apache 2.4 + mod_perl (installed with apt-get)
+
== 2021 roadmap ==
−
** running two separate instances on different ports, 1 for OFF and 1 for OBF
+
* Please expand desirable objectives (not hardware needs)
−
* 1 MongoDB
+
*
 +
* …
 +
 
 +
== Current infrastructure ==
 +
 
 +
=== off1 ===
 +
Dedicated free.org server:
 +
 
 +
'''Hardware configuration:'''
 +
* Dell 1U R440 server (1 Xeon Silver 4110: 8c/16t)
 +
* RAM: 64GB (DDR4@2666)
 +
* Disks:
 +
** 2x4 TB SATA HDD (hardware RAID1) : /srv
 +
** 2x14 TB SATA HDD (hardware RAID1) : /srv2 (images, imports)
 +
'''Running services:'''
 +
* OS: debian 9.5
 +
* 1 nginx reverse proxy to serve static files
 +
* 1 apache + mod_perl to generate dynamic pages + API
 +
** running 4 separate instances on different ports: OFF, OBF, OPFF, OPF
 +
 
 +
=== off2 ===
 +
Dedicated free.org server:
 +
 
 +
'''Hardware configuration:'''
 +
* Dell 1U R440 server (1 Xeon Silver 4110: 8c/16t)
 +
* RAM: 64GB (DDR4@2666)
 +
* Disks:
 +
** 2x4 TB SATA HDD (hardware RAID1) : /srv
 +
** 2x14 TB SATA HDD (hardware RAID1) : /srv2 (images, imports)
 +
'''Running services:'''
 +
* OS: debian 9.5
 +
* 1 apache + mod_perl (not used)
 +
* 1 MongoDB (production)
 +
* Robotoff (dockerized)
 +
 
 +
=== no name yet ===
 +
Dedicated OVH - soyoustart server (rented):
 +
 
 +
'''Hardware configuration:'''
 +
* E3-SAT-1-32
 +
* Storage: SoftRaid 3x2To SATA
 +
* RAM: 32GB DDR3 1333 MHz
 +
* CPU: Xeon E3-1245v2 (4c/8th)
 +
'''Running services:'''
 +
* wiki
 +
* zammad (soon)
 +
 
 +
=== dev ===
 +
Dedicated OVH - soyoustart server (rented):
 +
 
 +
'''Hardware configuration:'''
 +
* Super Micro server (1 Xeon D-1520)
 +
* Storage: 4 x 2TB HDD, software RAID1 / /boot + LVM /srv /srv2
 +
* RAM: 32GB (DDR4@2666)
 +
* CPU: Xeon D-1520
 +
'''Running services:'''
 +
* OS: debian 9.8
 +
* 1 nginx reverse proxy
 +
* 1 apache + mod_perl (https://openfoodfacts.dev - auth: off / off)
 +
* 1 mongodb
 +
 
 +
=== dev2 ===
 +
Dedicated server (self-hosted by [[User:Cquest|cquest]])
 +
 
 +
'''Hardware configuration:'''
 +
* Dell R710 server (2U)
 +
* Storage (ZFS pool) with:
 +
** 6 x 2TB HITACHI HUS72402 (SAS)
 +
** 500GB Samsung 860 SSD: OS and 256GB read cache (L2ARC)
 +
** 16GB Intel Optane SSD: write journal (ZIL)
 +
* RAM: 96GB (DDR3@1333)
 +
* CPU: 2 Xeon X5675 = 12 cores @ 3.07GHz
 +
'''Running services:'''
 +
* OS: Proxmox 6 (debian 10 based)
 +
 
 +
'''Network''':
 +
* uplink: optic fiber from OVH (1G down/500M up) with dedicated IPv4 and IPv6/72
 +
* ethernet 1Gbps (bonded for backup)
 +
* infiniband 40Gbps (proxmox cluster backbone + access to storage server)
 +
 
 +
=== backup ===
 +
Shared storage server (self-hosted by [[User:Cquest|cquest]])
 +
 
 +
'''Hardware configuration:'''
 +
* Supermicro storage server (4U)
 +
* Storage (ZFS pool) with:
 +
** 36 SAS bays (populated disks evolving with time are needs)
 +
** 1TB Samsung 860 SSD: OS and 256GB read cache (L2ARC)
 +
** 16GB Intel Optane SSD: write journal (ZIL)
 +
* RAM: 128GB
 +
* CPU:
 +
'''Running services:'''
 +
* OS: Proxmox 6 (debian 10 based)
 +
* Local NFS access from dev2 to /srv /srv2 backups (nghtly rsync + ZFS snapshots)
 +
* Proxmox CT/VM replication
 +
 
 +
=== ovh1 ===
 +
VM/CT server for computing usages.
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: AMD Epyc 7451 (24c/48t)
 +
* RAM: 256GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.1) on vRack
 +
* 2 x 1TB SSD NVMe
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on RAID1 /dev/md2
 +
* 512MB swap
 +
* 920GB ZFS pool on remaining space (mirror)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
 +
 
 +
=== ovh2 ===
 +
VM/CT server for computing usages (same as ovh1).
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: AMD Epyc 7451 (24c/48t)
 +
* RAM: 256GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.2) on vRack
 +
* 2 x 1TB SSD NVMe
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on RAID1 /dev/md2
 +
* 512MB swap
 +
* 920GB ZFS pool on remaining space (mirror)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
 +
 
 +
=== ovh3 ===
 +
Mainly storage server, VM/CT host is needed.
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: Intel Xeon-D 1541 (8c/16t)
 +
* RAM: 32GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.3)
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on SSD
 +
* 512MB swap
 +
* 5GB ZFS ZIL (write journal)
 +
* 472GB ZFS cache (ARC2 read cache)
 +
 
 +
* ZFS storage (6 x 12TB HDD) in RAIDZ2 (2 disks for redundancy, around 42TB available)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
 +
 
 +
=== ovh4 ===
 +
Mainly storage server, VM/CT host is needed.
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: Intel Xeon-D 1541 (8c/16t)
 +
* RAM: 32GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.4)
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on SSD
 +
* 512MB swap
 +
* 5GB ZFS ZIL (write journal)
 +
* 472GB ZFS cache (ARC2 read cache)
 +
 
 +
* ZFS storage (6 x 12TB HDD) in RAIDZ2 (2 disks for redundancy, around 42TB available)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
   −
=== History ===
+
== History ==
    
From 2012 to 2016, OFF and OBF have been running on part of an OVH server (also used for other purposes):
 
From 2012 to 2016, OFF and OBF have been running on part of an OVH server (also used for other purposes):
Line 20: Line 197:  
* MongoDB 2.4.12 (installed from mongodb provided packages)
 
* MongoDB 2.4.12 (installed from mongodb provided packages)
    +
From 2016 to 2018, 1 dedicated OVH server running:
 +
* 1 nginx reverse proxy to serve static files (installed with apt-get)
 +
* 1 apache 2.4 + mod_perl (installed with apt-get)
 +
** running two separate instances on different ports, 1 for OFF and 1 for OBF
 +
* 1 MongoDB
   −
 
+
== 2012-2016 server install log ==
−
== New server install log ==
      
OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects.
 
OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects.
Line 30: Line 211:     
* E3-SAT-3 Intel Xeon E3 1245v2 4 c / 8 t 3.4 GHz+ 32 Go 2x 2 To SATA3 Soft 16* 40.00€ HT (SOIT 48.00€ TTC)
 
* E3-SAT-3 Intel Xeon E3 1245v2 4 c / 8 t 3.4 GHz+ 32 Go 2x 2 To SATA3 Soft 16* 40.00€ HT (SOIT 48.00€ TTC)
−
* http://www.soyoustart.com/fr/offres/e3-sat-3.xml
+
* https://www.soyoustart.com/fr/offres/e3-sat-3.xml
    
=== Server setup ===
 
=== Server setup ===
Line 85: Line 266:     
Product Opener needs a domain, with a A record for the domain itself and another wildcard A record for all subdomains.
 
Product Opener needs a domain, with a A record for the domain itself and another wildcard A record for all subdomains.
 +
 +
* openbeautyfacts.org. 0 A 178.33.104.169
 +
* *.openbeautyfacts.org. 0 A 178.33.104.169
    
For testing the new server, we will be using openfoodfacts.eu
 
For testing the new server, we will be using openfoodfacts.eu
Line 153: Line 337:  
# You should look at the following URL's in order to grasp a solid understanding
 
# You should look at the following URL's in order to grasp a solid understanding
 
# of Nginx configuration files in order to fully unleash the power of Nginx.
 
# of Nginx configuration files in order to fully unleash the power of Nginx.
−
# http://wiki.nginx.org/Pitfalls
+
# https://wiki.nginx.org/Pitfalls
−
# http://wiki.nginx.org/QuickStart
+
# https://wiki.nginx.org/QuickStart
−
# http://wiki.nginx.org/Configuration
+
# https://wiki.nginx.org/Configuration
 
#
 
#
 
# Generally, you will want to move this file somewhere, and start with a clean
 
# Generally, you will want to move this file somewhere, and start with a clean
Line 166: Line 350:  
#
 
#
 
server {
 
server {
−
         listen 80 default_server;
+
         #listen 80 default_server;
−
         listen [::]:80 default_server;
+
         #listen [::]:80 default_server;
   −
         server_name openfoodfacts.eu *.openfoodfacts.eu;
+
         server_name openfoodfacts.org *.openfoodfacts.org openfoodfacts.eu *.openfoodfacts.eu;
    
         # SSL configuration
 
         # SSL configuration
Line 182: Line 366:     
         root /home/off/html;
 
         root /home/off/html;
 +
 +
        access_log /home/off/logs/nginx.access2.log;
 +
        error_log /home/off/logs/nginx.error2.log;
 +
 +
        gzip on;
 +
        gzip_min_length 1000;
 +
    
         # Add index.php to the list if you are using PHP
 
         # Add index.php to the list if you are using PHP
 
         index index.html index.htm index.nginx-debian.html;
 
         index index.html index.htm index.nginx-debian.html;
 +
 +
        location ~* \.(eot|ttf|woff|woff2)$ {
 +
                add_header Access-Control-Allow-Origin *;
 +
        }
 +
 +
        location ~ ^/images/products/ {
 +
                add_header Link "<http://creativecommons.org/licenses/by-sa/3.0/>; rel='license'; title='CC-BY-SA 3.0'";
 +
        }
 +
 +
        location ~ ^/(favicon.ico) {
 +
                # First attempt to serve request as file, then
 +
                # as directory, then fall back to displaying a 404.
 +
                try_files $uri $uri/ =404;
 +
        }
 +
    
         location ~ ^/(images|js|rss|data|files|resources|foundation)/ {
 
         location ~ ^/(images|js|rss|data|files|resources|foundation)/ {
Line 199: Line 405:  
                 proxy_set_header Host $host;
 
                 proxy_set_header Host $host;
 
                 proxy_set_header X-Real-IP $remote_addr;
 
                 proxy_set_header X-Real-IP $remote_addr;
 +
                proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
 +
 
                 proxy_pass http://127.0.0.1:8001/cgi/display.pl?;
 
                 proxy_pass http://127.0.0.1:8001/cgi/display.pl?;
 
         }
 
         }
Line 205: Line 413:  
                 proxy_set_header Host $host;
 
                 proxy_set_header Host $host;
 
                 proxy_set_header X-Real-IP $remote_addr;
 
                 proxy_set_header X-Real-IP $remote_addr;
 +
                proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
 
                 proxy_pass http://127.0.0.1:8001;
 
                 proxy_pass http://127.0.0.1:8001;
 
         }
 
         }
Line 215: Line 424:  
         #}
 
         #}
 
}
 
}
 +
 +
 
</pre>
 
</pre>
   Line 227: Line 438:     
===== Apache configuration =====
 
===== Apache configuration =====
 +
 +
Note: imagemagick + mod_perl seems to crash Apache in the worker / event MPM, use prefork.
 +
 +
remove event and then:
 +
 +
/etc/apache2-off/mods-enabled# ln -s ../mods-available/mpm_prefork.conf mpm_prefork.conf
 +
/etc/apache2-off/mods-enabled# ln -s ../mods-available/mpm_prefork.load mpm_prefork.load
    
Set the user to off
 
Set the user to off
Line 245: Line 463:  
# LoadModule perl_module modules/mod_perl.so
 
# LoadModule perl_module modules/mod_perl.so
   −
PerlWarn Off
+
PerlSwitches -I/home/off/lib
−
PerlRequire /home/off/cgi/startup_apache2.pl
+
 
 +
PerlWarn On
 +
PerlRequire /home/off/lib/startup_apache2.pl
    +
 +
# log the X-Forwarded-For IP address (the client ip) in access_log
 +
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy
    
<Location /cgi>
 
<Location /cgi>
Line 260: Line 483:  
<VirtualHost *>
 
<VirtualHost *>
 
DocumentRoot /home/off/html
 
DocumentRoot /home/off/html
−
ServerName openfoodfacts.eu
+
ServerName openfoodfacts.org
 
ErrorLog /home/off/logs/error_log
 
ErrorLog /home/off/logs/error_log
 
CustomLog /home/off/logs/access_log combined
 
CustomLog /home/off/logs/access_log combined
−
LogLevel debug
+
LogLevel warn
 
ScriptAlias /cgi/ "/home/off/cgi/"
 
ScriptAlias /cgi/ "/home/off/cgi/"
   Line 316: Line 539:     
-> forgot to create startup_apache2.pl, going to copy the one from off and change the path in it.
 
-> forgot to create startup_apache2.pl, going to copy the one from off and change the path in it.
 +
 +
More debugging:
 +
 +
If mod_perl starts, it loads some modules from cgi/startup_apache2.pl, and may fail if some modules are missing.
 +
Check /home/obf/logs/error_log:
 +
 +
<pre>
 +
[Mon Jul 04 18:31:11.501044 2016] [perl:error] [pid 23498:tid 116442525869952] Can't locate Math/Random/Secure.pm in @INC (you may need to install the Math::Random::Secure module)
 +
</pre>
    
===== Running a 2nd Apache instance for OBF =====
 
===== Running a 2nd Apache instance for OBF =====
Line 321: Line 553:  
OFF and OBF share the same code (except some configuration in the Config.pm and SiteLang.pm modules), but the code is not made to run two separate servers.
 
OFF and OBF share the same code (except some configuration in the Config.pm and SiteLang.pm modules), but the code is not made to run two separate servers.
 
So we need two run two separate instances of Apache, one for OFF and one for OBF. This also allows to restart one but not the other, to run different versions of the code etc.
 
So we need two run two separate instances of Apache, one for OFF and one for OBF. This also allows to restart one but not the other, to run different versions of the code etc.
 +
 +
Note: we use the off user for both OFF and OBF so that OBF can read and write  in /home/off/users
    
There is some support to run multiple Apache instances. Documentation is in /usr/share/doc/apache2/README.multiple-instances
 
There is some support to run multiple Apache instances. Documentation is in /usr/share/doc/apache2/README.multiple-instances
Line 386: Line 620:     
mkdir /home/obf/logs
 
mkdir /home/obf/logs
 +
 +
 +
===== Forwarding client IPs from nginx to Apache configuration =====
 +
 +
in nginx config:
 +
 +
Apache:
 +
 +
install Apache2::Connection::XForwardedFor
 +
 +
Can't find the mod_perl include dir (reason: path /usr/include/apache2 doesn't exist)
 +
 +
-> just mkdir /usr/include/apache2 and reinstall module
    
==== Product Opener ====
 
==== Product Opener ====
Line 392: Line 639:     
* apt-get install zlib1g-dev
 
* apt-get install zlib1g-dev
 +
 +
====== GeoIP updatate ======
 +
 +
The packaged geoip database installed through "apt-get install geoip-database" is very old, need to install one manually:
 +
 +
<pre>
 +
root@ns3362784:/home/obf/logs# apt-get install geoip-database
 +
Reading package lists... Done
 +
Building dependency tree
 +
Reading state information... Done
 +
geoip-database is already the newest version.
 +
The following packages were automatically installed and are no longer required:
 +
  libasan0 libboost-dev libboost-filesystem1.55.0
 +
  libboost-program-options1.55.0 libboost-system1.55.0 libboost-thread1.55.0
 +
  libboost1.55-dev libgcc-4.8-dev libgoogle-perftools4 libpcrecpp0 libsnappy1
 +
  libstdc++-4.8-dev libtcmalloc-minimal4 libunwind8 libv8-3.14.5
 +
Use 'apt-get autoremove' to remove them.
 +
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
 +
root@ns3362784:/home/obf/logs# cd /usr/share/GeoIP/
 +
root@ns3362784:/usr/share/GeoIP# ls -lrt
 +
total 4900
 +
-rw-r--r-- 1 root root 1061883 Mar 17  2015 GeoIP.dat
 +
-rw-r--r-- 1 root root 3949299 Mar 17  2015 GeoIPv6.dat
 +
root@ns3362784:/usr/share/GeoIP# wget https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
 +
--2016-07-15 00:16:08--  https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
 +
Resolving geolite.maxmind.com (geolite.maxmind.com)... 2400:cb00:2048:1::6810:262f, 2400:cb00:2048:1::6810:252f, 104.16.38.47, ...
 +
Connecting to geolite.maxmind.com (geolite.maxmind.com)|2400:cb00:2048:1::6810:262f|:80... connected.
 +
HTTP request sent, awaiting response... 200 OK
 +
Length: 517602 (505K) [application/octet-stream]
 +
Saving to: ‘GeoIP.dat.gz’
 +
 +
GeoIP.dat.gz        100%[=====================>] 505.47K  --.-KB/s  in 0.06s
 +
 +
2016-07-15 00:16:08 (7.76 MB/s) - ‘GeoIP.dat.gz’ saved [517602/517602]
 +
 +
root@ns3362784:/usr/share/GeoIP# gzip -d GeoIP.dat.gz
 +
gzip: GeoIP.dat already exists; do you wish to overwrite (y or n)? y
 +
root@ns3362784:/usr/share/GeoIP# ls -lrt
 +
total 4744
 +
-rw-r--r-- 1 root root 3949299 Mar 17  2015 GeoIPv6.dat
 +
-rw-r--r-- 1 root root  904073 Jul  6 19:50 GeoIP.dat
 +
</pre>
 +
 +
Make a symbolic link (Perl module seems to think the database is at /usr/local/share/GeoIP/GeoIP.dat )
 +
 +
<pre>
 +
ln -s /usr/share/GeoIP /usr/local/share/GeoIP
 +
</pre>
    
===== Perl modules =====
 
===== Perl modules =====
Line 413: Line 708:  
</pre>
 
</pre>
   −
===== Symbolic links in cgi directory ======
+
===== Symbolic links in lib directory =====
    
ls -lrt |grep -- "->"
 
ls -lrt |grep -- "->"
Line 420: Line 715:     
<pre>
 
<pre>
−
/home/off/cgi# rm Blogs
+
/home/off/lib/ProductOpener# ln -s SiteLang_off.pm SiteLang.pm
−
/home/off/cgi# ln -s /home/off/cgi Blogs
+
/home/off/lib/ProductOpener# ln -s Config_off.pm Config.pm
   −
ln -s SiteLang_off.pm SiteLang.pm
+
/home/off/lib/ProductOpener# cp Config2_sample.pm Config2.pm
 +
-> put right values for server domain, home path, and mongodb database name
   −
vi /home/off/cgi/startup.pl
+
/home/off/scripts# ln -s ../lib/ProductOpener ProductOpener
−
# Needs to be configured
  −
use lib "/home/off/cgi/";
     −
vi /home/cgi/Config2.pm
  −
-> put right values for server domain, home path, and mongodb database name
   
</pre>
 
</pre>
−
      
===== robots.txt =====
 
===== robots.txt =====
Line 448: Line 739:       −
====== Data ======
+
===== Data =====
    
* Copy data to new server
 
* Copy data to new server
Line 456: Line 747:  
** chown -R off deleted* html index ingredients lang invalid lists products users tmp
 
** chown -R off deleted* html index ingredients lang invalid lists products users tmp
   −
====== Product opener debug =====
+
 
 +
===== MongoDB indexes =====
 +
 
 +
Some basic indexes, needs to be revisited.
 +
https://github.com/openfoodfacts/openfoodfacts-server/issues/341
 +
 
 +
<pre>
 +
mongo
 +
use off
 +
 
 +
 
 +
db.products.createIndex( { code : 1 }, { background : true });
 +
db.products.createIndex( { unique_scans_n : -1 }, { background : true });
 +
db.products.createIndex( { sortkey : -1 }, { background : true });
 +
db.products.createIndex( { last_modified_t : -1 }, { background : true });
 +
 
 +
db.products.createIndex( { created_t : 1 }, { background : true });
 +
db.products.createIndex( { lc : 1 }, { background : true });
 +
 
 +
db.products.createIndex( { _keywords : 1, last_modified_t : -1 }, { background : true });
 +
db.products.createIndex( { _keywords : 1, sortkey : -1 }, { background : true });
 +
 
 +
 
 +
db.products.createIndex( { countries_tags : 1 , last_modified_t : -1 }, { background : true });
 +
 
 +
db.products.createIndex( { countries_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { brands_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { categories_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { labels_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { packaging_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { origins_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { manufacturing_places_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { emb_codes_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { cities_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { ingredients_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { ingredients_from_palm_oil_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { vitamins_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { minerals_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { amino_acids_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { nucleotides_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { other_nutritional_substances_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { allergens_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { traces_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { nova_groups_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { nutrition_grades_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { misc_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { languages_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { users_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { editors_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { informers_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { correctors_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { checkers_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { photographers_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { states_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { entry_dates_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { last_edit_dates_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { purchase_places_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { stores_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { codes_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { debug_tags : 1 , sortkey : -1 }, { background : true });
 +
 
 +
 
 +
 
 +
 
 +
</pre>
 +
 
 +
===== Product opener debug =====
    
Once Apache starts:
 
Once Apache starts:
Line 467: Line 824:     
-> created startup_apache2.pl, loaded in off.conf
 
-> created startup_apache2.pl, loaded in off.conf
 +
 +
 +
===== Generate CSS and download dependencies =====
 +
 +
<pre>
 +
/srv/off/yarn install
 +
/srv/off/yarn run build
 +
</pre>
 +
 +
The commands above might fail if the correct versions of yarn and nodejs are not installed. To fix:
 +
 +
<pre>
 +
apt remove cmdtest
 +
apt remove yarn
 +
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
 +
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
 +
apt-get update
 +
apt-get install yarn
 +
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
 +
apt-get install -y nodejs
 +
yarn install
 +
yarn run build
 +
</pre>
 +
 +
==== Cron jobs ====
 +
 +
User root:
 +
 +
* # m h  dom mon dow  command
 +
* 0 10 * * * /home/scripts/lowdiskspace.pl
 +
* 0 15 * * * /home/scripts/mdstat.pl
 +
* 0 2 * * * /home/scripts/backup.sh
 +
* */45 * * * * wget -q -O /home/off/html/cestemballepresdechezvous.html https://fr.openfoodfacts.org/cgi/madenearyou.pl
 +
* */50 * * * * wget -q -O /home/off/html/madenearme-uk.html https://uk.openfoodfacts.org/cgi/madenearme.pl
 +
* */55 * * * * wget -q -O /home/off/html/madenearme.html https://world.openfoodfacts.org/cgi/madenearme.pl
 +
 +
 +
Need a ftp client for the OVH FTP backup space (100G)
 +
* apt-get install ftp
 +
 +
install Filesys::DiskFree
 +
 +
 +
 +
User obf:
 +
* 35 * * * * /home/obf/scripts/gen_feeds.sh > /dev/null
 +
* 15 4 * * * /home/obf/scripts/gen_feeds_daily.sh > /dev/null
Retrieved from "https://wiki.openfoodfacts.org/Special:MobileDiff/5066...19025"

Navigation menu