Changes

Jump to navigation Jump to search
18,789 bytes added ,  14:59, 23 February 2023
no edit summary
Line 1: Line 1: −
= Infrastructure =
+
[[Category:ProductOpener]]
 +
[[Category:Developer]]
 +
[[Category:Roadmap]]
 +
[[Category:Infrastructure]]
    
This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects.
 
This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects.
   −
== New server install log ==
+
== Infrastructure planning ==
 +
 
 +
* Hardware needs: [[Infrastructure Planning]]
 +
 
 +
== 2021 roadmap ==
 +
* Please expand desirable objectives (not hardware needs)
 +
*
 +
* …
 +
 
 +
== Current infrastructure ==
 +
 
 +
=== off1 ===
 +
Dedicated free.org server:
 +
 
 +
'''Hardware configuration:'''
 +
* Dell 1U R440 server (1 Xeon Silver 4110: 8c/16t)
 +
* RAM: 64GB (DDR4@2666)
 +
* Disks:
 +
** 2x4 TB SATA HDD (hardware RAID1) : /srv
 +
** 2x14 TB SATA HDD (hardware RAID1) : /srv2 (images, imports)
 +
'''Running services:'''
 +
* OS: debian 9.5
 +
* 1 nginx reverse proxy to serve static files
 +
* 1 apache + mod_perl to generate dynamic pages + API
 +
** running 4 separate instances on different ports: OFF, OBF, OPFF, OPF
 +
 
 +
=== off2 ===
 +
Dedicated free.org server:
 +
 
 +
'''Hardware configuration:'''
 +
* Dell 1U R440 server (1 Xeon Silver 4110: 8c/16t)
 +
* RAM: 64GB (DDR4@2666)
 +
* Disks:
 +
** 2x4 TB SATA HDD (hardware RAID1) : /srv
 +
** 2x14 TB SATA HDD (hardware RAID1) : /srv2 (images, imports)
 +
'''Running services:'''
 +
* OS: debian 9.5
 +
* 1 apache + mod_perl (not used)
 +
* 1 MongoDB (production)
 +
* Robotoff (dockerized)
 +
 
 +
=== no name yet ===
 +
Dedicated OVH - soyoustart server (rented):
 +
 
 +
'''Hardware configuration:'''
 +
* E3-SAT-1-32
 +
* Storage: SoftRaid 3x2To SATA
 +
* RAM: 32GB DDR3 1333 MHz
 +
* CPU: Xeon E3-1245v2 (4c/8th)
 +
'''Running services:'''
 +
* wiki
 +
* zammad (soon)
 +
 
 +
=== dev ===
 +
Dedicated OVH - soyoustart server (rented):
 +
 
 +
'''Hardware configuration:'''
 +
* Super Micro server (1 Xeon D-1520)
 +
* Storage: 4 x 2TB HDD, software RAID1 / /boot + LVM /srv /srv2
 +
* RAM: 32GB (DDR4@2666)
 +
* CPU: Xeon D-1520
 +
'''Running services:'''
 +
* OS: debian 9.8
 +
* 1 nginx reverse proxy
 +
* 1 apache + mod_perl (https://openfoodfacts.dev - auth: off / off)
 +
* 1 mongodb
 +
 
 +
=== dev2 ===
 +
Dedicated server (self-hosted by [[User:Cquest|cquest]])
 +
 
 +
'''Hardware configuration:'''
 +
* Dell R710 server (2U)
 +
* Storage (ZFS pool) with:
 +
** 6 x 2TB HITACHI HUS72402 (SAS)
 +
** 500GB Samsung 860 SSD: OS and 256GB read cache (L2ARC)
 +
** 16GB Intel Optane SSD: write journal (ZIL)
 +
* RAM: 96GB (DDR3@1333)
 +
* CPU: 2 Xeon X5675 = 12 cores @ 3.07GHz
 +
'''Running services:'''
 +
* OS: Proxmox 6 (debian 10 based)
 +
 
 +
'''Network''':
 +
* uplink: optic fiber from OVH (1G down/500M up) with dedicated IPv4 and IPv6/72
 +
* ethernet 1Gbps (bonded for backup)
 +
* infiniband 40Gbps (proxmox cluster backbone + access to storage server)
 +
 
 +
=== backup ===
 +
Shared storage server (self-hosted by [[User:Cquest|cquest]])
 +
 
 +
'''Hardware configuration:'''
 +
* Supermicro storage server (4U)
 +
* Storage (ZFS pool) with:
 +
** 36 SAS bays (populated disks evolving with time are needs)
 +
** 1TB Samsung 860 SSD: OS and 256GB read cache (L2ARC)
 +
** 16GB Intel Optane SSD: write journal (ZIL)
 +
* RAM: 128GB
 +
* CPU:
 +
'''Running services:'''
 +
* OS: Proxmox 6 (debian 10 based)
 +
* Local NFS access from dev2 to /srv /srv2 backups (nghtly rsync + ZFS snapshots)
 +
* Proxmox CT/VM replication
 +
 
 +
=== ovh1 ===
 +
VM/CT server for computing usages.
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: AMD Epyc 7451 (24c/48t)
 +
* RAM: 256GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.1) on vRack
 +
* 2 x 1TB SSD NVMe
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on RAID1 /dev/md2
 +
* 512MB swap
 +
* 920GB ZFS pool on remaining space (mirror)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
 +
 
 +
=== ovh2 ===
 +
VM/CT server for computing usages (same as ovh1).
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: AMD Epyc 7451 (24c/48t)
 +
* RAM: 256GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.2) on vRack
 +
* 2 x 1TB SSD NVMe
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on RAID1 /dev/md2
 +
* 512MB swap
 +
* 920GB ZFS pool on remaining space (mirror)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
 +
 
 +
=== ovh3 ===
 +
Mainly storage server, VM/CT host is needed.
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: Intel Xeon-D 1541 (8c/16t)
 +
* RAM: 32GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.3)
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on SSD
 +
* 512MB swap
 +
* 5GB ZFS ZIL (write journal)
 +
* 472GB ZFS cache (ARC2 read cache)
 +
 
 +
* ZFS storage (6 x 12TB HDD) in RAIDZ2 (2 disks for redundancy, around 42TB available)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
 +
 
 +
=== ovh4 ===
 +
Mainly storage server, VM/CT host is needed.
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: Intel Xeon-D 1541 (8c/16t)
 +
* RAM: 32GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.4)
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on SSD
 +
* 512MB swap
 +
* 5GB ZFS ZIL (write journal)
 +
* 472GB ZFS cache (ARC2 read cache)
 +
 
 +
* ZFS storage (6 x 12TB HDD) in RAIDZ2 (2 disks for redundancy, around 42TB available)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
 +
 
 +
== History ==
 +
 
 +
From 2012 to 2016, OFF and OBF have been running on part of an OVH server (also used for other purposes):
 +
* cat /etc/debian_version -> 6.0.10 (squeeze)
 +
* 1 apache 2.4 setup as reverse proxy to serve static files (manually built)
 +
* 1 apache 1.3 + mod_perl for OFF (manually built)
 +
* 1 apache 1.3 + mod_perl for OBF (manually built)
 +
* MongoDB 2.4.12 (installed from mongodb provided packages)
 +
 
 +
From 2016 to 2018, 1 dedicated OVH server running:
 +
* 1 nginx reverse proxy to serve static files (installed with apt-get)
 +
* 1 apache 2.4 + mod_perl (installed with apt-get)
 +
** running two separate instances on different ports, 1 for OFF and 1 for OBF
 +
* 1 MongoDB
 +
 
 +
== 2012-2016 server install log ==
    
OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects.
 
OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects.
Line 11: Line 211:     
* E3-SAT-3 Intel Xeon E3 1245v2 4 c / 8 t 3.4 GHz+ 32 Go 2x 2 To SATA3 Soft 16* 40.00€ HT (SOIT 48.00€ TTC)
 
* E3-SAT-3 Intel Xeon E3 1245v2 4 c / 8 t 3.4 GHz+ 32 Go 2x 2 To SATA3 Soft 16* 40.00€ HT (SOIT 48.00€ TTC)
* http://www.soyoustart.com/fr/offres/e3-sat-3.xml
+
* https://www.soyoustart.com/fr/offres/e3-sat-3.xml
    
=== Server setup ===
 
=== Server setup ===
Line 43: Line 243:  
* off user
 
* off user
    +
==== IP failover ====
 +
 +
1 ip failover for each service so that we can easily switch servers
 +
* OFF: 178.33.252.125
 +
* OBF: 178.33.104.169
 +
 +
Add to /etc/network/interfaces :
 +
 +
<pre>
 +
 +
post-up /sbin/ifconfig eth0:0 178.33.104.169 netmask 255.255.255.255 broadcast 178.33.104.169
 +
post-down /sbin/ifconfig eth0:0 down
 +
 +
post-up /sbin/ifconfig eth0:1 178.33.252.125 netmask 255.255.255.255 broadcast 178.33.252.125
 +
post-down /sbin/ifconfig eth0:1 down
 +
</pre>
 +
 +
/etc/init.d/networking restart
 +
* [ ok ] Restarting networking (via systemctl): networking.service.
    
==== DNS ====
 
==== DNS ====
    
Product Opener needs a domain, with a A record for the domain itself and another wildcard A record for all subdomains.
 
Product Opener needs a domain, with a A record for the domain itself and another wildcard A record for all subdomains.
 +
 +
* openbeautyfacts.org. 0 A 178.33.104.169
 +
* *.openbeautyfacts.org. 0 A 178.33.104.169
    
For testing the new server, we will be using openfoodfacts.eu
 
For testing the new server, we will be using openfoodfacts.eu
Line 115: Line 337:  
# You should look at the following URL's in order to grasp a solid understanding
 
# You should look at the following URL's in order to grasp a solid understanding
 
# of Nginx configuration files in order to fully unleash the power of Nginx.
 
# of Nginx configuration files in order to fully unleash the power of Nginx.
# http://wiki.nginx.org/Pitfalls
+
# https://wiki.nginx.org/Pitfalls
# http://wiki.nginx.org/QuickStart
+
# https://wiki.nginx.org/QuickStart
# http://wiki.nginx.org/Configuration
+
# https://wiki.nginx.org/Configuration
 
#
 
#
 
# Generally, you will want to move this file somewhere, and start with a clean
 
# Generally, you will want to move this file somewhere, and start with a clean
Line 128: Line 350:  
#
 
#
 
server {
 
server {
         listen 80 default_server;
+
         #listen 80 default_server;
         listen [::]:80 default_server;
+
         #listen [::]:80 default_server;
   −
         server_name openfoodfacts.eu *.openfoodfacts.eu;
+
         server_name openfoodfacts.org *.openfoodfacts.org openfoodfacts.eu *.openfoodfacts.eu;
    
         # SSL configuration
 
         # SSL configuration
Line 144: Line 366:     
         root /home/off/html;
 
         root /home/off/html;
 +
 +
        access_log /home/off/logs/nginx.access2.log;
 +
        error_log /home/off/logs/nginx.error2.log;
 +
 +
        gzip on;
 +
        gzip_min_length 1000;
 +
    
         # Add index.php to the list if you are using PHP
 
         # Add index.php to the list if you are using PHP
 
         index index.html index.htm index.nginx-debian.html;
 
         index index.html index.htm index.nginx-debian.html;
 +
 +
        location ~* \.(eot|ttf|woff|woff2)$ {
 +
                add_header Access-Control-Allow-Origin *;
 +
        }
 +
 +
        location ~ ^/images/products/ {
 +
                add_header Link "<http://creativecommons.org/licenses/by-sa/3.0/>; rel='license'; title='CC-BY-SA 3.0'";
 +
        }
 +
 +
        location ~ ^/(favicon.ico) {
 +
                # First attempt to serve request as file, then
 +
                # as directory, then fall back to displaying a 404.
 +
                try_files $uri $uri/ =404;
 +
        }
 +
    
         location ~ ^/(images|js|rss|data|files|resources|foundation)/ {
 
         location ~ ^/(images|js|rss|data|files|resources|foundation)/ {
Line 161: Line 405:  
                 proxy_set_header Host $host;
 
                 proxy_set_header Host $host;
 
                 proxy_set_header X-Real-IP $remote_addr;
 
                 proxy_set_header X-Real-IP $remote_addr;
 +
                proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
 +
 
                 proxy_pass http://127.0.0.1:8001/cgi/display.pl?;
 
                 proxy_pass http://127.0.0.1:8001/cgi/display.pl?;
 
         }
 
         }
Line 167: Line 413:  
                 proxy_set_header Host $host;
 
                 proxy_set_header Host $host;
 
                 proxy_set_header X-Real-IP $remote_addr;
 
                 proxy_set_header X-Real-IP $remote_addr;
 +
                proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
 
                 proxy_pass http://127.0.0.1:8001;
 
                 proxy_pass http://127.0.0.1:8001;
 
         }
 
         }
Line 177: Line 424:  
         #}
 
         #}
 
}
 
}
 +
 +
 
</pre>
 
</pre>
   Line 189: Line 438:     
===== Apache configuration =====
 
===== Apache configuration =====
 +
 +
Note: imagemagick + mod_perl seems to crash Apache in the worker / event MPM, use prefork.
 +
 +
remove event and then:
 +
 +
/etc/apache2-off/mods-enabled# ln -s ../mods-available/mpm_prefork.conf mpm_prefork.conf
 +
/etc/apache2-off/mods-enabled# ln -s ../mods-available/mpm_prefork.load mpm_prefork.load
    
Set the user to off
 
Set the user to off
Line 207: Line 463:  
# LoadModule perl_module modules/mod_perl.so
 
# LoadModule perl_module modules/mod_perl.so
   −
PerlWarn Off
+
PerlSwitches -I/home/off/lib
PerlRequire /home/off/cgi/startup_apache2.pl
+
 
 +
PerlWarn On
 +
PerlRequire /home/off/lib/startup_apache2.pl
 +
 
    +
# log the X-Forwarded-For IP address (the client ip) in access_log
 +
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy
    
<Location /cgi>
 
<Location /cgi>
Line 222: Line 483:  
<VirtualHost *>
 
<VirtualHost *>
 
DocumentRoot /home/off/html
 
DocumentRoot /home/off/html
ServerName openfoodfacts.eu
+
ServerName openfoodfacts.org
 
ErrorLog /home/off/logs/error_log
 
ErrorLog /home/off/logs/error_log
 
CustomLog /home/off/logs/access_log combined
 
CustomLog /home/off/logs/access_log combined
LogLevel debug
+
LogLevel warn
 
ScriptAlias /cgi/ "/home/off/cgi/"
 
ScriptAlias /cgi/ "/home/off/cgi/"
   Line 265: Line 526:     
mkdir /home/off/logs
 
mkdir /home/off/logs
 +
 +
 +
Debugging:
 +
 +
If Apache does not start, check the apache2 logs (e.g. for apache2-obf) :
 +
 +
<pre>
 +
/var/log/apache2-obf# tail -f error.log
 +
[Mon Jul 04 15:38:52.616689 2016] [perl:error] [pid 15197:tid 120373734786944] Can't locate /home/obf/cgi/startup_apache2.pl in @INC (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/x86_64-linux-gnu/perl5/5.20 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl . /etc/apache2-obf) at (eval 2) line 1.\n
 +
[Mon Jul 04 15:38:52.616762 2016] [perl:error] [pid 15197:tid 120373734786944] Can't load Perl file: /home/obf/cgi/startup_apache2.pl for server ns3362784.ip-37-187-74.eu:0, exiting...
 +
</pre>
 +
 +
-> forgot to create startup_apache2.pl, going to copy the one from off and change the path in it.
 +
 +
More debugging:
 +
 +
If mod_perl starts, it loads some modules from cgi/startup_apache2.pl, and may fail if some modules are missing.
 +
Check /home/obf/logs/error_log:
 +
 +
<pre>
 +
[Mon Jul 04 18:31:11.501044 2016] [perl:error] [pid 23498:tid 116442525869952] Can't locate Math/Random/Secure.pm in @INC (you may need to install the Math::Random::Secure module)
 +
</pre>
 +
 +
===== Running a 2nd Apache instance for OBF =====
 +
 +
OFF and OBF share the same code (except some configuration in the Config.pm and SiteLang.pm modules), but the code is not made to run two separate servers.
 +
So we need two run two separate instances of Apache, one for OFF and one for OBF. This also allows to restart one but not the other, to run different versions of the code etc.
 +
 +
Note: we use the off user for both OFF and OBF so that OBF can read and write  in /home/off/users
 +
 +
There is some support to run multiple Apache instances. Documentation is in /usr/share/doc/apache2/README.multiple-instances
 +
 +
<pre>
 +
sh /usr/share/doc/apache2/examples/setup-instance obf
 +
Setting up /etc/apache2-obf ...
 +
Setting up /etc/init.d/apache2-obf ...
 +
Setting up symlinks: a2enmod-obf a2dismod-obf a2ensite-obf a2dissite-obf a2enconf-obf a2disconf-obf apache2ctl-obf
 +
Setting up /etc/logrotate.d/apache2-obf and /var/log/apache2-obf ...
 +
 +
</pre>
 +
 +
Configuration files for the new instance are in /etc/apache2-obf :
 +
 +
* Copy and edit obf.conf:
 +
** /etc/apache2-obf/sites-available# cp off.conf obf.conf
 +
** vi obf.conf
 +
*** :%s/off/obf/g
 +
*** ServerName openbeautyfacts.org
 +
* /etc/apache2-obf/sites-enabled# ls -lrt off.conf
 +
* lrwxrwxrwx 1 root root 27 Jun 17 14:45 off.conf -> ../sites-available/off.conf
 +
* /etc/apache2-obf/sites-enabled# rm off.conf
 +
* /etc/apache2-obf/sites-enabled# ln -s ../sites-available/obf.conf obf.conf
 +
 +
vi /etc/apache2-obf/envvars
 +
 +
<pre>
 +
#export APACHE_RUN_USER=www-data
 +
export APACHE_RUN_USER=off
 +
#export APACHE_RUN_GROUP=www-data
 +
export APACHE_RUN_GROUP=off
 +
</pre>
 +
 +
 +
Port 8002
 +
 +
<pre>
 +
 +
/etc/apache2-obf# vi ports.conf
 +
 +
#Listen 80
 +
Listen 8002
 +
 +
</pre>
 +
 +
<pre>
 +
service apache2-obf start
 +
Failed to start apache2-obf.service: Unit apache2-obf.service failed to load: No such file or directory.
 +
 +
Found solution in https://lists.debian.org/debian-apache/2016/03/msg00017.html
 +
 +
systemctl enable apache2-obf
 +
Synchronizing state for apache2-obf.service with sysvinit using update-rc.d...
 +
Executing /usr/sbin/update-rc.d apache2-obf defaults
 +
Executing /usr/sbin/update-rc.d apache2-obf enable
 +
systemctl start apache2-obf
 +
 +
</pre>
 +
 +
 +
To check for errors:
 +
 +
systemctl -l status apache2-obf.service
 +
 +
mkdir /home/obf/logs
 +
 +
 +
===== Forwarding client IPs from nginx to Apache configuration =====
 +
 +
in nginx config:
 +
 +
Apache:
 +
 +
install Apache2::Connection::XForwardedFor
 +
 +
Can't find the mod_perl include dir (reason: path /usr/include/apache2 doesn't exist)
 +
 +
-> just mkdir /usr/include/apache2 and reinstall module
    
==== Product Opener ====
 
==== Product Opener ====
Line 271: Line 639:     
* apt-get install zlib1g-dev
 
* apt-get install zlib1g-dev
 +
 +
====== GeoIP updatate ======
 +
 +
The packaged geoip database installed through "apt-get install geoip-database" is very old, need to install one manually:
 +
 +
<pre>
 +
root@ns3362784:/home/obf/logs# apt-get install geoip-database
 +
Reading package lists... Done
 +
Building dependency tree
 +
Reading state information... Done
 +
geoip-database is already the newest version.
 +
The following packages were automatically installed and are no longer required:
 +
  libasan0 libboost-dev libboost-filesystem1.55.0
 +
  libboost-program-options1.55.0 libboost-system1.55.0 libboost-thread1.55.0
 +
  libboost1.55-dev libgcc-4.8-dev libgoogle-perftools4 libpcrecpp0 libsnappy1
 +
  libstdc++-4.8-dev libtcmalloc-minimal4 libunwind8 libv8-3.14.5
 +
Use 'apt-get autoremove' to remove them.
 +
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
 +
root@ns3362784:/home/obf/logs# cd /usr/share/GeoIP/
 +
root@ns3362784:/usr/share/GeoIP# ls -lrt
 +
total 4900
 +
-rw-r--r-- 1 root root 1061883 Mar 17  2015 GeoIP.dat
 +
-rw-r--r-- 1 root root 3949299 Mar 17  2015 GeoIPv6.dat
 +
root@ns3362784:/usr/share/GeoIP# wget https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
 +
--2016-07-15 00:16:08--  https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
 +
Resolving geolite.maxmind.com (geolite.maxmind.com)... 2400:cb00:2048:1::6810:262f, 2400:cb00:2048:1::6810:252f, 104.16.38.47, ...
 +
Connecting to geolite.maxmind.com (geolite.maxmind.com)|2400:cb00:2048:1::6810:262f|:80... connected.
 +
HTTP request sent, awaiting response... 200 OK
 +
Length: 517602 (505K) [application/octet-stream]
 +
Saving to: ‘GeoIP.dat.gz’
 +
 +
GeoIP.dat.gz        100%[=====================>] 505.47K  --.-KB/s  in 0.06s
 +
 +
2016-07-15 00:16:08 (7.76 MB/s) - ‘GeoIP.dat.gz’ saved [517602/517602]
 +
 +
root@ns3362784:/usr/share/GeoIP# gzip -d GeoIP.dat.gz
 +
gzip: GeoIP.dat already exists; do you wish to overwrite (y or n)? y
 +
root@ns3362784:/usr/share/GeoIP# ls -lrt
 +
total 4744
 +
-rw-r--r-- 1 root root 3949299 Mar 17  2015 GeoIPv6.dat
 +
-rw-r--r-- 1 root root  904073 Jul  6 19:50 GeoIP.dat
 +
</pre>
 +
 +
Make a symbolic link (Perl module seems to think the database is at /usr/local/share/GeoIP/GeoIP.dat )
 +
 +
<pre>
 +
ln -s /usr/share/GeoIP /usr/local/share/GeoIP
 +
</pre>
    
===== Perl modules =====
 
===== Perl modules =====
Line 292: Line 708:  
</pre>
 
</pre>
   −
===== Symbolic links in cgi directory ======
+
===== Symbolic links in lib directory =====
    
ls -lrt |grep -- "->"
 
ls -lrt |grep -- "->"
Line 299: Line 715:     
<pre>
 
<pre>
/home/off/cgi# rm Blogs
+
/home/off/lib/ProductOpener# ln -s SiteLang_off.pm SiteLang.pm
/home/off/cgi# ln -s /home/off/cgi Blogs
+
/home/off/lib/ProductOpener# ln -s Config_off.pm Config.pm
   −
ln -s SiteLang_off.pm SiteLang.pm
+
/home/off/lib/ProductOpener# cp Config2_sample.pm Config2.pm
 +
-> put right values for server domain, home path, and mongodb database name
   −
vi /home/off/cgi/startup.pl
+
/home/off/scripts# ln -s ../lib/ProductOpener ProductOpener
# Needs to be configured
  −
use lib "/home/off/cgi/";
     −
vi /home/cgi/Config2.pm
  −
-> put right values for server domain, home path, and mongodb database name
   
</pre>
 
</pre>
      
===== robots.txt =====
 
===== robots.txt =====
Line 327: Line 739:       −
====== Data ======
+
===== Data =====
    
* Copy data to new server
 
* Copy data to new server
Line 335: Line 747:  
** chown -R off deleted* html index ingredients lang invalid lists products users tmp
 
** chown -R off deleted* html index ingredients lang invalid lists products users tmp
   −
====== Product opener debug =====
+
 
 +
===== MongoDB indexes =====
 +
 
 +
Some basic indexes, needs to be revisited.
 +
https://github.com/openfoodfacts/openfoodfacts-server/issues/341
 +
 
 +
<pre>
 +
mongo
 +
use off
 +
 
 +
 
 +
db.products.createIndex( { code : 1 }, { background : true });
 +
db.products.createIndex( { unique_scans_n : -1 }, { background : true });
 +
db.products.createIndex( { sortkey : -1 }, { background : true });
 +
db.products.createIndex( { last_modified_t : -1 }, { background : true });
 +
 
 +
db.products.createIndex( { created_t : 1 }, { background : true });
 +
db.products.createIndex( { lc : 1 }, { background : true });
 +
 
 +
db.products.createIndex( { _keywords : 1, last_modified_t : -1 }, { background : true });
 +
db.products.createIndex( { _keywords : 1, sortkey : -1 }, { background : true });
 +
 
 +
 
 +
db.products.createIndex( { countries_tags : 1 , last_modified_t : -1 }, { background : true });
 +
 
 +
db.products.createIndex( { countries_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { brands_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { categories_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { labels_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { packaging_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { origins_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { manufacturing_places_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { emb_codes_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { cities_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { ingredients_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { ingredients_from_palm_oil_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { vitamins_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { minerals_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { amino_acids_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { nucleotides_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { other_nutritional_substances_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { allergens_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { traces_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { nova_groups_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { nutrition_grades_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { misc_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { languages_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { users_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { editors_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { informers_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { correctors_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { checkers_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { photographers_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { states_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { entry_dates_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { last_edit_dates_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { purchase_places_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { stores_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { codes_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { debug_tags : 1 , sortkey : -1 }, { background : true });
 +
 
 +
 
 +
 
 +
 
 +
</pre>
 +
 
 +
===== Product opener debug =====
    
Once Apache starts:
 
Once Apache starts:
Line 346: Line 824:     
-> created startup_apache2.pl, loaded in off.conf
 
-> created startup_apache2.pl, loaded in off.conf
 +
 +
 +
===== Generate CSS and download dependencies =====
 +
 +
<pre>
 +
/srv/off/yarn install
 +
/srv/off/yarn run build
 +
</pre>
 +
 +
The commands above might fail if the correct versions of yarn and nodejs are not installed. To fix:
 +
 +
<pre>
 +
apt remove cmdtest
 +
apt remove yarn
 +
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
 +
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
 +
apt-get update
 +
apt-get install yarn
 +
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
 +
apt-get install -y nodejs
 +
yarn install
 +
yarn run build
 +
</pre>
 +
 +
==== Cron jobs ====
 +
 +
User root:
 +
 +
* # m h  dom mon dow  command
 +
* 0 10 * * * /home/scripts/lowdiskspace.pl
 +
* 0 15 * * * /home/scripts/mdstat.pl
 +
* 0 2 * * * /home/scripts/backup.sh
 +
* */45 * * * * wget -q -O /home/off/html/cestemballepresdechezvous.html https://fr.openfoodfacts.org/cgi/madenearyou.pl
 +
* */50 * * * * wget -q -O /home/off/html/madenearme-uk.html https://uk.openfoodfacts.org/cgi/madenearme.pl
 +
* */55 * * * * wget -q -O /home/off/html/madenearme.html https://world.openfoodfacts.org/cgi/madenearme.pl
 +
 +
 +
Need a ftp client for the OVH FTP backup space (100G)
 +
* apt-get install ftp
 +
 +
install Filesys::DiskFree
 +
 +
 +
 +
User obf:
 +
* 35 * * * * /home/obf/scripts/gen_feeds.sh > /dev/null
 +
* 15 4 * * * /home/obf/scripts/gen_feeds_daily.sh > /dev/null

Navigation menu