Line 1: |
Line 1: |
− | = Infrastructure =
| + | [[Category:ProductOpener]] |
| + | [[Category:Developer]] |
| + | [[Category:Roadmap]] |
| + | [[Category:Infrastructure]] |
| | | |
| This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects. | | This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects. |
| | | |
− | == New server install log == | + | == Infrastructure planning == |
| + | |
| + | * Hardware needs: [[Infrastructure Planning]] |
| + | |
| + | == 2021 roadmap == |
| + | * Please expand desirable objectives (not hardware needs) |
| + | * |
| + | * … |
| + | |
| + | == Current infrastructure == |
| + | |
| + | === off1 === |
| + | Dedicated free.org server: |
| + | |
| + | '''Hardware configuration:''' |
| + | * Dell 1U R440 server (1 Xeon Silver 4110: 8c/16t) |
| + | * RAM: 64GB (DDR4@2666) |
| + | * Disks: |
| + | ** 2x4 TB SATA HDD (hardware RAID1) : /srv |
| + | ** 2x14 TB SATA HDD (hardware RAID1) : /srv2 (images, imports) |
| + | '''Running services:''' |
| + | * OS: debian 9.5 |
| + | * 1 nginx reverse proxy to serve static files |
| + | * 1 apache + mod_perl to generate dynamic pages + API |
| + | ** running 4 separate instances on different ports: OFF, OBF, OPFF, OPF |
| + | |
| + | === off2 === |
| + | Dedicated free.org server: |
| + | |
| + | '''Hardware configuration:''' |
| + | * Dell 1U R440 server (1 Xeon Silver 4110: 8c/16t) |
| + | * RAM: 64GB (DDR4@2666) |
| + | * Disks: |
| + | ** 2x4 TB SATA HDD (hardware RAID1) : /srv |
| + | ** 2x14 TB SATA HDD (hardware RAID1) : /srv2 (images, imports) |
| + | '''Running services:''' |
| + | * OS: debian 9.5 |
| + | * 1 apache + mod_perl (not used) |
| + | * 1 MongoDB (production) |
| + | * Robotoff (dockerized) |
| + | |
| + | === no name yet === |
| + | Dedicated OVH - soyoustart server (rented): |
| + | |
| + | '''Hardware configuration:''' |
| + | * E3-SAT-1-32 |
| + | * Storage: SoftRaid 3x2To SATA |
| + | * RAM: 32GB DDR3 1333 MHz |
| + | * CPU: Xeon E3-1245v2 (4c/8th) |
| + | '''Running services:''' |
| + | * wiki |
| + | * zammad (soon) |
| + | |
| + | === dev === |
| + | Dedicated OVH - soyoustart server (rented): |
| + | |
| + | '''Hardware configuration:''' |
| + | * Super Micro server (1 Xeon D-1520) |
| + | * Storage: 4 x 2TB HDD, software RAID1 / /boot + LVM /srv /srv2 |
| + | * RAM: 32GB (DDR4@2666) |
| + | * CPU: Xeon D-1520 |
| + | '''Running services:''' |
| + | * OS: debian 9.8 |
| + | * 1 nginx reverse proxy |
| + | * 1 apache + mod_perl (https://openfoodfacts.dev - auth: off / off) |
| + | * 1 mongodb |
| + | |
| + | === dev2 === |
| + | Dedicated server (self-hosted by [[User:Cquest|cquest]]) |
| + | |
| + | '''Hardware configuration:''' |
| + | * Dell R710 server (2U) |
| + | * Storage (ZFS pool) with: |
| + | ** 6 x 2TB HITACHI HUS72402 (SAS) |
| + | ** 500GB Samsung 860 SSD: OS and 256GB read cache (L2ARC) |
| + | ** 16GB Intel Optane SSD: write journal (ZIL) |
| + | * RAM: 96GB (DDR3@1333) |
| + | * CPU: 2 Xeon X5675 = 12 cores @ 3.07GHz |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (debian 10 based) |
| + | |
| + | '''Network''': |
| + | * uplink: optic fiber from OVH (1G down/500M up) with dedicated IPv4 and IPv6/72 |
| + | * ethernet 1Gbps (bonded for backup) |
| + | * infiniband 40Gbps (proxmox cluster backbone + access to storage server) |
| + | |
| + | === backup === |
| + | Shared storage server (self-hosted by [[User:Cquest|cquest]]) |
| + | |
| + | '''Hardware configuration:''' |
| + | * Supermicro storage server (4U) |
| + | * Storage (ZFS pool) with: |
| + | ** 36 SAS bays (populated disks evolving with time are needs) |
| + | ** 1TB Samsung 860 SSD: OS and 256GB read cache (L2ARC) |
| + | ** 16GB Intel Optane SSD: write journal (ZIL) |
| + | * RAM: 128GB |
| + | * CPU: |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (debian 10 based) |
| + | * Local NFS access from dev2 to /srv /srv2 backups (nghtly rsync + ZFS snapshots) |
| + | * Proxmox CT/VM replication |
| + | |
| + | === ovh1 === |
| + | VM/CT server for computing usages. |
| + | |
| + | See [[Infrastructure/VM cluster]]. |
| + | |
| + | '''Hardware configuration:''' |
| + | * CPU: AMD Epyc 7451 (24c/48t) |
| + | * RAM: 256GB |
| + | * 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.1) on vRack |
| + | * 2 x 1TB SSD NVMe |
| + | |
| + | Partition / storage: |
| + | * Ext4 root partition: 32Gb on RAID1 /dev/md2 |
| + | * 512MB swap |
| + | * 920GB ZFS pool on remaining space (mirror) |
| + | |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (based on debian 10 buster) |
| + | |
| + | === ovh2 === |
| + | VM/CT server for computing usages (same as ovh1). |
| + | |
| + | See [[Infrastructure/VM cluster]]. |
| + | |
| + | '''Hardware configuration:''' |
| + | * CPU: AMD Epyc 7451 (24c/48t) |
| + | * RAM: 256GB |
| + | * 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.2) on vRack |
| + | * 2 x 1TB SSD NVMe |
| + | |
| + | Partition / storage: |
| + | * Ext4 root partition: 32Gb on RAID1 /dev/md2 |
| + | * 512MB swap |
| + | * 920GB ZFS pool on remaining space (mirror) |
| + | |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (based on debian 10 buster) |
| + | |
| + | === ovh3 === |
| + | Mainly storage server, VM/CT host is needed. |
| + | |
| + | See [[Infrastructure/VM cluster]]. |
| + | |
| + | '''Hardware configuration:''' |
| + | * CPU: Intel Xeon-D 1541 (8c/16t) |
| + | * RAM: 32GB |
| + | * 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.3) |
| + | |
| + | Partition / storage: |
| + | * Ext4 root partition: 32Gb on SSD |
| + | * 512MB swap |
| + | * 5GB ZFS ZIL (write journal) |
| + | * 472GB ZFS cache (ARC2 read cache) |
| + | |
| + | * ZFS storage (6 x 12TB HDD) in RAIDZ2 (2 disks for redundancy, around 42TB available) |
| + | |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (based on debian 10 buster) |
| + | |
| + | === ovh4 === |
| + | Mainly storage server, VM/CT host is needed. |
| + | |
| + | See [[Infrastructure/VM cluster]]. |
| + | |
| + | '''Hardware configuration:''' |
| + | * CPU: Intel Xeon-D 1541 (8c/16t) |
| + | * RAM: 32GB |
| + | * 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.4) |
| + | |
| + | Partition / storage: |
| + | * Ext4 root partition: 32Gb on SSD |
| + | * 512MB swap |
| + | * 5GB ZFS ZIL (write journal) |
| + | * 472GB ZFS cache (ARC2 read cache) |
| + | |
| + | * ZFS storage (6 x 12TB HDD) in RAIDZ2 (2 disks for redundancy, around 42TB available) |
| + | |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (based on debian 10 buster) |
| + | |
| + | == History == |
| + | |
| + | From 2012 to 2016, OFF and OBF have been running on part of an OVH server (also used for other purposes): |
| + | * cat /etc/debian_version -> 6.0.10 (squeeze) |
| + | * 1 apache 2.4 setup as reverse proxy to serve static files (manually built) |
| + | * 1 apache 1.3 + mod_perl for OFF (manually built) |
| + | * 1 apache 1.3 + mod_perl for OBF (manually built) |
| + | * MongoDB 2.4.12 (installed from mongodb provided packages) |
| + | |
| + | From 2016 to 2018, 1 dedicated OVH server running: |
| + | * 1 nginx reverse proxy to serve static files (installed with apt-get) |
| + | * 1 apache 2.4 + mod_perl (installed with apt-get) |
| + | ** running two separate instances on different ports, 1 for OFF and 1 for OBF |
| + | * 1 MongoDB |
| + | |
| + | == 2012-2016 server install log == |
| | | |
| OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects. | | OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects. |
Line 11: |
Line 211: |
| | | |
| * E3-SAT-3 Intel Xeon E3 1245v2 4 c / 8 t 3.4 GHz+ 32 Go 2x 2 To SATA3 Soft 16* 40.00€ HT (SOIT 48.00€ TTC) | | * E3-SAT-3 Intel Xeon E3 1245v2 4 c / 8 t 3.4 GHz+ 32 Go 2x 2 To SATA3 Soft 16* 40.00€ HT (SOIT 48.00€ TTC) |
− | * http://www.soyoustart.com/fr/offres/e3-sat-3.xml | + | * https://www.soyoustart.com/fr/offres/e3-sat-3.xml |
| | | |
| === Server setup === | | === Server setup === |
Line 43: |
Line 243: |
| * off user | | * off user |
| | | |
| + | ==== IP failover ==== |
| + | |
| + | 1 ip failover for each service so that we can easily switch servers |
| + | * OFF: 178.33.252.125 |
| + | * OBF: 178.33.104.169 |
| + | |
| + | Add to /etc/network/interfaces : |
| + | |
| + | <pre> |
| + | |
| + | post-up /sbin/ifconfig eth0:0 178.33.104.169 netmask 255.255.255.255 broadcast 178.33.104.169 |
| + | post-down /sbin/ifconfig eth0:0 down |
| + | |
| + | post-up /sbin/ifconfig eth0:1 178.33.252.125 netmask 255.255.255.255 broadcast 178.33.252.125 |
| + | post-down /sbin/ifconfig eth0:1 down |
| + | </pre> |
| + | |
| + | /etc/init.d/networking restart |
| + | * [ ok ] Restarting networking (via systemctl): networking.service. |
| | | |
| ==== DNS ==== | | ==== DNS ==== |
| | | |
| Product Opener needs a domain, with a A record for the domain itself and another wildcard A record for all subdomains. | | Product Opener needs a domain, with a A record for the domain itself and another wildcard A record for all subdomains. |
| + | |
| + | * openbeautyfacts.org. 0 A 178.33.104.169 |
| + | * *.openbeautyfacts.org. 0 A 178.33.104.169 |
| | | |
| For testing the new server, we will be using openfoodfacts.eu | | For testing the new server, we will be using openfoodfacts.eu |
Line 115: |
Line 337: |
| # You should look at the following URL's in order to grasp a solid understanding | | # You should look at the following URL's in order to grasp a solid understanding |
| # of Nginx configuration files in order to fully unleash the power of Nginx. | | # of Nginx configuration files in order to fully unleash the power of Nginx. |
− | # http://wiki.nginx.org/Pitfalls | + | # https://wiki.nginx.org/Pitfalls |
− | # http://wiki.nginx.org/QuickStart | + | # https://wiki.nginx.org/QuickStart |
− | # http://wiki.nginx.org/Configuration | + | # https://wiki.nginx.org/Configuration |
| # | | # |
| # Generally, you will want to move this file somewhere, and start with a clean | | # Generally, you will want to move this file somewhere, and start with a clean |
Line 128: |
Line 350: |
| # | | # |
| server { | | server { |
− | listen 80 default_server; | + | #listen 80 default_server; |
− | listen [::]:80 default_server; | + | #listen [::]:80 default_server; |
| | | |
− | server_name openfoodfacts.eu *.openfoodfacts.eu; | + | server_name openfoodfacts.org *.openfoodfacts.org openfoodfacts.eu *.openfoodfacts.eu; |
| | | |
| # SSL configuration | | # SSL configuration |
Line 144: |
Line 366: |
| | | |
| root /home/off/html; | | root /home/off/html; |
| + | |
| + | access_log /home/off/logs/nginx.access2.log; |
| + | error_log /home/off/logs/nginx.error2.log; |
| + | |
| + | gzip on; |
| + | gzip_min_length 1000; |
| + | |
| | | |
| # Add index.php to the list if you are using PHP | | # Add index.php to the list if you are using PHP |
| index index.html index.htm index.nginx-debian.html; | | index index.html index.htm index.nginx-debian.html; |
| + | |
| + | location ~* \.(eot|ttf|woff|woff2)$ { |
| + | add_header Access-Control-Allow-Origin *; |
| + | } |
| + | |
| + | location ~ ^/images/products/ { |
| + | add_header Link "<http://creativecommons.org/licenses/by-sa/3.0/>; rel='license'; title='CC-BY-SA 3.0'"; |
| + | } |
| + | |
| + | location ~ ^/(favicon.ico) { |
| + | # First attempt to serve request as file, then |
| + | # as directory, then fall back to displaying a 404. |
| + | try_files $uri $uri/ =404; |
| + | } |
| + | |
| | | |
| location ~ ^/(images|js|rss|data|files|resources|foundation)/ { | | location ~ ^/(images|js|rss|data|files|resources|foundation)/ { |
Line 161: |
Line 405: |
| proxy_set_header Host $host; | | proxy_set_header Host $host; |
| proxy_set_header X-Real-IP $remote_addr; | | proxy_set_header X-Real-IP $remote_addr; |
| + | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; |
| + | |
| proxy_pass http://127.0.0.1:8001/cgi/display.pl?; | | proxy_pass http://127.0.0.1:8001/cgi/display.pl?; |
| } | | } |
Line 167: |
Line 413: |
| proxy_set_header Host $host; | | proxy_set_header Host $host; |
| proxy_set_header X-Real-IP $remote_addr; | | proxy_set_header X-Real-IP $remote_addr; |
| + | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; |
| proxy_pass http://127.0.0.1:8001; | | proxy_pass http://127.0.0.1:8001; |
| } | | } |
Line 177: |
Line 424: |
| #} | | #} |
| } | | } |
| + | |
| + | |
| </pre> | | </pre> |
| | | |
Line 189: |
Line 438: |
| | | |
| ===== Apache configuration ===== | | ===== Apache configuration ===== |
| + | |
| + | Note: imagemagick + mod_perl seems to crash Apache in the worker / event MPM, use prefork. |
| + | |
| + | remove event and then: |
| + | |
| + | /etc/apache2-off/mods-enabled# ln -s ../mods-available/mpm_prefork.conf mpm_prefork.conf |
| + | /etc/apache2-off/mods-enabled# ln -s ../mods-available/mpm_prefork.load mpm_prefork.load |
| + | |
| + | Set the user to off |
| + | |
| + | vi /etc/apache2/envvars |
| + | |
| + | <pre> |
| + | #export APACHE_RUN_USER=www-data |
| + | export APACHE_RUN_USER=off |
| + | #export APACHE_RUN_GROUP=www-data |
| + | export APACHE_RUN_GROUP=off |
| + | </pre> |
| | | |
| off.conf: | | off.conf: |
Line 196: |
Line 463: |
| # LoadModule perl_module modules/mod_perl.so | | # LoadModule perl_module modules/mod_perl.so |
| | | |
− | PerlWarn Off | + | PerlSwitches -I/home/off/lib |
− | PerlRequire /home/off/cgi/startup.pl | + | |
| + | PerlWarn On |
| + | PerlRequire /home/off/lib/startup_apache2.pl |
| | | |
| + | |
| + | # log the X-Forwarded-For IP address (the client ip) in access_log |
| + | LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy |
| | | |
| <Location /cgi> | | <Location /cgi> |
Line 211: |
Line 483: |
| <VirtualHost *> | | <VirtualHost *> |
| DocumentRoot /home/off/html | | DocumentRoot /home/off/html |
− | ServerName openfoodfacts.eu | + | ServerName openfoodfacts.org |
| ErrorLog /home/off/logs/error_log | | ErrorLog /home/off/logs/error_log |
| CustomLog /home/off/logs/access_log combined | | CustomLog /home/off/logs/access_log combined |
− | LogLevel debug | + | LogLevel warn |
| ScriptAlias /cgi/ "/home/off/cgi/" | | ScriptAlias /cgi/ "/home/off/cgi/" |
| | | |
Line 254: |
Line 526: |
| | | |
| mkdir /home/off/logs | | mkdir /home/off/logs |
| + | |
| + | |
| + | Debugging: |
| + | |
| + | If Apache does not start, check the apache2 logs (e.g. for apache2-obf) : |
| + | |
| + | <pre> |
| + | /var/log/apache2-obf# tail -f error.log |
| + | [Mon Jul 04 15:38:52.616689 2016] [perl:error] [pid 15197:tid 120373734786944] Can't locate /home/obf/cgi/startup_apache2.pl in @INC (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/x86_64-linux-gnu/perl5/5.20 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl . /etc/apache2-obf) at (eval 2) line 1.\n |
| + | [Mon Jul 04 15:38:52.616762 2016] [perl:error] [pid 15197:tid 120373734786944] Can't load Perl file: /home/obf/cgi/startup_apache2.pl for server ns3362784.ip-37-187-74.eu:0, exiting... |
| + | </pre> |
| + | |
| + | -> forgot to create startup_apache2.pl, going to copy the one from off and change the path in it. |
| + | |
| + | More debugging: |
| + | |
| + | If mod_perl starts, it loads some modules from cgi/startup_apache2.pl, and may fail if some modules are missing. |
| + | Check /home/obf/logs/error_log: |
| + | |
| + | <pre> |
| + | [Mon Jul 04 18:31:11.501044 2016] [perl:error] [pid 23498:tid 116442525869952] Can't locate Math/Random/Secure.pm in @INC (you may need to install the Math::Random::Secure module) |
| + | </pre> |
| + | |
| + | ===== Running a 2nd Apache instance for OBF ===== |
| + | |
| + | OFF and OBF share the same code (except some configuration in the Config.pm and SiteLang.pm modules), but the code is not made to run two separate servers. |
| + | So we need two run two separate instances of Apache, one for OFF and one for OBF. This also allows to restart one but not the other, to run different versions of the code etc. |
| + | |
| + | Note: we use the off user for both OFF and OBF so that OBF can read and write in /home/off/users |
| + | |
| + | There is some support to run multiple Apache instances. Documentation is in /usr/share/doc/apache2/README.multiple-instances |
| + | |
| + | <pre> |
| + | sh /usr/share/doc/apache2/examples/setup-instance obf |
| + | Setting up /etc/apache2-obf ... |
| + | Setting up /etc/init.d/apache2-obf ... |
| + | Setting up symlinks: a2enmod-obf a2dismod-obf a2ensite-obf a2dissite-obf a2enconf-obf a2disconf-obf apache2ctl-obf |
| + | Setting up /etc/logrotate.d/apache2-obf and /var/log/apache2-obf ... |
| + | |
| + | </pre> |
| + | |
| + | Configuration files for the new instance are in /etc/apache2-obf : |
| + | |
| + | * Copy and edit obf.conf: |
| + | ** /etc/apache2-obf/sites-available# cp off.conf obf.conf |
| + | ** vi obf.conf |
| + | *** :%s/off/obf/g |
| + | *** ServerName openbeautyfacts.org |
| + | * /etc/apache2-obf/sites-enabled# ls -lrt off.conf |
| + | * lrwxrwxrwx 1 root root 27 Jun 17 14:45 off.conf -> ../sites-available/off.conf |
| + | * /etc/apache2-obf/sites-enabled# rm off.conf |
| + | * /etc/apache2-obf/sites-enabled# ln -s ../sites-available/obf.conf obf.conf |
| + | |
| + | vi /etc/apache2-obf/envvars |
| + | |
| + | <pre> |
| + | #export APACHE_RUN_USER=www-data |
| + | export APACHE_RUN_USER=off |
| + | #export APACHE_RUN_GROUP=www-data |
| + | export APACHE_RUN_GROUP=off |
| + | </pre> |
| + | |
| + | |
| + | Port 8002 |
| + | |
| + | <pre> |
| + | |
| + | /etc/apache2-obf# vi ports.conf |
| + | |
| + | #Listen 80 |
| + | Listen 8002 |
| + | |
| + | </pre> |
| + | |
| + | <pre> |
| + | service apache2-obf start |
| + | Failed to start apache2-obf.service: Unit apache2-obf.service failed to load: No such file or directory. |
| + | |
| + | Found solution in https://lists.debian.org/debian-apache/2016/03/msg00017.html |
| + | |
| + | systemctl enable apache2-obf |
| + | Synchronizing state for apache2-obf.service with sysvinit using update-rc.d... |
| + | Executing /usr/sbin/update-rc.d apache2-obf defaults |
| + | Executing /usr/sbin/update-rc.d apache2-obf enable |
| + | systemctl start apache2-obf |
| + | |
| + | </pre> |
| + | |
| + | |
| + | To check for errors: |
| + | |
| + | systemctl -l status apache2-obf.service |
| + | |
| + | mkdir /home/obf/logs |
| + | |
| + | |
| + | ===== Forwarding client IPs from nginx to Apache configuration ===== |
| + | |
| + | in nginx config: |
| + | |
| + | Apache: |
| + | |
| + | install Apache2::Connection::XForwardedFor |
| + | |
| + | Can't find the mod_perl include dir (reason: path /usr/include/apache2 doesn't exist) |
| + | |
| + | -> just mkdir /usr/include/apache2 and reinstall module |
| | | |
| ==== Product Opener ==== | | ==== Product Opener ==== |
Line 260: |
Line 639: |
| | | |
| * apt-get install zlib1g-dev | | * apt-get install zlib1g-dev |
| + | |
| + | ====== GeoIP updatate ====== |
| + | |
| + | The packaged geoip database installed through "apt-get install geoip-database" is very old, need to install one manually: |
| + | |
| + | <pre> |
| + | root@ns3362784:/home/obf/logs# apt-get install geoip-database |
| + | Reading package lists... Done |
| + | Building dependency tree |
| + | Reading state information... Done |
| + | geoip-database is already the newest version. |
| + | The following packages were automatically installed and are no longer required: |
| + | libasan0 libboost-dev libboost-filesystem1.55.0 |
| + | libboost-program-options1.55.0 libboost-system1.55.0 libboost-thread1.55.0 |
| + | libboost1.55-dev libgcc-4.8-dev libgoogle-perftools4 libpcrecpp0 libsnappy1 |
| + | libstdc++-4.8-dev libtcmalloc-minimal4 libunwind8 libv8-3.14.5 |
| + | Use 'apt-get autoremove' to remove them. |
| + | 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. |
| + | root@ns3362784:/home/obf/logs# cd /usr/share/GeoIP/ |
| + | root@ns3362784:/usr/share/GeoIP# ls -lrt |
| + | total 4900 |
| + | -rw-r--r-- 1 root root 1061883 Mar 17 2015 GeoIP.dat |
| + | -rw-r--r-- 1 root root 3949299 Mar 17 2015 GeoIPv6.dat |
| + | root@ns3362784:/usr/share/GeoIP# wget https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz |
| + | --2016-07-15 00:16:08-- https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz |
| + | Resolving geolite.maxmind.com (geolite.maxmind.com)... 2400:cb00:2048:1::6810:262f, 2400:cb00:2048:1::6810:252f, 104.16.38.47, ... |
| + | Connecting to geolite.maxmind.com (geolite.maxmind.com)|2400:cb00:2048:1::6810:262f|:80... connected. |
| + | HTTP request sent, awaiting response... 200 OK |
| + | Length: 517602 (505K) [application/octet-stream] |
| + | Saving to: ‘GeoIP.dat.gz’ |
| + | |
| + | GeoIP.dat.gz 100%[=====================>] 505.47K --.-KB/s in 0.06s |
| + | |
| + | 2016-07-15 00:16:08 (7.76 MB/s) - ‘GeoIP.dat.gz’ saved [517602/517602] |
| + | |
| + | root@ns3362784:/usr/share/GeoIP# gzip -d GeoIP.dat.gz |
| + | gzip: GeoIP.dat already exists; do you wish to overwrite (y or n)? y |
| + | root@ns3362784:/usr/share/GeoIP# ls -lrt |
| + | total 4744 |
| + | -rw-r--r-- 1 root root 3949299 Mar 17 2015 GeoIPv6.dat |
| + | -rw-r--r-- 1 root root 904073 Jul 6 19:50 GeoIP.dat |
| + | </pre> |
| + | |
| + | Make a symbolic link (Perl module seems to think the database is at /usr/local/share/GeoIP/GeoIP.dat ) |
| + | |
| + | <pre> |
| + | ln -s /usr/share/GeoIP /usr/local/share/GeoIP |
| + | </pre> |
| | | |
| ===== Perl modules ===== | | ===== Perl modules ===== |
Line 281: |
Line 708: |
| </pre> | | </pre> |
| | | |
− | ===== Symbolic links in cgi directory ====== | + | ===== Symbolic links in lib directory ===== |
| | | |
| ls -lrt |grep -- "->" | | ls -lrt |grep -- "->" |
Line 288: |
Line 715: |
| | | |
| <pre> | | <pre> |
− | /home/off/cgi# rm Blogs | + | /home/off/lib/ProductOpener# ln -s SiteLang_off.pm SiteLang.pm |
− | /home/off/cgi# ln -s /home/off/cgi Blogs | + | /home/off/lib/ProductOpener# ln -s Config_off.pm Config.pm |
| | | |
− | ln -s SiteLang_off.pm SiteLang.pm
| + | /home/off/lib/ProductOpener# cp Config2_sample.pm Config2.pm |
| + | -> put right values for server domain, home path, and mongodb database name |
| | | |
− | vi /home/off/cgi/startup.pl
| + | /home/off/scripts# ln -s ../lib/ProductOpener ProductOpener |
− | # Needs to be configured
| |
− | use lib "/home/off/cgi/";
| |
| | | |
− | vi /home/cgi/Config2.pm
| |
− | -> put right values for server domain, home path, and mongodb database name
| |
| </pre> | | </pre> |
− |
| |
| | | |
| ===== robots.txt ===== | | ===== robots.txt ===== |
Line 314: |
Line 737: |
| ~ | | ~ |
| </pre> | | </pre> |
| + | |
| + | |
| + | ===== Data ===== |
| + | |
| + | * Copy data to new server |
| + | ** Populate Mongodb by running /home/off/cgi/update_all_products_from_dir_in_mongodb.pl |
| + | * Copy images |
| + | * Make sur permissions are correct |
| + | ** chown -R off deleted* html index ingredients lang invalid lists products users tmp |
| + | |
| + | |
| + | ===== MongoDB indexes ===== |
| + | |
| + | Some basic indexes, needs to be revisited. |
| + | https://github.com/openfoodfacts/openfoodfacts-server/issues/341 |
| + | |
| + | <pre> |
| + | mongo |
| + | use off |
| + | |
| + | |
| + | db.products.createIndex( { code : 1 }, { background : true }); |
| + | db.products.createIndex( { unique_scans_n : -1 }, { background : true }); |
| + | db.products.createIndex( { sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { last_modified_t : -1 }, { background : true }); |
| + | |
| + | db.products.createIndex( { created_t : 1 }, { background : true }); |
| + | db.products.createIndex( { lc : 1 }, { background : true }); |
| + | |
| + | db.products.createIndex( { _keywords : 1, last_modified_t : -1 }, { background : true }); |
| + | db.products.createIndex( { _keywords : 1, sortkey : -1 }, { background : true }); |
| + | |
| + | |
| + | db.products.createIndex( { countries_tags : 1 , last_modified_t : -1 }, { background : true }); |
| + | |
| + | db.products.createIndex( { countries_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { brands_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { categories_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { labels_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { packaging_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { origins_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { manufacturing_places_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { emb_codes_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { cities_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { ingredients_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { ingredients_from_palm_oil_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { vitamins_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { minerals_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { amino_acids_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { nucleotides_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { other_nutritional_substances_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { allergens_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { traces_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { nova_groups_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { nutrition_grades_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { misc_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { languages_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { users_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { editors_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { informers_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { correctors_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { checkers_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { photographers_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { states_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { entry_dates_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { last_edit_dates_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { purchase_places_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { stores_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { codes_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { debug_tags : 1 , sortkey : -1 }, { background : true }); |
| + | |
| + | |
| + | |
| + | |
| + | </pre> |
| + | |
| + | ===== Product opener debug ===== |
| + | |
| + | Once Apache starts: |
| + | |
| + | /home/off/logs# tail -f error_log |
| + | |
| + | <pre> |
| + | [Fri Jun 17 14:51:34.123696 2016] [perl:error] [pid 18764:tid 117837813839616] [client 127.0.0.1:35749] Can't locate object method "remote_ip" via package "Apache2::Connection" at /home/off/cgi/startup.pl line 76.\n, referer: http://openfoodfacts.eu/ |
| + | </pre> |
| + | |
| + | -> created startup_apache2.pl, loaded in off.conf |
| + | |
| + | |
| + | ===== Generate CSS and download dependencies ===== |
| + | |
| + | <pre> |
| + | /srv/off/yarn install |
| + | /srv/off/yarn run build |
| + | </pre> |
| + | |
| + | The commands above might fail if the correct versions of yarn and nodejs are not installed. To fix: |
| + | |
| + | <pre> |
| + | apt remove cmdtest |
| + | apt remove yarn |
| + | curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - |
| + | echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list |
| + | apt-get update |
| + | apt-get install yarn |
| + | curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - |
| + | apt-get install -y nodejs |
| + | yarn install |
| + | yarn run build |
| + | </pre> |
| + | |
| + | ==== Cron jobs ==== |
| + | |
| + | User root: |
| + | |
| + | * # m h dom mon dow command |
| + | * 0 10 * * * /home/scripts/lowdiskspace.pl |
| + | * 0 15 * * * /home/scripts/mdstat.pl |
| + | * 0 2 * * * /home/scripts/backup.sh |
| + | * */45 * * * * wget -q -O /home/off/html/cestemballepresdechezvous.html https://fr.openfoodfacts.org/cgi/madenearyou.pl |
| + | * */50 * * * * wget -q -O /home/off/html/madenearme-uk.html https://uk.openfoodfacts.org/cgi/madenearme.pl |
| + | * */55 * * * * wget -q -O /home/off/html/madenearme.html https://world.openfoodfacts.org/cgi/madenearme.pl |
| + | |
| + | |
| + | Need a ftp client for the OVH FTP backup space (100G) |
| + | * apt-get install ftp |
| + | |
| + | install Filesys::DiskFree |
| + | |
| + | |
| + | |
| + | User obf: |
| + | * 35 * * * * /home/obf/scripts/gen_feeds.sh > /dev/null |
| + | * 15 4 * * * /home/obf/scripts/gen_feeds_daily.sh > /dev/null |