Line 1: |
Line 1: |
− | = Infrastructure =
| + | [[Category:ProductOpener]] |
| + | [[Category:Developer]] |
| + | [[Category:Roadmap]] |
| + | [[Category:Infrastructure]] |
| | | |
| This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects. | | This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects. |
| | | |
− | == High level infrastructure design == | + | == Infrastructure planning == |
| | | |
− | 1 dedicated OVH server running: | + | * Hardware needs: [[Infrastructure Planning]] |
− | * 1 nginx reverse proxy to serve static files (installed with apt-get) | + | |
− | * 1 apache 2.4 + mod_perl (installed with apt-get) | + | == 2021 roadmap == |
− | ** running two separate instances on different ports, 1 for OFF and 1 for OBF | + | * Please expand desirable objectives (not hardware needs) |
− | * 1 MongoDB | + | * |
| + | * … |
| + | |
| + | == Current infrastructure == |
| + | |
| + | === off1 === |
| + | Dedicated free.org server: |
| + | |
| + | '''Hardware configuration:''' |
| + | * Dell 1U R440 server (1 Xeon Silver 4110: 8c/16t) |
| + | * RAM: 64GB (DDR4@2666) |
| + | * Disks: |
| + | ** 2x4 TB SATA HDD (hardware RAID1) : /srv |
| + | ** 2x14 TB SATA HDD (hardware RAID1) : /srv2 (images, imports) |
| + | '''Running services:''' |
| + | * OS: debian 9.5 |
| + | * 1 nginx reverse proxy to serve static files |
| + | * 1 apache + mod_perl to generate dynamic pages + API |
| + | ** running 4 separate instances on different ports: OFF, OBF, OPFF, OPF |
| + | |
| + | === off2 === |
| + | Dedicated free.org server: |
| + | |
| + | '''Hardware configuration:''' |
| + | * Dell 1U R440 server (1 Xeon Silver 4110: 8c/16t) |
| + | * RAM: 64GB (DDR4@2666) |
| + | * Disks: |
| + | ** 2x4 TB SATA HDD (hardware RAID1) : /srv |
| + | ** 2x14 TB SATA HDD (hardware RAID1) : /srv2 (images, imports) |
| + | '''Running services:''' |
| + | * OS: debian 9.5 |
| + | * 1 apache + mod_perl (not used) |
| + | * 1 MongoDB (production) |
| + | * Robotoff (dockerized) |
| + | |
| + | === no name yet === |
| + | Dedicated OVH - soyoustart server (rented): |
| + | |
| + | '''Hardware configuration:''' |
| + | * E3-SAT-1-32 |
| + | * Storage: SoftRaid 3x2To SATA |
| + | * RAM: 32GB DDR3 1333 MHz |
| + | * CPU: Xeon E3-1245v2 (4c/8th) |
| + | '''Running services:''' |
| + | * wiki |
| + | * zammad (soon) |
| + | |
| + | === dev === |
| + | Dedicated OVH - soyoustart server (rented): |
| + | |
| + | '''Hardware configuration:''' |
| + | * Super Micro server (1 Xeon D-1520) |
| + | * Storage: 4 x 2TB HDD, software RAID1 / /boot + LVM /srv /srv2 |
| + | * RAM: 32GB (DDR4@2666) |
| + | * CPU: Xeon D-1520 |
| + | '''Running services:''' |
| + | * OS: debian 9.8 |
| + | * 1 nginx reverse proxy |
| + | * 1 apache + mod_perl (https://openfoodfacts.dev - auth: off / off) |
| + | * 1 mongodb |
| + | |
| + | === dev2 === |
| + | Dedicated server (self-hosted by [[User:Cquest|cquest]]) |
| + | |
| + | '''Hardware configuration:''' |
| + | * Dell R710 server (2U) |
| + | * Storage (ZFS pool) with: |
| + | ** 6 x 2TB HITACHI HUS72402 (SAS) |
| + | ** 500GB Samsung 860 SSD: OS and 256GB read cache (L2ARC) |
| + | ** 16GB Intel Optane SSD: write journal (ZIL) |
| + | * RAM: 96GB (DDR3@1333) |
| + | * CPU: 2 Xeon X5675 = 12 cores @ 3.07GHz |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (debian 10 based) |
| + | |
| + | '''Network''': |
| + | * uplink: optic fiber from OVH (1G down/500M up) with dedicated IPv4 and IPv6/72 |
| + | * ethernet 1Gbps (bonded for backup) |
| + | * infiniband 40Gbps (proxmox cluster backbone + access to storage server) |
| + | |
| + | === backup === |
| + | Shared storage server (self-hosted by [[User:Cquest|cquest]]) |
| + | |
| + | '''Hardware configuration:''' |
| + | * Supermicro storage server (4U) |
| + | * Storage (ZFS pool) with: |
| + | ** 36 SAS bays (populated disks evolving with time are needs) |
| + | ** 1TB Samsung 860 SSD: OS and 256GB read cache (L2ARC) |
| + | ** 16GB Intel Optane SSD: write journal (ZIL) |
| + | * RAM: 128GB |
| + | * CPU: |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (debian 10 based) |
| + | * Local NFS access from dev2 to /srv /srv2 backups (nghtly rsync + ZFS snapshots) |
| + | * Proxmox CT/VM replication |
| + | |
| + | === ovh1 === |
| + | VM/CT server for computing usages. |
| + | |
| + | See [[Infrastructure/VM cluster]]. |
| + | |
| + | '''Hardware configuration:''' |
| + | * CPU: AMD Epyc 7451 (24c/48t) |
| + | * RAM: 256GB |
| + | * 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.1) on vRack |
| + | * 2 x 1TB SSD NVMe |
| + | |
| + | Partition / storage: |
| + | * Ext4 root partition: 32Gb on RAID1 /dev/md2 |
| + | * 512MB swap |
| + | * 920GB ZFS pool on remaining space (mirror) |
| + | |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (based on debian 10 buster) |
| + | |
| + | === ovh2 === |
| + | VM/CT server for computing usages (same as ovh1). |
| + | |
| + | See [[Infrastructure/VM cluster]]. |
| + | |
| + | '''Hardware configuration:''' |
| + | * CPU: AMD Epyc 7451 (24c/48t) |
| + | * RAM: 256GB |
| + | * 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.2) on vRack |
| + | * 2 x 1TB SSD NVMe |
| + | |
| + | Partition / storage: |
| + | * Ext4 root partition: 32Gb on RAID1 /dev/md2 |
| + | * 512MB swap |
| + | * 920GB ZFS pool on remaining space (mirror) |
| + | |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (based on debian 10 buster) |
| + | |
| + | === ovh3 === |
| + | Mainly storage server, VM/CT host is needed. |
| + | |
| + | See [[Infrastructure/VM cluster]]. |
| + | |
| + | '''Hardware configuration:''' |
| + | * CPU: Intel Xeon-D 1541 (8c/16t) |
| + | * RAM: 32GB |
| + | * 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.3) |
| + | |
| + | Partition / storage: |
| + | * Ext4 root partition: 32Gb on SSD |
| + | * 512MB swap |
| + | * 5GB ZFS ZIL (write journal) |
| + | * 472GB ZFS cache (ARC2 read cache) |
| + | |
| + | * ZFS storage (6 x 12TB HDD) in RAIDZ2 (2 disks for redundancy, around 42TB available) |
| + | |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (based on debian 10 buster) |
| + | |
| + | === ovh4 === |
| + | Mainly storage server, VM/CT host is needed. |
| + | |
| + | See [[Infrastructure/VM cluster]]. |
| + | |
| + | '''Hardware configuration:''' |
| + | * CPU: Intel Xeon-D 1541 (8c/16t) |
| + | * RAM: 32GB |
| + | * 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.4) |
| + | |
| + | Partition / storage: |
| + | * Ext4 root partition: 32Gb on SSD |
| + | * 512MB swap |
| + | * 5GB ZFS ZIL (write journal) |
| + | * 472GB ZFS cache (ARC2 read cache) |
| + | |
| + | * ZFS storage (6 x 12TB HDD) in RAIDZ2 (2 disks for redundancy, around 42TB available) |
| + | |
| + | '''Running services:''' |
| + | * OS: Proxmox 6 (based on debian 10 buster) |
| | | |
− | === History ===
| + | == History == |
| | | |
| From 2012 to 2016, OFF and OBF have been running on part of an OVH server (also used for other purposes): | | From 2012 to 2016, OFF and OBF have been running on part of an OVH server (also used for other purposes): |
Line 20: |
Line 197: |
| * MongoDB 2.4.12 (installed from mongodb provided packages) | | * MongoDB 2.4.12 (installed from mongodb provided packages) |
| | | |
| + | From 2016 to 2018, 1 dedicated OVH server running: |
| + | * 1 nginx reverse proxy to serve static files (installed with apt-get) |
| + | * 1 apache 2.4 + mod_perl (installed with apt-get) |
| + | ** running two separate instances on different ports, 1 for OFF and 1 for OBF |
| + | * 1 MongoDB |
| | | |
− | | + | == 2012-2016 server install log == |
− | == New server install log == | |
| | | |
| OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects. | | OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects. |
Line 30: |
Line 211: |
| | | |
| * E3-SAT-3 Intel Xeon E3 1245v2 4 c / 8 t 3.4 GHz+ 32 Go 2x 2 To SATA3 Soft 16* 40.00€ HT (SOIT 48.00€ TTC) | | * E3-SAT-3 Intel Xeon E3 1245v2 4 c / 8 t 3.4 GHz+ 32 Go 2x 2 To SATA3 Soft 16* 40.00€ HT (SOIT 48.00€ TTC) |
− | * http://www.soyoustart.com/fr/offres/e3-sat-3.xml | + | * https://www.soyoustart.com/fr/offres/e3-sat-3.xml |
| | | |
| === Server setup === | | === Server setup === |
Line 156: |
Line 337: |
| # You should look at the following URL's in order to grasp a solid understanding | | # You should look at the following URL's in order to grasp a solid understanding |
| # of Nginx configuration files in order to fully unleash the power of Nginx. | | # of Nginx configuration files in order to fully unleash the power of Nginx. |
− | # http://wiki.nginx.org/Pitfalls | + | # https://wiki.nginx.org/Pitfalls |
− | # http://wiki.nginx.org/QuickStart | + | # https://wiki.nginx.org/QuickStart |
− | # http://wiki.nginx.org/Configuration | + | # https://wiki.nginx.org/Configuration |
| # | | # |
| # Generally, you will want to move this file somewhere, and start with a clean | | # Generally, you will want to move this file somewhere, and start with a clean |
Line 257: |
Line 438: |
| | | |
| ===== Apache configuration ===== | | ===== Apache configuration ===== |
| + | |
| + | Note: imagemagick + mod_perl seems to crash Apache in the worker / event MPM, use prefork. |
| + | |
| + | remove event and then: |
| + | |
| + | /etc/apache2-off/mods-enabled# ln -s ../mods-available/mpm_prefork.conf mpm_prefork.conf |
| + | /etc/apache2-off/mods-enabled# ln -s ../mods-available/mpm_prefork.load mpm_prefork.load |
| | | |
| Set the user to off | | Set the user to off |
Line 274: |
Line 462: |
| /etc/apache2/sites-available# cat off.conf | | /etc/apache2/sites-available# cat off.conf |
| # LoadModule perl_module modules/mod_perl.so | | # LoadModule perl_module modules/mod_perl.so |
| + | |
| + | PerlSwitches -I/home/off/lib |
| | | |
| PerlWarn On | | PerlWarn On |
− | PerlRequire /home/off/cgi/startup_apache2.pl | + | PerlRequire /home/off/lib/startup_apache2.pl |
| + | |
| | | |
| # log the X-Forwarded-For IP address (the client ip) in access_log | | # log the X-Forwarded-For IP address (the client ip) in access_log |
Line 471: |
Line 662: |
| -rw-r--r-- 1 root root 1061883 Mar 17 2015 GeoIP.dat | | -rw-r--r-- 1 root root 1061883 Mar 17 2015 GeoIP.dat |
| -rw-r--r-- 1 root root 3949299 Mar 17 2015 GeoIPv6.dat | | -rw-r--r-- 1 root root 3949299 Mar 17 2015 GeoIPv6.dat |
− | root@ns3362784:/usr/share/GeoIP# wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz | + | root@ns3362784:/usr/share/GeoIP# wget https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz |
− | --2016-07-15 00:16:08-- http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz | + | --2016-07-15 00:16:08-- https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz |
| Resolving geolite.maxmind.com (geolite.maxmind.com)... 2400:cb00:2048:1::6810:262f, 2400:cb00:2048:1::6810:252f, 104.16.38.47, ... | | Resolving geolite.maxmind.com (geolite.maxmind.com)... 2400:cb00:2048:1::6810:262f, 2400:cb00:2048:1::6810:252f, 104.16.38.47, ... |
| Connecting to geolite.maxmind.com (geolite.maxmind.com)|2400:cb00:2048:1::6810:262f|:80... connected. | | Connecting to geolite.maxmind.com (geolite.maxmind.com)|2400:cb00:2048:1::6810:262f|:80... connected. |
Line 517: |
Line 708: |
| </pre> | | </pre> |
| | | |
− | ===== Symbolic links in cgi directory ===== | + | ===== Symbolic links in lib directory ===== |
| | | |
| ls -lrt |grep -- "->" | | ls -lrt |grep -- "->" |
Line 524: |
Line 715: |
| | | |
| <pre> | | <pre> |
− | /home/off/cgi# rm Blogs | + | /home/off/lib/ProductOpener# ln -s SiteLang_off.pm SiteLang.pm |
− | /home/off/cgi# ln -s /home/off/cgi Blogs | + | /home/off/lib/ProductOpener# ln -s Config_off.pm Config.pm |
| | | |
− | ln -s SiteLang_off.pm SiteLang.pm
| + | /home/off/lib/ProductOpener# cp Config2_sample.pm Config2.pm |
| + | -> put right values for server domain, home path, and mongodb database name |
| | | |
− | vi /home/off/cgi/startup.pl
| + | /home/off/scripts# ln -s ../lib/ProductOpener ProductOpener |
− | # Needs to be configured
| |
− | use lib "/home/off/cgi/";
| |
| | | |
− | vi /home/cgi/Config2.pm
| |
− | -> put right values for server domain, home path, and mongodb database name
| |
| </pre> | | </pre> |
| | | |
Line 567: |
Line 755: |
| <pre> | | <pre> |
| mongo | | mongo |
− | use obf | + | use off |
| | | |
| | | |
Line 573: |
Line 761: |
| db.products.createIndex( { unique_scans_n : -1 }, { background : true }); | | db.products.createIndex( { unique_scans_n : -1 }, { background : true }); |
| db.products.createIndex( { sortkey : -1 }, { background : true }); | | db.products.createIndex( { sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { last_modified_t : -1 }, { background : true }); |
| | | |
− | db.products.createIndex( { sortkey : -1 }, { background : true }); | + | db.products.createIndex( { created_t : 1 }, { background : true }); |
− | db.products.createIndex( { last_modified_t : -1 }, { background : true }); | + | db.products.createIndex( { lc : 1 }, { background : true }); |
| | | |
| db.products.createIndex( { _keywords : 1, last_modified_t : -1 }, { background : true }); | | db.products.createIndex( { _keywords : 1, last_modified_t : -1 }, { background : true }); |
| db.products.createIndex( { _keywords : 1, sortkey : -1 }, { background : true }); | | db.products.createIndex( { _keywords : 1, sortkey : -1 }, { background : true }); |
| | | |
− | db.products.createIndex( { countries_tags : 1 }, { background : true });
| |
| | | |
| db.products.createIndex( { countries_tags : 1 , last_modified_t : -1 }, { background : true }); | | db.products.createIndex( { countries_tags : 1 , last_modified_t : -1 }, { background : true }); |
| + | |
| db.products.createIndex( { countries_tags : 1 , sortkey : -1 }, { background : true }); | | db.products.createIndex( { countries_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { brands_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { categories_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { labels_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { packaging_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { origins_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { manufacturing_places_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { emb_codes_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { cities_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { ingredients_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { ingredients_from_palm_oil_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { vitamins_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { minerals_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { amino_acids_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { nucleotides_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { other_nutritional_substances_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { allergens_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { traces_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { nova_groups_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { nutrition_grades_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { misc_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { languages_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { users_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { editors_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { informers_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { correctors_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { checkers_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { photographers_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { states_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { entry_dates_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { last_edit_dates_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { purchase_places_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { stores_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { codes_tags : 1 , sortkey : -1 }, { background : true }); |
| + | db.products.createIndex( { debug_tags : 1 , sortkey : -1 }, { background : true }); |
| | | |
− | db.products.createIndex( { brands_tags : 1 }, { background : true });
| |
− | db.products.createIndex( { categories_tags : 1 }, { background : true });
| |
| | | |
| | | |
Line 605: |
Line 826: |
| | | |
| | | |
| + | ===== Generate CSS and download dependencies ===== |
| + | |
| + | <pre> |
| + | /srv/off/yarn install |
| + | /srv/off/yarn run build |
| + | </pre> |
| | | |
| + | The commands above might fail if the correct versions of yarn and nodejs are not installed. To fix: |
| + | |
| + | <pre> |
| + | apt remove cmdtest |
| + | apt remove yarn |
| + | curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - |
| + | echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list |
| + | apt-get update |
| + | apt-get install yarn |
| + | curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - |
| + | apt-get install -y nodejs |
| + | yarn install |
| + | yarn run build |
| + | </pre> |
| | | |
| ==== Cron jobs ==== | | ==== Cron jobs ==== |
Line 615: |
Line 856: |
| * 0 15 * * * /home/scripts/mdstat.pl | | * 0 15 * * * /home/scripts/mdstat.pl |
| * 0 2 * * * /home/scripts/backup.sh | | * 0 2 * * * /home/scripts/backup.sh |
− | * */45 * * * * wget -q -O /home/off/html/cestemballepresdechezvous.html http://fr.openfoodfacts.org/cgi/madenearyou.pl | + | * */45 * * * * wget -q -O /home/off/html/cestemballepresdechezvous.html https://fr.openfoodfacts.org/cgi/madenearyou.pl |
− | * */50 * * * * wget -q -O /home/off/html/madenearme-uk.html http://uk.openfoodfacts.org/cgi/madenearme.pl | + | * */50 * * * * wget -q -O /home/off/html/madenearme-uk.html https://uk.openfoodfacts.org/cgi/madenearme.pl |
− | * */55 * * * * wget -q -O /home/off/html/madenearme.html http://world.openfoodfacts.org/cgi/madenearme.pl | + | * */55 * * * * wget -q -O /home/off/html/madenearme.html https://world.openfoodfacts.org/cgi/madenearme.pl |
| | | |
| | | |