Changes

Jump to navigation Jump to search

Infrastructure (view source)

Revision as of 14:59, 23 February 2023

8,497 bytes added ,  14:59, 23 February 2023
no edit summary
Line 1: Line 1:  
[[Category:ProductOpener]]
 
[[Category:ProductOpener]]
 
[[Category:Developer]]
 
[[Category:Developer]]
−
= Infrastructure =
+
[[Category:Roadmap]]
 +
[[Category:Infrastructure]]
    
This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects.
 
This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects.
   −
== High level infrastructure design ==
+
== Infrastructure planning ==
   −
1 dedicated OVH server running:
+
* Hardware needs: [[Infrastructure Planning]]
−
* 1 nginx reverse proxy to serve static files (installed with apt-get)
+
 
−
* 1 apache 2.4 + mod_perl (installed with apt-get)
+
== 2021 roadmap ==
−
** running two separate instances on different ports, 1 for OFF and 1 for OBF
+
* Please expand desirable objectives (not hardware needs)
−
* 1 MongoDB
+
*
 +
* …
 +
 
 +
== Current infrastructure ==
 +
 
 +
=== off1 ===
 +
Dedicated free.org server:
 +
 
 +
'''Hardware configuration:'''
 +
* Dell 1U R440 server (1 Xeon Silver 4110: 8c/16t)
 +
* RAM: 64GB (DDR4@2666)
 +
* Disks:
 +
** 2x4 TB SATA HDD (hardware RAID1) : /srv
 +
** 2x14 TB SATA HDD (hardware RAID1) : /srv2 (images, imports)
 +
'''Running services:'''
 +
* OS: debian 9.5
 +
* 1 nginx reverse proxy to serve static files
 +
* 1 apache + mod_perl to generate dynamic pages + API
 +
** running 4 separate instances on different ports: OFF, OBF, OPFF, OPF
 +
 
 +
=== off2 ===
 +
Dedicated free.org server:
 +
 
 +
'''Hardware configuration:'''
 +
* Dell 1U R440 server (1 Xeon Silver 4110: 8c/16t)
 +
* RAM: 64GB (DDR4@2666)
 +
* Disks:
 +
** 2x4 TB SATA HDD (hardware RAID1) : /srv
 +
** 2x14 TB SATA HDD (hardware RAID1) : /srv2 (images, imports)
 +
'''Running services:'''
 +
* OS: debian 9.5
 +
* 1 apache + mod_perl (not used)
 +
* 1 MongoDB (production)
 +
* Robotoff (dockerized)
 +
 
 +
=== no name yet ===
 +
Dedicated OVH - soyoustart server (rented):
 +
 
 +
'''Hardware configuration:'''
 +
* E3-SAT-1-32
 +
* Storage: SoftRaid 3x2To SATA
 +
* RAM: 32GB DDR3 1333 MHz
 +
* CPU: Xeon E3-1245v2 (4c/8th)
 +
'''Running services:'''
 +
* wiki
 +
* zammad (soon)
 +
 
 +
=== dev ===
 +
Dedicated OVH - soyoustart server (rented):
 +
 
 +
'''Hardware configuration:'''
 +
* Super Micro server (1 Xeon D-1520)
 +
* Storage: 4 x 2TB HDD, software RAID1 / /boot + LVM /srv /srv2
 +
* RAM: 32GB (DDR4@2666)
 +
* CPU: Xeon D-1520
 +
'''Running services:'''
 +
* OS: debian 9.8
 +
* 1 nginx reverse proxy
 +
* 1 apache + mod_perl (https://openfoodfacts.dev - auth: off / off)
 +
* 1 mongodb
 +
 
 +
=== dev2 ===
 +
Dedicated server (self-hosted by [[User:Cquest|cquest]])
 +
 
 +
'''Hardware configuration:'''
 +
* Dell R710 server (2U)
 +
* Storage (ZFS pool) with:
 +
** 6 x 2TB HITACHI HUS72402 (SAS)
 +
** 500GB Samsung 860 SSD: OS and 256GB read cache (L2ARC)
 +
** 16GB Intel Optane SSD: write journal (ZIL)
 +
* RAM: 96GB (DDR3@1333)
 +
* CPU: 2 Xeon X5675 = 12 cores @ 3.07GHz
 +
'''Running services:'''
 +
* OS: Proxmox 6 (debian 10 based)
 +
 
 +
'''Network''':
 +
* uplink: optic fiber from OVH (1G down/500M up) with dedicated IPv4 and IPv6/72
 +
* ethernet 1Gbps (bonded for backup)
 +
* infiniband 40Gbps (proxmox cluster backbone + access to storage server)
 +
 
 +
=== backup ===
 +
Shared storage server (self-hosted by [[User:Cquest|cquest]])
 +
 
 +
'''Hardware configuration:'''
 +
* Supermicro storage server (4U)
 +
* Storage (ZFS pool) with:
 +
** 36 SAS bays (populated disks evolving with time are needs)
 +
** 1TB Samsung 860 SSD: OS and 256GB read cache (L2ARC)
 +
** 16GB Intel Optane SSD: write journal (ZIL)
 +
* RAM: 128GB
 +
* CPU:
 +
'''Running services:'''
 +
* OS: Proxmox 6 (debian 10 based)
 +
* Local NFS access from dev2 to /srv /srv2 backups (nghtly rsync + ZFS snapshots)
 +
* Proxmox CT/VM replication
 +
 
 +
=== ovh1 ===
 +
VM/CT server for computing usages.
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: AMD Epyc 7451 (24c/48t)
 +
* RAM: 256GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.1) on vRack
 +
* 2 x 1TB SSD NVMe
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on RAID1 /dev/md2
 +
* 512MB swap
 +
* 920GB ZFS pool on remaining space (mirror)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
 +
 
 +
=== ovh2 ===
 +
VM/CT server for computing usages (same as ovh1).
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: AMD Epyc 7451 (24c/48t)
 +
* RAM: 256GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.2) on vRack
 +
* 2 x 1TB SSD NVMe
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on RAID1 /dev/md2
 +
* 512MB swap
 +
* 920GB ZFS pool on remaining space (mirror)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
 +
 
 +
=== ovh3 ===
 +
Mainly storage server, VM/CT host is needed.
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: Intel Xeon-D 1541 (8c/16t)
 +
* RAM: 32GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.3)
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on SSD
 +
* 512MB swap
 +
* 5GB ZFS ZIL (write journal)
 +
* 472GB ZFS cache (ARC2 read cache)
 +
 
 +
* ZFS storage (6 x 12TB HDD) in RAIDZ2 (2 disks for redundancy, around 42TB available)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
 +
 
 +
=== ovh4 ===
 +
Mainly storage server, VM/CT host is needed.
 +
 
 +
See [[Infrastructure/VM cluster]].
 +
 
 +
'''Hardware configuration:'''
 +
* CPU: Intel Xeon-D 1541 (8c/16t)
 +
* RAM: 32GB
 +
* 1 GB/s public bandwith; 3 GB/s private bandwith (10.0.0.4)
 +
 
 +
Partition / storage:
 +
* Ext4 root partition: 32Gb on SSD
 +
* 512MB swap
 +
* 5GB ZFS ZIL (write journal)
 +
* 472GB ZFS cache (ARC2 read cache)
 +
 
 +
* ZFS storage (6 x 12TB HDD) in RAIDZ2 (2 disks for redundancy, around 42TB available)
 +
 
 +
'''Running services:'''
 +
* OS: Proxmox 6 (based on debian 10 buster)
   −
=== History ===
+
== History ==
    
From 2012 to 2016, OFF and OBF have been running on part of an OVH server (also used for other purposes):
 
From 2012 to 2016, OFF and OBF have been running on part of an OVH server (also used for other purposes):
Line 22: Line 197:  
* MongoDB 2.4.12 (installed from mongodb provided packages)
 
* MongoDB 2.4.12 (installed from mongodb provided packages)
    +
From 2016 to 2018, 1 dedicated OVH server running:
 +
* 1 nginx reverse proxy to serve static files (installed with apt-get)
 +
* 1 apache 2.4 + mod_perl (installed with apt-get)
 +
** running two separate instances on different ports, 1 for OFF and 1 for OBF
 +
* 1 MongoDB
   −
 
+
== 2012-2016 server install log ==
−
== New server install log ==
      
OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects.
 
OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects.
Line 259: Line 438:     
===== Apache configuration =====
 
===== Apache configuration =====
 +
 +
Note: imagemagick + mod_perl seems to crash Apache in the worker / event MPM, use prefork.
 +
 +
remove event and then:
 +
 +
/etc/apache2-off/mods-enabled# ln -s ../mods-available/mpm_prefork.conf mpm_prefork.conf
 +
/etc/apache2-off/mods-enabled# ln -s ../mods-available/mpm_prefork.load mpm_prefork.load
    
Set the user to off
 
Set the user to off
Line 476: Line 662:  
-rw-r--r-- 1 root root 1061883 Mar 17  2015 GeoIP.dat
 
-rw-r--r-- 1 root root 1061883 Mar 17  2015 GeoIP.dat
 
-rw-r--r-- 1 root root 3949299 Mar 17  2015 GeoIPv6.dat
 
-rw-r--r-- 1 root root 3949299 Mar 17  2015 GeoIPv6.dat
−
root@ns3362784:/usr/share/GeoIP# wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
+
root@ns3362784:/usr/share/GeoIP# wget https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
−
--2016-07-15 00:16:08--  http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
+
--2016-07-15 00:16:08--  https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
 
Resolving geolite.maxmind.com (geolite.maxmind.com)... 2400:cb00:2048:1::6810:262f, 2400:cb00:2048:1::6810:252f, 104.16.38.47, ...
 
Resolving geolite.maxmind.com (geolite.maxmind.com)... 2400:cb00:2048:1::6810:262f, 2400:cb00:2048:1::6810:252f, 104.16.38.47, ...
 
Connecting to geolite.maxmind.com (geolite.maxmind.com)|2400:cb00:2048:1::6810:262f|:80... connected.
 
Connecting to geolite.maxmind.com (geolite.maxmind.com)|2400:cb00:2048:1::6810:262f|:80... connected.
Line 569: Line 755:  
<pre>
 
<pre>
 
mongo
 
mongo
−
use obf
+
use off
      Line 575: Line 761:  
db.products.createIndex( { unique_scans_n : -1 }, { background : true });
 
db.products.createIndex( { unique_scans_n : -1 }, { background : true });
 
db.products.createIndex( { sortkey : -1 }, { background : true });
 
db.products.createIndex( { sortkey : -1 }, { background : true });
 +
db.products.createIndex( { last_modified_t : -1 }, { background : true });
   −
db.products.createIndex( { sortkey : -1 }, { background : true });
+
db.products.createIndex( { created_t : 1 }, { background : true });
−
db.products.createIndex( { last_modified_t : -1 }, { background : true });
+
db.products.createIndex( { lc : 1 }, { background : true });
    
db.products.createIndex( { _keywords : 1, last_modified_t : -1 }, { background : true });
 
db.products.createIndex( { _keywords : 1, last_modified_t : -1 }, { background : true });
 
db.products.createIndex( { _keywords : 1, sortkey : -1 }, { background : true });
 
db.products.createIndex( { _keywords : 1, sortkey : -1 }, { background : true });
   −
db.products.createIndex( { countries_tags : 1 }, { background : true });
      
db.products.createIndex( { countries_tags : 1 , last_modified_t : -1 }, { background : true });
 
db.products.createIndex( { countries_tags : 1 , last_modified_t : -1 }, { background : true });
 +
 
db.products.createIndex( { countries_tags : 1 , sortkey : -1 }, { background : true });
 
db.products.createIndex( { countries_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { brands_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { categories_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { labels_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { packaging_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { origins_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { manufacturing_places_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { emb_codes_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { cities_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { ingredients_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { ingredients_from_palm_oil_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { vitamins_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { minerals_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { amino_acids_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { nucleotides_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { other_nutritional_substances_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { allergens_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { traces_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { nova_groups_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { nutrition_grades_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { misc_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { languages_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { users_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { editors_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { informers_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { correctors_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { checkers_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { photographers_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { states_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { entry_dates_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { last_edit_dates_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { purchase_places_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { stores_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { codes_tags : 1 , sortkey : -1 }, { background : true });
 +
db.products.createIndex( { debug_tags : 1 , sortkey : -1 }, { background : true });
   −
db.products.createIndex( { brands_tags : 1 }, { background : true });
  −
db.products.createIndex( { categories_tags : 1 }, { background : true });
  −
  −
db.products.createIndex( { created_t : 1 });
        Line 609: Line 826:        +
===== Generate CSS and download dependencies =====
 +
 +
<pre>
 +
/srv/off/yarn install
 +
/srv/off/yarn run build
 +
</pre>
    +
The commands above might fail if the correct versions of yarn and nodejs are not installed. To fix:
 +
 +
<pre>
 +
apt remove cmdtest
 +
apt remove yarn
 +
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
 +
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
 +
apt-get update
 +
apt-get install yarn
 +
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
 +
apt-get install -y nodejs
 +
yarn install
 +
yarn run build
 +
</pre>
    
==== Cron jobs ====
 
==== Cron jobs ====
Retrieved from "https://wiki.openfoodfacts.org/Special:MobileDiff/6674...19025"

Navigation menu