Difference between revisions of "Infrastructure Planning"

From Open Food Facts wiki
Jump to navigation Jump to search
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
[[Category:Infrastructure]]
 
== Current infrastructure ==
 
== Current infrastructure ==
  
Line 7: Line 8:
 
* dev: dev server, robotoff, remote backup
 
* dev: dev server, robotoff, remote backup
  
== Hardware needs 2019 ==
+
== Hardware needs 2020 ==
  
 
=== Production ===
 
=== Production ===
Line 13: Line 14:
 
* Frontend + backend for the web site and API
 
* Frontend + backend for the web site and API
 
** off1
 
** off1
 +
 
* MongoDB
 
* MongoDB
 
** off2
 
** off2
* Robotoff
+
 
 +
* '''Robotoff'''
 
** New server
 
** New server
*** Models are memory hungry, at least 32 GB of RAM is needed.
+
*** Models are memory hungry, at least 32 GB of RAM is needed. Ideally, 64 GB.
*** GPU servers deliver the best performances for prediction, but are rather expensive. Given the current load, CPU servers are enough.
+
*** Ideally, 1 GPU server. Otherwise, high-performance CPU are suitable, 8 cores at least (the more, the better).
*** Disk: speed is only important when loading model from disk, which only occurs at startup. SSDs are fine. 500 GB of space should be ok.
+
*** Disk: speed is only important when loading model from disk, which only occurs at startup. Ideally NVMe, classic SSDs are fine. 500 GB of space should be ok.
* New producer platform
+
 
 +
* '''New producer platform'''
 
** New server
 
** New server
 
*** Little web traffic, lots of image data (sometimes in huge PDFs or png)
 
*** Little web traffic, lots of image data (sometimes in huge PDFs or png)
 
*** mid-level cpu and ram, at least 2x4 Tb disks
 
*** mid-level cpu and ram, at least 2x4 Tb disks
* New content platform (wiki, blog) ??
+
 
** New server for different kind of tools (example: Zammad, MediaWiki, Wordpress, Nextcloud?)
+
* '''New content platform''' (wiki, blog...) ??
 +
** New server for different kind of tools (example: Zammad, MediaWiki, Wordpress, Matomo, Rserver?, Nextcloud?)
 
*** Little web traffic, not much disk needed (except for videos if needed?)
 
*** Little web traffic, not much disk needed (except for videos if needed?)
 
*** mid-level cpu and ram, 2x500 Gb SSD disks probably ok
 
*** mid-level cpu and ram, 2x500 Gb SSD disks probably ok
Line 35: Line 40:
 
=== Dev ===
 
=== Dev ===
  
* OFF + producer platform dev
+
* '''OFF + producer platform dev'''
 
** New server
 
** New server
 
*** mid level cpu and ram, at least 2x8 Tb disks
 
*** mid level cpu and ram, at least 2x8 Tb disks

Latest revision as of 14:59, 23 February 2023

Current infrastructure

Details on https://en.wiki.openfoodfacts.org/Infrastructure#Current_infrastructure

  • off1: frontend + backend for the web site and API
  • off2: mongodb + off1 backup
  • dev: dev server, robotoff, remote backup

Hardware needs 2020

Production

  • Frontend + backend for the web site and API
    • off1
  • MongoDB
    • off2
  • Robotoff
    • New server
      • Models are memory hungry, at least 32 GB of RAM is needed. Ideally, 64 GB.
      • Ideally, 1 GPU server. Otherwise, high-performance CPU are suitable, 8 cores at least (the more, the better).
      • Disk: speed is only important when loading model from disk, which only occurs at startup. Ideally NVMe, classic SSDs are fine. 500 GB of space should be ok.
  • New producer platform
    • New server
      • Little web traffic, lots of image data (sometimes in huge PDFs or png)
      • mid-level cpu and ram, at least 2x4 Tb disks
  • New content platform (wiki, blog...) ??
    • New server for different kind of tools (example: Zammad, MediaWiki, Wordpress, Matomo, Rserver?, Nextcloud?)
      • Little web traffic, not much disk needed (except for videos if needed?)
      • mid-level cpu and ram, 2x500 Gb SSD disks probably ok

Backup

  • New backup server or cloud storage

Dev

  • OFF + producer platform dev
    • New server
      • mid level cpu and ram, at least 2x8 Tb disks