Difference between revisions of "Infrastructure Planning"

From Open Food Facts wiki
Jump to navigation Jump to search
 
(5 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
[[Category:Infrastructure]]
 +
 +
Welcome :-) <br>
 +
This page describes the hardware/software infrastructure for the Open Food Facts, Open Beauty Facts, Open Pet Food Facts and Open Products Facts projects, as well as all the complementary deployments.<br>
 +
'''It is not updated anymore, as we have moved to GitHub for planning, postmortems…'''<br>
 +
Please check our coordination repo: https://github.com/openfoodfacts/openfoodfacts-infrastructure<br>
 +
Note, we also have an #infrastructure channel on Slack, and a monthly video call. We can arrange ad-hoc calls if you'd like to get started faster.
 +
 
== Current infrastructure ==
 
== Current infrastructure ==
  
Line 7: Line 15:
 
* dev: dev server, robotoff, remote backup
 
* dev: dev server, robotoff, remote backup
  
== Hardware needs 2019 ==
+
== Hardware needs 2020 ==
  
 
=== Production ===
 
=== Production ===
Line 13: Line 21:
 
* Frontend + backend for the web site and API
 
* Frontend + backend for the web site and API
 
** off1
 
** off1
 +
 
* MongoDB
 
* MongoDB
 
** off2
 
** off2
* Robotoff
+
 
 +
* '''Robotoff'''
 
** New server
 
** New server
*** Models are memory hungry, at least 32 GB of RAM is needed.
+
*** Models are memory hungry, at least 32 GB of RAM is needed. Ideally, 64 GB.
*** GPU servers deliver the best performances for prediction, but are rather expensive. Given the current load, CPU servers are enough.
+
*** Ideally, 1 GPU server. Otherwise, high-performance CPU are suitable, 8 cores at least (the more, the better).
*** Disk: speed is only important when loading model from disk, which only occurs at startup. SSDs are fine. 500 GB of space should be ok.
+
*** Disk: speed is only important when loading model from disk, which only occurs at startup. Ideally NVMe, classic SSDs are fine. 500 GB of space should be ok.
* New producer platform
+
 
 +
* '''New producer platform'''
 
** New server
 
** New server
 
*** Little web traffic, lots of image data (sometimes in huge PDFs or png)
 
*** Little web traffic, lots of image data (sometimes in huge PDFs or png)
 
*** mid-level cpu and ram, at least 2x4 Tb disks
 
*** mid-level cpu and ram, at least 2x4 Tb disks
* New content platform (wiki, blog) ??
+
 
** New server for different kind of tools (example: Zammad, MediaWiki, Wordpress, Nextcloud?)
+
* '''New content platform''' (wiki, blog...) ??
 +
** New server for different kind of tools (example: Zammad, MediaWiki, Wordpress, Matomo, Rserver?, Nextcloud?)
 
*** Little web traffic, not much disk needed (except for videos if needed?)
 
*** Little web traffic, not much disk needed (except for videos if needed?)
 
*** mid-level cpu and ram, 2x500 Gb SSD disks probably ok
 
*** mid-level cpu and ram, 2x500 Gb SSD disks probably ok
Line 35: Line 47:
 
=== Dev ===
 
=== Dev ===
  
* OFF + producer platform dev
+
* '''OFF + producer platform dev'''
 
** New server
 
** New server
 
*** mid level cpu and ram, at least 2x8 Tb disks
 
*** mid level cpu and ram, at least 2x8 Tb disks

Latest revision as of 07:17, 2 June 2024


Welcome :-)
This page describes the hardware/software infrastructure for the Open Food Facts, Open Beauty Facts, Open Pet Food Facts and Open Products Facts projects, as well as all the complementary deployments.
It is not updated anymore, as we have moved to GitHub for planning, postmortems…
Please check our coordination repo: https://github.com/openfoodfacts/openfoodfacts-infrastructure
Note, we also have an #infrastructure channel on Slack, and a monthly video call. We can arrange ad-hoc calls if you'd like to get started faster.

Current infrastructure

Details on https://en.wiki.openfoodfacts.org/Infrastructure#Current_infrastructure

  • off1: frontend + backend for the web site and API
  • off2: mongodb + off1 backup
  • dev: dev server, robotoff, remote backup

Hardware needs 2020

Production

  • Frontend + backend for the web site and API
    • off1
  • MongoDB
    • off2
  • Robotoff
    • New server
      • Models are memory hungry, at least 32 GB of RAM is needed. Ideally, 64 GB.
      • Ideally, 1 GPU server. Otherwise, high-performance CPU are suitable, 8 cores at least (the more, the better).
      • Disk: speed is only important when loading model from disk, which only occurs at startup. Ideally NVMe, classic SSDs are fine. 500 GB of space should be ok.
  • New producer platform
    • New server
      • Little web traffic, lots of image data (sometimes in huge PDFs or png)
      • mid-level cpu and ram, at least 2x4 Tb disks
  • New content platform (wiki, blog...) ??
    • New server for different kind of tools (example: Zammad, MediaWiki, Wordpress, Matomo, Rserver?, Nextcloud?)
      • Little web traffic, not much disk needed (except for videos if needed?)
      • mid-level cpu and ram, 2x500 Gb SSD disks probably ok

Backup

  • New backup server or cloud storage

Dev

  • OFF + producer platform dev
    • New server
      • mid level cpu and ram, at least 2x8 Tb disks