Difference between revisions of "Infrastructure Planning"
Jump to navigation
Jump to search
Raphael0202 (talk | contribs) (Add robotoff requirements) |
|||
| (7 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
| + | [[Category:Infrastructure]] | ||
| + | [[Category:Roadmap]] | ||
| + | Welcome :-) <br> | ||
| + | This page describes the hardware/software infrastructure for the Open Food Facts, Open Beauty Facts, Open Pet Food Facts and Open Products Facts projects, as well as all the complementary deployments.<br> | ||
| + | '''It is not updated anymore, as we have moved to GitHub for planning, postmortems…'''<br> | ||
| + | Please check our coordination repo: https://github.com/openfoodfacts/openfoodfacts-infrastructure<br> | ||
| + | Note, we also have an #infrastructure channel on Slack, and a monthly video call. We can arrange ad-hoc calls if you'd like to get started faster. | ||
| + | |||
== Current infrastructure == | == Current infrastructure == | ||
| Line 7: | Line 15: | ||
* dev: dev server, robotoff, remote backup | * dev: dev server, robotoff, remote backup | ||
| − | == Hardware needs | + | == Hardware needs 2020 == |
=== Production === | === Production === | ||
| Line 13: | Line 21: | ||
* Frontend + backend for the web site and API | * Frontend + backend for the web site and API | ||
** off1 | ** off1 | ||
| + | |||
* MongoDB | * MongoDB | ||
** off2 | ** off2 | ||
| − | * Robotoff | + | |
| + | * '''Robotoff''' | ||
** New server | ** New server | ||
| − | *** Models are memory hungry, at least 32 GB of RAM is needed. | + | *** Models are memory hungry, at least 32 GB of RAM is needed. Ideally, 64 GB. |
| − | *** GPU | + | *** Ideally, 1 GPU server. Otherwise, high-performance CPU are suitable, 8 cores at least (the more, the better). |
| − | * New producer platform | + | *** Disk: speed is only important when loading model from disk, which only occurs at startup. Ideally NVMe, classic SSDs are fine. 500 GB of space should be ok. |
| + | |||
| + | * '''New producer platform''' | ||
** New server | ** New server | ||
*** Little web traffic, lots of image data (sometimes in huge PDFs or png) | *** Little web traffic, lots of image data (sometimes in huge PDFs or png) | ||
*** mid-level cpu and ram, at least 2x4 Tb disks | *** mid-level cpu and ram, at least 2x4 Tb disks | ||
| − | * New content platform (wiki, blog) ?? | + | |
| − | ** New server for different kind of tools (example: Zammad, MediaWiki, Wordpress, Nextcloud?) | + | * '''New content platform''' (wiki, blog...) ?? |
| + | ** New server for different kind of tools (example: Zammad, MediaWiki, Wordpress, Matomo, Rserver?, Nextcloud?) | ||
*** Little web traffic, not much disk needed (except for videos if needed?) | *** Little web traffic, not much disk needed (except for videos if needed?) | ||
*** mid-level cpu and ram, 2x500 Gb SSD disks probably ok | *** mid-level cpu and ram, 2x500 Gb SSD disks probably ok | ||
| Line 34: | Line 47: | ||
=== Dev === | === Dev === | ||
| − | * OFF + producer platform dev | + | * '''OFF + producer platform dev''' |
** New server | ** New server | ||
*** mid level cpu and ram, at least 2x8 Tb disks | *** mid level cpu and ram, at least 2x8 Tb disks | ||
Latest revision as of 06:55, 19 August 2024
Welcome :-)
This page describes the hardware/software infrastructure for the Open Food Facts, Open Beauty Facts, Open Pet Food Facts and Open Products Facts projects, as well as all the complementary deployments.
It is not updated anymore, as we have moved to GitHub for planning, postmortems…
Please check our coordination repo: https://github.com/openfoodfacts/openfoodfacts-infrastructure
Note, we also have an #infrastructure channel on Slack, and a monthly video call. We can arrange ad-hoc calls if you'd like to get started faster.
Current infrastructure
Details on https://en.wiki.openfoodfacts.org/Infrastructure#Current_infrastructure
- off1: frontend + backend for the web site and API
- off2: mongodb + off1 backup
- dev: dev server, robotoff, remote backup
Hardware needs 2020
Production
- Frontend + backend for the web site and API
- off1
- MongoDB
- off2
- Robotoff
- New server
- Models are memory hungry, at least 32 GB of RAM is needed. Ideally, 64 GB.
- Ideally, 1 GPU server. Otherwise, high-performance CPU are suitable, 8 cores at least (the more, the better).
- Disk: speed is only important when loading model from disk, which only occurs at startup. Ideally NVMe, classic SSDs are fine. 500 GB of space should be ok.
- New server
- New producer platform
- New server
- Little web traffic, lots of image data (sometimes in huge PDFs or png)
- mid-level cpu and ram, at least 2x4 Tb disks
- New server
- New content platform (wiki, blog...) ??
- New server for different kind of tools (example: Zammad, MediaWiki, Wordpress, Matomo, Rserver?, Nextcloud?)
- Little web traffic, not much disk needed (except for videos if needed?)
- mid-level cpu and ram, 2x500 Gb SSD disks probably ok
- New server for different kind of tools (example: Zammad, MediaWiki, Wordpress, Matomo, Rserver?, Nextcloud?)
Backup
- New backup server or cloud storage
Dev
- OFF + producer platform dev
- New server
- mid level cpu and ram, at least 2x8 Tb disks
- New server