984
edits
Changes
Jump to navigation
Jump to search
Folksonomy Engine/Technical specifications (view source)
Revision as of 14:06, 14 September 2021
, 14:06, 14 September 2021First version (to be continued)
== Requirements ==
* Scalability
** Number of property/value pairs could be huge: potentially millions in a few years; potentially dozens of millions in 5 years.
** Write and read must be fast and scale to, at least, thousands of requests per second.
* Development
** Data model should be as simple as possible (really).
** As en open source software Open Food Facts appreciate contributions: the technical stack should be widely known to maximize the chance that other developers participate.
** Open Food Facts permanent dev team is small: the technical stack should be known by the team.
** Open Food Facts tries to choose improved technologies and mature standards.
** Backend and frontend should be clearly separated: it helps data reusability; it allows backend and frontend devs to concentrate on their own work.
* Usages
** Data should be easy to reuse by developers.
*** A dynamic API documentation should be really helpful.
== Technical choices ==
=== Backend ===
* The whole Open Food Facts infrastructure is based on Linux Debian, which is efficient and widely known.
* We choose Python as the backend programming language:
** Python is widely adopted (one of the main languages taught at school).
** It scales well.
** Part of team knows Python.
* Building an API from scratch would have been too long so choose the FastAPI framework which integrates interesting benefits:
** Building an API with FastAPI is very easy: it's possible to build a simple API with its generated dynamic documentation in a few hours.
** Said to be fast: https://www.techempower.com/ benchmarks are showing it is ~3 times faster than very porpular Python framework such as Django or Flask.
** FastAPI can be run with or without a web server.
==== Database ====
Folksonomy Engine has a very different goals and requirements than Open Food Facts backend (Perl + MongoDB):
* The data model should be very simple.
* It could lead to 10 times more records as the total number of products gathered by Open Food Facts.
The data model simplicity lead us to choose a classical RDBMS known for its scalability: PostGRESQL. Postgre has also interesting advantages and features which can be useful for Folksonomy Engine (fully open source, transactions, stored procedures, triggers, JSON...).
==== Data model ====
For performance concerns, we separated the data model implementation into two tables:
* one small table deals with the live data
* the other one stores all the past versions of the records.
=== Frontend ===
For its first version, at least (it could change), the whole frontend will be developed in Javascript.
* Scalability
** Number of property/value pairs could be huge: potentially millions in a few years; potentially dozens of millions in 5 years.
** Write and read must be fast and scale to, at least, thousands of requests per second.
* Development
** Data model should be as simple as possible (really).
** As en open source software Open Food Facts appreciate contributions: the technical stack should be widely known to maximize the chance that other developers participate.
** Open Food Facts permanent dev team is small: the technical stack should be known by the team.
** Open Food Facts tries to choose improved technologies and mature standards.
** Backend and frontend should be clearly separated: it helps data reusability; it allows backend and frontend devs to concentrate on their own work.
* Usages
** Data should be easy to reuse by developers.
*** A dynamic API documentation should be really helpful.
== Technical choices ==
=== Backend ===
* The whole Open Food Facts infrastructure is based on Linux Debian, which is efficient and widely known.
* We choose Python as the backend programming language:
** Python is widely adopted (one of the main languages taught at school).
** It scales well.
** Part of team knows Python.
* Building an API from scratch would have been too long so choose the FastAPI framework which integrates interesting benefits:
** Building an API with FastAPI is very easy: it's possible to build a simple API with its generated dynamic documentation in a few hours.
** Said to be fast: https://www.techempower.com/ benchmarks are showing it is ~3 times faster than very porpular Python framework such as Django or Flask.
** FastAPI can be run with or without a web server.
==== Database ====
Folksonomy Engine has a very different goals and requirements than Open Food Facts backend (Perl + MongoDB):
* The data model should be very simple.
* It could lead to 10 times more records as the total number of products gathered by Open Food Facts.
The data model simplicity lead us to choose a classical RDBMS known for its scalability: PostGRESQL. Postgre has also interesting advantages and features which can be useful for Folksonomy Engine (fully open source, transactions, stored procedures, triggers, JSON...).
==== Data model ====
For performance concerns, we separated the data model implementation into two tables:
* one small table deals with the live data
* the other one stores all the past versions of the records.
=== Frontend ===
For its first version, at least (it could change), the whole frontend will be developed in Javascript.