Folksonomy Engine/Technical specifications

Requirements

• Scalability
  • Number of property/value pairs could be huge: potentially millions in a few years; potentially dozens of millions in 5 years.
  • Write and read must be fast and scale to, at least, thousands of requests per second.
• Development
  • Data model should be as simple as possible (really).
  • As en open source software Open Food Facts appreciate contributions: the technical stack should be widely known to maximize the chance that other developers participate.
  • Open Food Facts permanent dev team is small: the technical stack should be known by the team.
  • Open Food Facts tries to choose improved technologies and mature standards.
  • Backend and frontend should be clearly separated: it helps data reusability; it allows backend and frontend devs to concentrate on their own work.
• Usages
  • Data should be easy to reuse by developers.
    • A dynamic API documentation should be really helpful.

Technical choices

Backend

• The whole Open Food Facts infrastructure is based on Linux Debian, which is efficient and widely known.
• We choose Python as the backend programming language:
  • Python is widely adopted (one of the main languages taught at school).
  • It scales well.
  • Part of team knows Python.
• Building an API from scratch would have been too long so choose the FastAPI framework which integrates interesting benefits:
  • Building an API with FastAPI is very easy: it's possible to build a simple API with its generated dynamic documentation in a few hours.
  • Said to be fast: https://www.techempower.com/ benchmarks are showing it is ~3 times faster than very porpular Python framework such as Django or Flask.
  • FastAPI can be run with or without a web server.

Database

Folksonomy Engine has a very different goals and requirements than Open Food Facts backend (Perl + MongoDB):

• The data model should be very simple.
• It could lead to 10 times more records as the total number of products gathered by Open Food Facts.

The data model simplicity lead us to choose a classical RDBMS known for its scalability: PostGRESQL. Postgre has also interesting advantages and features which can be useful for Folksonomy Engine (fully open source, transactions, stored procedures, triggers, JSON...).

Data model

For performance concerns, we separated the data model implementation into two tables:

• one small table deals with the live data
• the other one stores all the past versions of the records.

Frontend

For its first version, at least (it could change), the whole frontend will be developed in Javascript.