Line 23: |
Line 23: |
| ** Python is widely adopted (one of the main languages taught at school). | | ** Python is widely adopted (one of the main languages taught at school). |
| ** It scales well. | | ** It scales well. |
− | ** Part of the team knows Python. | + | ** Part of the team is already using Python. |
| * Building an API from scratch would have been too long, so choose the FastAPI framework which integrates interesting benefits: | | * Building an API from scratch would have been too long, so choose the FastAPI framework which integrates interesting benefits: |
− | ** [https://www.tutlinks.com/fastapi-with-postgresql-crud-async/ Building an API with FastAPI is very easy]: it's possible to build a simple API with its generated dynamic documentation in a few hours. | + | **[https://www.tutlinks.com/fastapi-with-postgresql-crud-async/ Building an API with FastAPI is very easy]: it's possible to build a simple API with its self generated documentation (openapi standard). |
| ** Said to be fast: https://www.techempower.com/ benchmarks are showing it is ~3 times faster than very popular Python framework such as Django or Flask. | | ** Said to be fast: https://www.techempower.com/ benchmarks are showing it is ~3 times faster than very popular Python framework such as Django or Flask. |
| ** FastAPI can be run with or without a web server. | | ** FastAPI can be run with or without a web server. |
| + | ** RESTful compliance is largely handled by FastAPI itself. |
| | | |
| ==== Database ==== | | ==== Database ==== |
Line 35: |
Line 36: |
| * It could lead to 10 times more records as the total number of products gathered by Open Food Facts. | | * It could lead to 10 times more records as the total number of products gathered by Open Food Facts. |
| | | |
− | The data model simplicity lead us to choose a classical RDBMS known for its scalability: PostGRESQL. Postgre has also interesting advantages and features which can be useful for Folksonomy Engine (fully open source, transactions, stored procedures, triggers, JSON...). | + | The data model simplicity lead us to choose a classical RDBMS known for its scalability: PostgreSQL. PostgreSQL has also interesting advantages and features which can be useful for Folksonomy Engine (fully open source, transactions, stored procedures, triggers, JSON...). |
| | | |
| ==== Data model ==== | | ==== Data model ==== |
− | For performance concerns, we separated the data model implementation into two tables: | + | For performance concerns, we separated the data model implementation into two parts : |
| | | |
− | * one small table deals with the live data | + | * one small table deals with the live data (current record versions) |
| * the other one stores all the past versions of the records. | | * the other one stores all the past versions of the records. |
| + | Triggers at PostgreSQL level are used to maintain integrity between the current and archived versions of the records. |
| | | |
| === Frontend === | | === Frontend === |
Line 53: |
Line 55: |
| === Tests === | | === Tests === |
| If not specified, all our tests are made on a Raspberry Pi 4: because of the very low price of it, hardware -- and tests -- are easily reproducible. | | If not specified, all our tests are made on a Raspberry Pi 4: because of the very low price of it, hardware -- and tests -- are easily reproducible. |
| + | |
| + | All tests have been run with: |
| + | * a standard installation, as [https://github.com/openfoodfacts/folksonomy_api/blob/main/INSTALL.md described here] |
| + | * no optimization at all |
| + | * <code>uvicorn</code> launched with 2 workers. |
| + | $ uvicorn folksonomy.api:app --reload --host 192.168.0.42 --workers 2 |
| + | |
| + | We used the classical <code>ab</code> test tool from the Apache Foundation, and a custom script to test new key/value injection (see bellow). |
| | | |
| Testing 10,000 queries to <code>/ping</code> (includes a database request) give the following results: | | Testing 10,000 queries to <code>/ping</code> (includes a database request) give the following results: |
Line 59: |
Line 69: |
| Requests per second: 307.11 [#/sec] (mean) | | Requests per second: 307.11 [#/sec] (mean) |
| Time per request: 32.562 [ms] (mean) | | Time per request: 32.562 [ms] (mean) |
| + | Testing 10,000 queries to a particular key/value gives the following results: |
| + | $ ab -n 10000 -c 10 http://192.168.0.42:8000/product/6389599748279 |
| + | |
| + | Requests per second: 227.45 [#/sec] (mean) |
| + | Time per request: 43.966 [ms] (mean) |
| Publishing 2,000 new key/value pairs on 2,000 different products give the following result: | | Publishing 2,000 new key/value pairs on 2,000 different products give the following result: |
| $ time ./fe_test.sh | | $ time ./fe_test.sh |