Line 136: |
Line 136: |
| | | |
| === MongoDB dump === | | === MongoDB dump === |
− | The MongoDB dump needs to be reused with MongoDB. It allows building a full replication of the Open Food Facts database and use MongoDB for selecting, filtering and exporting data. Using MongoDB allows faster manipulations compared to the other methods. | + | The [https://world.openfoodfacts.org/data MongoDB dump] needs to be reused with MongoDB. It allows building a full replication of the Open Food Facts database and use MongoDB for selecting, filtering and exporting data. Using MongoDB allows faster manipulations compared to the other methods. |
| | | |
| First, you '''need a running MongoDB installation'''. Open Food Facts is using MongoDB 4.4. It has been reported that prior version should not work for Open Food Facts dump. | | First, you '''need a running MongoDB installation'''. Open Food Facts is using MongoDB 4.4. It has been reported that prior version should not work for Open Food Facts dump. |
Line 201: |
Line 201: |
| # Official csv export fields are coming from @export_fields variable in /lib/ProductOpener/Config_off.pm | | # Official csv export fields are coming from @export_fields variable in /lib/ProductOpener/Config_off.pm |
| mongoexport -d off -c products --type=csv --fieldFile official_csv_export_fields.txt -q '{"countries_tags": "en:germany"}' --limit 5 --quiet | | mongoexport -d off -c products --type=csv --fieldFile official_csv_export_fields.txt -q '{"countries_tags": "en:germany"}' --limit 5 --quiet |
| + | |
| + | </pre> |
| + | |
| + | ==== List all fields used in the database ==== |
| + | <pre> |
| + | # Open Food Facts database contains hundreds of fields. |
| + | |
| + | # An easy way to list them all is to use "variety" Schema Analyzer: |
| + | # https://github.com/variety/variety |
| + | |
| + | # 1. Install "variety" |
| + | git clone https://github.com/variety/variety.git |
| + | |
| + | # 2. Use it |
| + | cd ./variety |
| + | |
| + | # Analyzing can be very long (hours). You can restrict the analysis to a small number |
| + | time mongo off --eval "var collection = 'products', limit = 1000" variety.js > off_schema_1000.txt |
| + | # (17 s) |
| + | |
| + | time mongo off --eval "var collection = 'products', limit = 10000" variety.js > off_schema_10000.txt |
| + | # (3 minutes) |
| + | |
| + | time mongo off --eval "var collection = 'products', limit = 100000" variety.js > off_schema_100000.txt |
| + | # (75 minutes) |
| + | |
| + | time mongo off --eval "var collection = 'products'" variety.js > off_schema_all.txt |
| + | # (more than two days) |
| + | |
| | | |
| </pre> | | </pre> |