Changes

Jump to navigation Jump to search
CSV export details
Line 1: Line 1: −
Open Food Facts data is released as Open Data: it can be reused freely by anyone, under the Open Database License (ODBL).
+
[[Category:Reuse]]
 +
Open Food Facts data is released as Open Data: it can be reused freely by anyone, under the Open Database License (ODBL). While this page is related to practical reuse, you must really be aware of [[ODBL License|rights and duties provided by the Open Database]] License (ODBL).
    
== Where is the data? ==
 
== Where is the data? ==
Line 7: Line 8:  
Then use the advanced search. The Open Food Facts advanced search feature allows to download selections of the data. See: https://world.openfoodfacts.org/cgi/search.pl
 
Then use the advanced search. The Open Food Facts advanced search feature allows to download selections of the data. See: https://world.openfoodfacts.org/cgi/search.pl
   −
When you search is done, you will be able to download the selection in '''CSV or Excel format''', just give a try!
+
When your search is done, you will be able to download the selection in '''CSV or Excel format''', just give a try!
    
=== Looking for the whole database? ===
 
=== Looking for the whole database? ===
 
The whole database can be downloaded at https://world.openfoodfacts.org/data
 
The whole database can be downloaded at https://world.openfoodfacts.org/data
   −
It's very big. Open Food Facts hosts more than 1,200,000 products (as of April 2020). So you will probably need skills to reuse the data.
+
It's very big. Open Food Facts hosts more than 1,400,000 products (as of July 2020). So you will probably need skills to reuse the data.
   −
You'll be able to find there different kinds of data.
+
You'll be able to find here different kinds of data.
    
==== The MongoDB daily export ====
 
==== The MongoDB daily export ====
 
It represents the most complete data; it's very big and you have to know how to deal with MongoDB.
 
It represents the most complete data; it's very big and you have to know how to deal with MongoDB.
 +
 +
==== The JSONL daily export ====
 +
While still undocumented, there is a daily export of the whole database in jsonl format. It represents the same data as the MongoDB export. It's very big! More than 14GB uncompressed.
 +
 +
You can find it at https://static.openfoodfacts.org/data/openfoodfacts-products.jsonl.gz
    
==== The CSV daily export ====
 
==== The CSV daily export ====
It represents a subset of the database but it is generally fitted to the majority of usages. It's a 2.3GB file (as of April 2020), so it can't be opened by Libre Office or Excel with an 8GB machine.
+
It contains all the products, but with a subset of the database fields. [https://world.openfoodfacts.org/data/data-fields.txt This subset is very large] and include main characteristics (EAN, name, brand...), many tags (such as categories, origins, labels, packaging...), ingredients and nutrition facts. Thus, it is generally fitted to the majority of usages. It's a 2.3GB file (as of April 2020), so it can't be opened by Libre Office or Excel with an 8GB machine.
    
== How to reuse? ==
 
== How to reuse? ==
Line 71: Line 77:  
==== R stat ====
 
==== R stat ====
 
For people who have R stat skills, there are [https://www.kaggle.com/openfoodfacts/world-food-facts/kernels?sortBy=hotness&group=everyone&pageSize=20&datasetId=20&language=R more than 50 notebooks from Kaggle community].
 
For people who have R stat skills, there are [https://www.kaggle.com/openfoodfacts/world-food-facts/kernels?sortBy=hotness&group=everyone&pageSize=20&datasetId=20&language=R more than 50 notebooks from Kaggle community].
 +
 +
=== JSONL export ===
 +
JSONL is a huge file! It's not possible to play with it with common editors or common tools. But there is some command line tools that allows interesting things, like [https://stedolan.github.io/jq/manual/v1.6/ jq].
 +
 +
==== jq ====
 +
* start decompress the file (be carreful => 14GB after decompression):
 +
$ gunzip openfoodfacts-products.jsonl.gz
 +
* work on a small subset to test. E.g. for 100 products:
 +
$ head -n 100 openfoodfacts-products.jsonl > small.jsonl
 +
 +
You can start playing with jq. Here are examples.
 +
$ cat small.jsonl | jq . # print all file in JSON format
 +
 +
$ cat small.jsonl | jq -r .code # print all products' codes.
 +
 +
$ cat small.jsonl | jq -r '[.code,.product_name] | @csv' # output CSV data containing code,product_name
 +
 +
Then you can try on the whole database:
 +
$ cat openfoodfacts-products.jsonl | jq -r '[.code,.product_name] | @csv' > names.csv # output CSV file (name.csv) containing all products with code,product_name
 +
 +
If you don't have enough disk place to uncompress the .gz file, you can use zcat directly on the compressed file. Example:
 +
$ zcat openfoodfacts-products.jsonl.gz | jq -r '[.code,.product_name] | @csv' # output CSV data containing code,product_name

Navigation menu