Changes

Jump to navigation Jump to search
CSV export details
Line 1: Line 1: −
Open Food Facts data is released as Open Data: it can be reused freely by anyone, under the Open Database License (ODBL).
+
[[Category:Reuse]]
 +
Open Food Facts data is released as Open Data: it can be reused freely by anyone, under the Open Database License (ODBL). While this page is related to practical reuse, you must really be aware of [[ODBL License|rights and duties provided by the Open Database]] License (ODBL).
    
== Where is the data? ==
 
== Where is the data? ==
Line 7: Line 8:  
Then use the advanced search. The Open Food Facts advanced search feature allows to download selections of the data. See: https://world.openfoodfacts.org/cgi/search.pl
 
Then use the advanced search. The Open Food Facts advanced search feature allows to download selections of the data. See: https://world.openfoodfacts.org/cgi/search.pl
   −
When you search is done, you will be able to download the selection in '''CSV or Excel format''', just give a try!
+
When your search is done, you will be able to download the selection in '''CSV or Excel format''', just give a try!
    
=== Looking for the whole database? ===
 
=== Looking for the whole database? ===
 
The whole database can be downloaded at https://world.openfoodfacts.org/data
 
The whole database can be downloaded at https://world.openfoodfacts.org/data
   −
It's very big. Open Food Facts hosts more than 1,200,000 products (as of April 2020). So you will probably need skills to reuse the data.
+
It's very big. Open Food Facts hosts more than 1,400,000 products (as of July 2020). So you will probably need skills to reuse the data.
   −
You'll be able to find there different kinds of data.
+
You'll be able to find here different kinds of data.
    
==== The MongoDB daily export ====
 
==== The MongoDB daily export ====
 
It represents the most complete data; it's very big and you have to know how to deal with MongoDB.
 
It represents the most complete data; it's very big and you have to know how to deal with MongoDB.
   −
==== The jsonl daily export ====
+
==== The JSONL daily export ====
While still undocumented, there is a daily export of the whole database in jsonl format. It represents the same data as the MongoDB export. It's very big! More than 17GB uncompressed.
+
While still undocumented, there is a daily export of the whole database in jsonl format. It represents the same data as the MongoDB export. It's very big! More than 14GB uncompressed.
    
You can find it at https://static.openfoodfacts.org/data/openfoodfacts-products.jsonl.gz
 
You can find it at https://static.openfoodfacts.org/data/openfoodfacts-products.jsonl.gz
    
==== The CSV daily export ====
 
==== The CSV daily export ====
It represents a subset of the database but it is generally fitted to the majority of usages. It's a 2.3GB file (as of April 2020), so it can't be opened by Libre Office or Excel with an 8GB machine.
+
It contains all the products, but with a subset of the database fields. [https://world.openfoodfacts.org/data/data-fields.txt This subset is very large] and include main characteristics (EAN, name, brand...), many tags (such as categories, origins, labels, packaging...), ingredients and nutrition facts. Thus, it is generally fitted to the majority of usages. It's a 2.3GB file (as of April 2020), so it can't be opened by Libre Office or Excel with an 8GB machine.
    
== How to reuse? ==
 
== How to reuse? ==
Line 77: Line 78:  
For people who have R stat skills, there are [https://www.kaggle.com/openfoodfacts/world-food-facts/kernels?sortBy=hotness&group=everyone&pageSize=20&datasetId=20&language=R more than 50 notebooks from Kaggle community].
 
For people who have R stat skills, there are [https://www.kaggle.com/openfoodfacts/world-food-facts/kernels?sortBy=hotness&group=everyone&pageSize=20&datasetId=20&language=R more than 50 notebooks from Kaggle community].
   −
=== jsonl export ===
+
=== JSONL export ===
jsonl is a huge file! It's not possible to play with it with common editors or common tools. But there is some command line tools that allows interesting things, like [https://stedolan.github.io/jq/manual/v1.6/ jq].
+
JSONL is a huge file! It's not possible to play with it with common editors or common tools. But there is some command line tools that allows interesting things, like [https://stedolan.github.io/jq/manual/v1.6/ jq].
    
==== jq ====
 
==== jq ====
* start decompress the file (be carreful => 17GB after decompression):
+
* start decompress the file (be carreful => 14GB after decompression):
 
  $ gunzip openfoodfacts-products.jsonl.gz
 
  $ gunzip openfoodfacts-products.jsonl.gz
 
* work on a small subset to test. E.g. for 100 products:
 
* work on a small subset to test. E.g. for 100 products:
Line 91: Line 92:  
  $ cat small.jsonl | jq -r .code # print all products' codes.
 
  $ cat small.jsonl | jq -r .code # print all products' codes.
   −
  $ cat small.jsonl | jq -r '[.code,.product_name] | @csv' # output a CSV file containing code,product_name
+
  $ cat small.jsonl | jq -r '[.code,.product_name] | @csv' # output CSV data containing code,product_name
 +
 
 +
Then you can try on the whole database:
 +
$ cat openfoodfacts-products.jsonl | jq -r '[.code,.product_name] | @csv' > names.csv # output CSV file (name.csv) containing all products with code,product_name
 +
 
 +
If you don't have enough disk place to uncompress the .gz file, you can use zcat directly on the compressed file. Example:
 +
$ zcat openfoodfacts-products.jsonl.gz | jq -r '[.code,.product_name] | @csv' # output CSV data containing code,product_name

Navigation menu