Changes

Jump to navigation Jump to search
JSONL with DuckDB
Line 125: Line 125:     
=== JSONL export ===
 
=== JSONL export ===
JSONL is a huge file! It's not possible to play with it with common editors or common tools. But there is some command line tools that allows interesting things, like [https://stedolan.github.io/jq/manual/v1.6/ jq].
+
JSONL is a huge file! It's not possible to play with it with common editors or common tools. There is some command line tools that allows interesting things, like [https://duckdb.org/ DuckDB] or [https://stedolan.github.io/jq/manual/v1.6/ jq].
 +
 
 +
==== DuckDB ====
 +
DuckDB recognizes the compressed JSONL file, there is no need to uncompress it. You can start playing with JSONL export with very few commands.
 +
 
 +
At first create a DuckDB file:
 +
$ duckdb off.db
 +
Then you can start playing with SQL commands.
 +
SELECT code, product_name, countries_en FROM read_ndjson('openfoodfacts-products.jsonl.gz', ignore_errors=True)
 +
WHERE countries_en LIKE "%Germany%" LIMIT 10;
 +
The advantages of DuckDB over the other solutions are that it is straightfoward and very fast.
 +
 
 +
If you want to go further, you can check the [https://blog.openfoodfacts.org/en/news/food-transparency-in-the-palm-of-your-hand-explore-the-largest-open-food-database-using-duckdb-%f0%9f%a6%86x%f0%9f%8d%8a blog article from Jeremy Arancio].
    
==== jq ====
 
==== jq ====

Navigation menu