979
edits
Changes
Jump to navigation
Jump to search
Line 125:
Line 125: − + + + + + + + + + + + + +
JSONL with DuckDB
=== JSONL export ===
=== JSONL export ===
JSONL is a huge file! It's not possible to play with it with common editors or common tools. But there is some command line tools that allows interesting things, like [https://stedolan.github.io/jq/manual/v1.6/ jq].
JSONL is a huge file! It's not possible to play with it with common editors or common tools. There is some command line tools that allows interesting things, like [https://duckdb.org/ DuckDB] or [https://stedolan.github.io/jq/manual/v1.6/ jq].
==== DuckDB ====
DuckDB recognizes the compressed JSONL file, there is no need to uncompress it. You can start playing with JSONL export with very few commands.
At first create a DuckDB file:
$ duckdb off.db
Then you can start playing with SQL commands.
SELECT code, product_name, countries_en FROM read_ndjson('openfoodfacts-products.jsonl.gz', ignore_errors=True)
WHERE countries_en LIKE "%Germany%" LIMIT 10;
The advantages of DuckDB over the other solutions are that it is straightfoward and very fast.
If you want to go further, you can check the [https://blog.openfoodfacts.org/en/news/food-transparency-in-the-palm-of-your-hand-explore-the-largest-open-food-database-using-duckdb-%f0%9f%a6%86x%f0%9f%8d%8a blog article from Jeremy Arancio].
==== jq ====
==== jq ====