6,701
edits
Changes
Jump to navigation
Jump to search
Line 229:
Line 229: − − =Methodology to extract data for European Food Establishments= − ==General== − ===Build URL Data Files=== − * All root directories for EU Approval list by country can be found here https://github.com/openfoodfacts/eu-food-data − * Build a url listing all needed url. It is best to use FR template (FR-urls.txt to be uploaded), to identify missing data. It is also important to match country specific denomination with OFF taxonomy for EU Approval Section. − − ===Build Raw CSV/TXT files=== − *Several formats are used in EU countries. A specific approach is needed for each of them. Refer below for details for each country. − *For non text-PDF files, use Tesseract and this Python script https://github.com/virantha/pypdfocr to get text PDF − *For text PDF files, use Tabula and use the command line tool to create scripts https://github.com/tabulapdf/tabula-extractor/wiki/Using-the-command-line-tabula-extractor-tool − ===Get Templated CSV/TXT file=== − *Cleanup is needed to ensure data accuracy − *an XLS file is available for some countries (eg France) to speed up the process − *Section needs to be normalize to make comparisons between countries − − ===Geocode=== − *The best way is to use the geopy library to use nominatim in a Python script − *To check geocoding accuracy, it is best to make a pivot table of all cities in the target country, and to add a lookup function in the main data file. Adding the name of the country in the address usually lead to better results. − *Local Authorities List in UK http://localweblist.net/ − ===Polishing files=== − *As a company has one approval codes but can have it for several section under the EU regulation (slaughterhouse, warehouse, fishery products - Freezing vessel ...), you must used a pivot table and some cleaning to get a file with the following columns: name of the establishment, approval code, type (concatenation of the Section name under which the company is approved), one colum for each EU section, adresse, latitude, longitude − *The "type" column is used for search query − *To get a nice vizualisation and make corrections, use http://blog.perrygeo.net/2013/09/30/leaflet-simple-csv/ to make a map of all listed companies. You can filter company on their section by simply using the search field − − ===Find Additional Data on companies=== − *Infogreffe − *https://opencorporates.com/ − − ==France== − *I just build a script that takes all the french agreement info from Agriculture Ministry and concatenate them in one file. Next step is to do the same for UK. The step after that is to cleverly agregate the duplicates (some companies have several health agreements under the same agreement number) − https://github.com/openfoodfacts/eu-food-data/blob/master/scripts/FR-script.py − This script use this file to get the list of URL to retrieve https://github.com/openfoodfacts/eu-food-data/blob/master/fr/urls-fr.txt − *If nominatim limits queries, you can use data files from wikipedia for lat/lng for each zipcode https://www.data.gouv.fr/fr/datasets/listes-des-communes-geolocalisees-par-regions-departements-circonscriptions-nd/ − − ==UK== − *As UK is divided in 4 regions (Ireland, England, Wales and Scotland) and because they have different file format, we use a 3-file script − **https://github.com/openfoodfacts/eu-food-data/blob/master/scripts/UK-urls.txt => all UK urls − **https://github.com/openfoodfacts/eu-food-data/blob/master/scripts/UK-methods.txt => list which method to use depending on the file type − **https://github.com/openfoodfacts/eu-food-data/blob/master/scripts/UK-script.py => the script itself − *Source: Food Standards Agency - http://www.food.gov.uk/business-industry/meat/audit − − ==DE== − *@vince has performed a first extraction based on file in https://github.com/openfoodfacts/eu-food-data/tree/master/de with the help of the following script − cat export.csv | awk -F';' '{print "\"" $4 "\"; Deutschland"}' | sed "1d" | sort -u | iconv -f ISO-8859-15 -t UTF-8 > to_geocode.csv − ==NL== − *Data from NWA (Netherlands Food & Consumer Product Safety Authority) − **English (PDF): https://english-acc.nvwa.nl/documents/document/approved-establishments1/animal-byproducts − *A nice map with all the companies with approval from Netherlands under the EU regulation: http://www.alteconomics.fr/Leaflet − − ==Inspiration== − *http://free.sourcemap.com/ − − − − [[Category:Food codes]] − [[fr:Projet:Codes_propriétaires]]
Food Traceability Codes (view source)
Revision as of 08:51, 5 February 2016
, 08:51, 5 February 2016moving it elsewhere
* https://github.com/openfoodfacts/bio-codes
* https://github.com/openfoodfacts/bio-codes
* Integrated as sub-labels
* Integrated as sub-labels