Difference between revisions of "Food Traceability Codes"
Line 246: | Line 246: | ||
*To check geocoding accuracy, it is best to make a pivot table of all cities in the target country, and to add a lookup function in the main data file. Adding the name of the country in the address usually lead to better results. | *To check geocoding accuracy, it is best to make a pivot table of all cities in the target country, and to add a lookup function in the main data file. Adding the name of the country in the address usually lead to better results. | ||
*Local Authorities List in UK http://localweblist.net/ | *Local Authorities List in UK http://localweblist.net/ | ||
+ | ===Polishing files=== | ||
+ | *As a company has one approval codes but can have it for several section under the EU regulation (slaughterhouse, warehouse, fishery products - Freezing vessel ...), you must used a pivot table and some cleaning to get a file with the following columns: name of the establishment, approval code, type (concatenation of the Section name under which the company is approved), one colum for each EU section, adresse, latitude, longitude | ||
+ | *The "type" column is used for search query | ||
+ | *To get a nice vizualisation and make corrections, use http://blog.perrygeo.net/2013/09/30/leaflet-simple-csv/ to make a map of all listed companies. You can filter company on their section by simply using the search field | ||
===Find Additional Data on companies=== | ===Find Additional Data on companies=== |
Revision as of 12:06, 9 October 2015
This project aims to investigate packaging codes and all non-standard codes, usually printed on the packaging using a machine, such as batch numbers, the quality numbers ... and get the most information possible.
- Food Establishments Project: http://en.wiki.openfoodfacts.org/Project:Food_establishments
- Map made with this data: http://madenear.me/
- Files are here: https://bitbucket.org/openfoodfacts/product-opener/src/b2b3e84d40182d1162ec4060894a20f133a870b7/packager-codes/?at=master
- Script used to merge files:https://bitbucket.org/openfoodfacts/product-opener/src/3942937fec34ecda9bfc88a1c9cef0d55a4a9d27/cgi/update_packager_codes.pl?at=master&fileviewer=file-view-default
- Packaging Taxonomy and logos: http://en.wiki.openfoodfacts.org/Packaging_taxonomy http://en.wiki.openfoodfacts.org/Global_packaging_taxonomy_logos
Codes by country or region
France
Etiquetage des produits pré-emballés (Code Emballeur - EMB)
- Définition: produit constitué par une denrée alimentaire et l'emballage dans lequel elle a été conditionnée avant sa présentation à la vente. Les denrées préemballées sont des produits le plus souvent vendus dans les rayons de libre service et soumis à des règles strictes.
Deux grandes règles doivent être respectées :
- l'étiquetage doit faire figurer diverses informations qui renseignent objectivement le consommateur. Elles doivent être rédigées au moins en français
- l'étiquetage doit être loyal et ne doit pas induire le consommateur en erreur (composition du produit, origine…)
Les mentions obligatoires devant figurer sur les produits sont :
- la dénomination de vente qui définit le produit (ex. : confiture extra de framboises).
- l'origine, si son omission risque d'induire le consommateur en erreur (ex. : chorizo espagnol, fabriqué en France), on peut indiquer seulement le pays, ou le département, voire la commune (ex. : Fabriqué en Ardèche).
- la liste des ingrédients mis en œuvre par ordre d'importance décroissante (y compris additifs et arômes).
- La quantité de certains ingrédients, par exemple ceux mis en valeur sur l'étiquetage ou dans la dénomination de vente (ex. : gâteau aux fraises, pizza au jambon)
- la quantité nette du produit en volume (produit liquide) ou masse (autres produits). Si le produit est présenté dans un liquide, indication du poids net égoutté.
- la date de consommation pour les denrées périssables : Date Limite de Consommation (DLC) « à consommer jusqu'au… » ou pour les produits de conservation : Date Limite d'Utilisation Optimale (DLUO) « à consommer de préférence jusqu'au/avant le… ». Ces mentions doivent figurer en toutes lettres.
- l'identification du fabricant : nom, adresse. Si conditionnement par un prestataire, indiquer les coordonnées du centre d'emballage (précédé de « emb » (ex. : EMB A07555)).
- le numéro du lot de fabrication, à des fins de traçabilité. Cette mention est obligatoire depuis 2005. En revanche, la dénomination du lot est libre (ex.: lot 0607).
- le mode d'emploi, si la denrée nécessite un usage approprié ou une conservation spécifique (ex. : à conserver dans un endroit sec)
- la marque de salubrité (ou estampille sanitaire), qui identifie les établissements préparant, traitant, transformant, manipulant ou entreposant des denrées animales ou d'origine animale (marque obtenue auprès de la DSV)
Les mentions ou expressions réglementées :
- « biologique » : produits issus de l'Agriculture Biologique et certifiés par un organisme de contrôle
- « pur » : terme utilisable que pour certains produits (pur jus, pur porc)
- « campagne », « fermier », « paysan » : attention à la prise en compte du caractère fermier lié à l'exploitation agricole
- « artisanal » : produit qui doit être fabriqué par un artisan, inscrit au registre des métiers
- « maison » ou « fait maison » : que si le produit est préparé de manière non industrielle, sur le lieu de vente
- « à l'ancienne », « traditionnel » : seulement si produits fabriqués selon des usages anciens répertoriés. Pas d'additifs
- « naturel » : produit non transformé, non traité, tel qu'on le trouve à l'état naturel
- « du terroir » : signifie l'utilisation de matières premières obtenues à partir de procédés de fabrication, issues d'une aire géographique restreinte. Privilégier l'origine géographique ou le terme « fermier »
- Decryptage du code emballeur EMBxxxx: deux premiers chiffres = département, trois chiffres suivants = code INSEE de la commune, lettre = établissement si plusieurs sur la commune
Sources pour trouver les codes emballeurs
- agriculture / data.gouv.fr => Données sur les estampilles sanitaires récupérées et script codé (cf Github
- Usine Nouvelle -> liste des établissements par région: http://industrie.usinenouvelle.com/classement-industrie?region=24§eur=380
- Pagespro
- Infogreffe => recherche par Code Naf
- Franceagroalimentaire
- Pas de liste disponible de code emballeur (données protégées selon l'administration)
Some Leclerc products don't have this info
argumentation de Leclerc "c'est la centrale d'achat qui a "fait faire l'emplissage"", et l'administration (au moins dans le Val de Marne) accepte ce raisonnement.
Ventes de denrées non préemballées, vendues en vrac
Définition: Les denrées non préemballées sont les denrées présentées à la vente en vrac. C'est à dire présentées sans emballage à la vente et emballé à la demande du client au moment de l'achat (ex. : fruits ou légumes en vrac, baguette de pain, pâtisserie non emballée, traiteur…).
Une affichette (ou un écriteau) doit être placée à proximité de ce qui est proposé à la vente en mentionnant :
- la dénomination de vente
- l'état physique du produit (ex. décongelé)
- le prix de vente (ex. à la pièce et/ou au poids selon le cas)
Conserves et semi-conserve
Le décret du 19 février 1991 sur l'étiquetage a supprimé l'obligation réglementaire d'une lettre-code de l'année de fabrication. Cependant, l'utilisation de ces lettres-codes permet une identification claire de l'année de fabrication par les distributeurs et les gestionnaires de stocks de produits appertisés. Ces lettres-codes sont communiquées aux Autorités de Tutelle, aux Organisations Professionnelles françaises et européennes, à l'Administration.
Source: http://ogi.ctcpa.eu/media/fck/file/LETTRES-CODES-ANNEE-2014.pdf / http://www.economie.gouv.fr/dgccrf/Publications/Vie-pratique/Fiches-pratiques/Etiquetage-des-conserves
Produits de la mer et eau douce
Huile d'olive
http://www.economie.gouv.fr/dgccrf/huiles-dolive-des-ameliorations-sur-letiquetage
Tracabilité OGM
- http://www.economie.gouv.fr/dgccrf/consommation/Etiquetage-des-produits/OGM
- http://ec.europa.eu/food/food/biotechnology/gmfood/qanda_fr.pdf
Vins
http://www.economie.gouv.fr/dgccrf/Publications/Vie-pratique/Fiches-pratiques/Etiquetage-des-vins
Oeufs
http://www.economie.gouv.fr/dgccrf/Publications/Vie-pratique/Fiches-pratiques/Etiquetage-des-oeufs
Fruits & Légumes
Autres
Europe
Businesses in the EU handling food products, whether for human consumption or not, must be registered. In some cases, they cannot operate unless approved by their country's competent authority. EU countries draw up public lists of the approved establishments.
In accordance with Regulation (EC) No 852/2004 of the European Parliament and of the Council of 29 April 2004 on the hygiene of foodstuffs, establishments preparing foodstuffs must be registered in each Member State. Furthermore, establishments handling products of animal origin for which hygiene conditions are laid down in Regulation (EC) No 853/2004 (http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2009:300:0001:0033:EN:PDF) of the European Parliament and of the Council of 29 April 2004 laying down specific rules for food of animal origin, must be approved by the competent authority of each Member State. According to Regulation (EC) No 882/2004 of the European Parliament and of the Council of 29 April 2004 on official controls performed to ensure the verification of compliance with feed and food law, animal health and animal welfare rules, Member States must make up-to-date lists of such establishments available to other Member States and to the public.
In order to assist Member States in making up-to-date lists available to other Member States and to the public, the Commission has created a single, central and easy access to the different lists of approved food establishments published by each Member State.
Every national website is to contain a first page regrouping food activities for which approval is compulsory. For ease of reference, this masterlist is developed following the sequence of the sections in Annex III to Regulation (EC) No 853/2004 (http://eur-lex.europa.eu/legal-content/EN/ALL/?uri=OJ:L:2004:226:TOC). Technical specifications pdf have been prepared to ensure wide availability of the information concerning these establishments and to improve the readability of the lists. They include layouts, together with relevant information and codes used to describe food activities.
There are also regulations for animal by-products and feed sector.
Member States will have to comply with these specifications as from 1 January 2007.
Estampille Sanitaire / EU Food Establishments
- Exemple: FR 38.012.001 CE
- Deux premières lettres = pays, deux premiers chiffres = département, trois chiffres suivants = commune au code INSEE, trois derniers chiffres = établissement
- Estampille sanitaire est délivrée par autorité du pays.
- Les entreprises obtenant une estampille sanitaire sont organisées par Sections (entreposage, abattage volaille ...). Un travail de normalisation de ces sections sur l'ensemble des pays européens est en cours par les équipes de OFF.
- http://en.wiki.openfoodfacts.org/Project:Food_establishments
- https://github.com/openfoodfacts/eu-food-data
- French list: http://agriculture.gouv.fr/liste-des-etablissements-agrees-ce
- See list of country codes EMB: http://ec.europa.eu/food/food/biosafety/establishments/list_en.htm
EU Organic Codes
UK
in the UK, all figures is a code attributed directly by the FSA, when one or two letters followed by figures are code attributed by local authorities (the letters identify the local authority). But some UK codes also include an "S" after the figures, we still don't know why.
US
Fruits & Vegetables
http://www.motherjones.com/environment/2014/09/fruit-veggies-produce-origins-trade-secret
Spain
- AECOSAN http://www.aecosan.msssi.gob.es/
- Search Engine: http://rgsa-web-aesan.msssi.es/rgsa/formulario_ue_js.jsp
Spanish wines
I found the "Bodegas" directory in the "Consejo Regulador" site: http://es.riojawine.com/multimedia/files/directorio.xls There you can find the row: Pernod Ricard Winemakers Spain (Campo Viejo) with "Reg Embotellador" = "1.850-LO" Our number!! This is in the city of Logroño (the capital of La Rioja) and the street is "La Rad de Santa Cruz, s/n".
Spanish packaging codes site : RGSEAA number
- the RGSEAA number that is used to build the ES RGSEAA EC number
- website: http://rgsa-web-aesan.msssi.es/rgsa/formulario_principal_js.jsp web,
- If you look at it, with that information (Razón Social=Pernod Ricard Winemakers Spain; Localidad=Logroño), you get the only number: 30.00641/LO (i.e. ES 30.00641/LO CE). The address also matches the data.
- The winery celler (bodega) is this, just by curiosity :-)
http://www.pernodricardbodegas.com/bodegas/campo-viejo/la-bodega/?lang=en http://www.aesan.msssi.gob.es/aesa/web/AESA.jsp
- All the associated companies and locations for the Spanish packaging codes (not only the animal derived ones) can be retrieved from: http://rgsa-web-aesan.msssi.es/rgsa/formulario_principal_js.jsp
- I keep a data sheet with all codes downloaded from that site.
- My last version is from 24-10-2014, so I guess is pretty up-to-date.
- I can upload the file if you want to some place (github, dropbox, ....).
- It is in odt format (openoffice/libreoffice) but I can convert it to another format if needed.
I also have a table (in wiki format) with the codes of the products added by myself with the name of the company, parent company, location and some other information. It is basically the info that I use to include beside the EU code or Tax Identification number and in the location of the processing company field. I have to clean it a bit and I'll post it here in a new wiki page.The only problem that I can see to match locations with locations in OpenStreetMap is that the first document uses for the places where other languages are used (beside to Spanish), sometimes the traditional name in Spanish, sometimes the name in the local language (basque, catalan...) and in other cases both. A mesh! In my table I use normally the Spanish names. We'll see if some modification have to be done to match with OpenStreetMap location names, whatever they are...Maybe a new field can be added to the app to put in there the company name (and parent companies) that match the EU packaging code (and match also the location of manufacturing or processing place). I normally include that information beside the packaging code, but that's ugly :wink: "
Is there a licence / terms of use for http://rgsa-web-aesan.msssi.es/rgsa/formulario_principal_js.jsp ?
I am looking inside the site (not just the searching engine) and I can't see anything regarding to the license or term of use. It's a public register depending on the public administration, so I don't think there is any problem to get the data. Otherwise wouldn't be available to the public.
Can we just scrape it to get the data for all the ES packaging codes we have in the database?
>> Yes, the only problem can be the significative "zeros" after the key. The format is: AA.xxxxxx/BB, where AA is the key of the type of food (i.e. vegetables, fish, processed products...), BB is the province and xxxxxx is the number. But zeros at the beginning of the number are nor significatives. I mean, 21.000345/MA and 21.345/MA represent the same number. When OFF started 2 years ago, the engine presented always 6 digit. Now, show you normally 5. Well... summarizing, right now in OFF not always match the number (counting zeros) with the number in the RGSA datebase. So, if you write a script to retrieve the data you'll have to take that in consideration.
works with a GET request: >> Yes, you can recuperate the data with a GET request with code or a script. I particularly get the data from the "easy" way: just write the key that I am lookin for (for example 21) in the space for the 2 first digit in the "N° RGSEAA:" field. After searching you get all the results for that particular key and then you choose the "Exportar Excel" option. You can do the same for all the keys and join all the data in a table in a local database, a datasheet or wathever...
Russian Federation
- http://www.fsvps.ru/fsvps/importExport?_language=en
- Example for a French company: http://www.fsvps.ru/fsvps/importExport/france/enterprise.html?id=18640&_language=en
Codes by product
Fruits & Vegetables
IFPS (International Federation for Produce Standards)
- IFPS is the International Federation for Produce Standards (IFPS) is composed of national produce associations from around the globe.The long term objective of the federation is to improve the supply chain efficiency of the fresh produce industry through developing, implementing and managing harmonized international standards. IFPS was previously known as The International Federation for Produce Coding (IFPC), IFPS is a coalition of fruit and vegetable associations from the around the globe that joined together in 2001 as equal partners to pursue the task of introducing a global standard for the use of international Price Look-Up (PLU) numbers.
- Members: Asociacion de Exportadores de Chile, Canadian Horticultural Council, Canadian Produce Marketing Association, Fresh Produce Consortium (UK), Frug I Com (Netherlands), Norges Frukt-og Gronnsaksgrossisters Forbund (Norway), Produce Marketing Association (US), Produce Marketing Association Australia-New Zealand, United Fresh (New Zealand), United Fresh Produce Association (US)
- Website: http://www.ifpsglobal.com/
PLU Codes
- Système volontaire (non obligatoire) codifiant le type de produit (ex banane cavendish) mais ne donne aucune information sur la provenance
Regular PLU Codes
PLU or Price Look Up codes have been used by supermarkets since 1990 to make check-out and inventory control easier, faster and more accurate. PLU codes are used to identify bulk produce (and related items such as nuts and herbs). They tell the supermarket cashier whether an apple is a conventionally grown Fuji apple which may sell for $1.29 per pound versus an organically grown Fuji apple which may sell for $2.29 per pound. IFPS is the global organization that assigns PLU codes to produce items.
PLU codes are 4 or 5 digit numbers and will appear on a small sticker applied to the individual piece of fresh produce. The PLU number identifies produce items based upon various attributes which can include the commodity, variety, growing methodology (e.g. organic), and the size. These numbers are assigned by the IFPS after rigorous review at both national and international levels. PLU codes ensure that the accurate price is paid by consumers by removing the need for cashiers to identify the product and whether or not it is conventionally or organically grown.
The 4-digit PLU codes for produce are randomly assigned within a series of numbers within the 3000 and 4000 series. There is no intelligence built into the 4-digit code. For example, no one number within the 4-digit number represents anything in particular. The 4-digit codes are for conventionally grown produce. 5-digit codes are used to identify organic produce. The prefix of '9' would be placed in front of the 4-digit conventionally grown code for organic produce. You will not see the 5-digit codes in the PLU codes database since they are simply prefixes added on to the conventionally grown produce PLU codes.
The PLU coding system is voluntary, not mandate by any governing body. There are currently over 1400 PLU codes issued for fresh produce and produce related items.
Standardized Universal Product Codes (UPCs) for fresh fruits and vegetables are also available on a subscription basis through the Produce Marketing Association (PMA) website.
- PLU Search Database: http://www.ifpsglobal.com/Identification/PLU-Codes/PLU-codes-Search or http://plucodes.com/
- Complete list of PLU: Through PLU Search Database or in PDF format at http://www.plucodes.com/docs/PLU_List_Commodity_Varietal_Bilingual.pdf
- PLU FAQ: http://www.ifpsglobal.com/Portals/22/IFPS%20Documents/PLU%20FAQ/PLU%20Site%20FAQs%20Aug%202015%20v2.pdf. For a raw extraction: https://docs.google.com/spreadsheets/d/1YieZxWFHv9bAHO9zRW4I5_ghnId5B2oxVz72vMvTF-I/edit#gid=1530069957 and a parsable version https://files.slack.com/files-pri/T02KVRT1Q-F02M8KZKV/download/minimalversion.xlsx
Retailer Assigned PLU Codes
Retailer Assigned PLU codes allow retailers to designate their own numbers for items that do not have a standard PLU code assigned. After a supplier makes specific arrangements with all retailers that will receive the item, a Retailer Assigned PLU code may be used. Each retailer may assign a different PLU code to the same item, so for items where multiple Retailer Assigned codes exist, we suggest you get the agreement in writing for your records. For some commodities where there are a large number of varieties, e.g. apples or oranges, a block of Retailer Assigned codes are available in the PLU codes database to identify these varieties which currently don't have a PLU code. Before selecting one to use, you must contact your retailer(s) to ensure they are not already using that code for another variety of that commodity.
Other Ressources
- Collection of fruits and vegetables tags: http://legufrulabelofolie.fr/index.php
FSC Codes
- Those are codes for the paper or cardboard used for packaging.
- Forest Stewardship Council
- http://info.fsc.org/details.php?id=a0240000005sQs5AAE&type=certificate&return=certificate.php
- http://world.openfoodfacts.org/labels (filter with "FSC")
FAO Fisheries & Aquaculture Codes
MSC Codes
- https://github.com/openfoodfacts/msc-codes
- MSC (sustainable fishing) label can be easily accessed with precision, it would be nice to put a link to the website of the MSC with information from fisherman.
For example 3256224531289 Product code indicating the MSC "MML-C-1021." By putting this code in the URL of the MSC, it gets the name and origin of the fisherman http://cert.msc.org/supplierdirectory/VController.aspx?CertNr=MML-C-1021
Soil Association
organic food
- www.soilassociation.org
- http://www.sacert.org/fooddrink/findalicensee
- they seems to provide a number such as DA20245
Egg Codes
Bio Codes
- https://github.com/openfoodfacts/bio-codes
- Integrated as sub-labels
Methodology to extract data for European Food Establishments
General
Get Data Files
- All data (url list and data files) can be found here https://github.com/openfoodfacts/eu-food-data
- There is a general csv file with the general link to the data repository for each country: https://github.com/openfoodfacts/eu-food-data/blob/master/list-eu-and-partner-countries.csv
- There are individual folders for each target country (specific url-list for each European Agreement Section and data files)
- A google sheet document is used to map all files available in the target countries. It also map the section name for every country in its own language (or translation in English) and the related European Section, which is used as a general taxonomy. This Google Sheet can be found here: https://docs.google.com/spreadsheets/d/1egdo58Ds8PNi5G_4F2UtWOWC1V0k3tXBgPhZXs5FRqM/edit?usp=sharing
- We can usually find txt or csv file but for some countries the data is only available in PDF. Those need OCR treatment before data extraction.
Build CSV/TXT files
- Several formats are used in EU countries. A specific approach is needed for each of them. Refer below for details for each country.
- For non text-PDF files, use Tesseract and this Python script https://github.com/virantha/pypdfocr to get text PDF
- For text PDF files, use Tabula and use the command line tool to create scripts https://github.com/tabulapdf/tabula-extractor/wiki/Using-the-command-line-tabula-extractor-tool
- Cleanup is needed to ensure data accuracy.
Geocode
- The best way is to use the geopy library to use nominatim in a Python script
- To check geocoding accuracy, it is best to make a pivot table of all cities in the target country, and to add a lookup function in the main data file. Adding the name of the country in the address usually lead to better results.
- Local Authorities List in UK http://localweblist.net/
Polishing files
- As a company has one approval codes but can have it for several section under the EU regulation (slaughterhouse, warehouse, fishery products - Freezing vessel ...), you must used a pivot table and some cleaning to get a file with the following columns: name of the establishment, approval code, type (concatenation of the Section name under which the company is approved), one colum for each EU section, adresse, latitude, longitude
- The "type" column is used for search query
- To get a nice vizualisation and make corrections, use http://blog.perrygeo.net/2013/09/30/leaflet-simple-csv/ to make a map of all listed companies. You can filter company on their section by simply using the search field
Find Additional Data on companies
- Infogreffe
- https://opencorporates.com/
France
- I just build a script that takes all the french agreement info from Agriculture Ministry and concatenate them in one file. Next step is to do the same for UK. The step after that is to cleverly agregate the duplicates (some companies have several health agreements under the same agreement number)
https://github.com/openfoodfacts/eu-food-data/blob/master/scripts/FR-script.py This script use this file to get the list of URL to retrieve https://github.com/openfoodfacts/eu-food-data/blob/master/fr/urls-fr.txt
- First work performed on
UK
- As UK is divided in 4 regions (Ireland, England, Wales and Scotland) and because they have different file format, we use a 3-file script
- https://github.com/openfoodfacts/eu-food-data/blob/master/scripts/UK-urls.txt => all UK urls
- https://github.com/openfoodfacts/eu-food-data/blob/master/scripts/UK-methods.txt => list which method to use depending on the file type
- https://github.com/openfoodfacts/eu-food-data/blob/master/scripts/UK-script.py => the script itself
- Source: Food Standards Agency - http://www.food.gov.uk/business-industry/meat/audit
DE
- @vince has performed a first extraction based on file in https://github.com/openfoodfacts/eu-food-data/tree/master/de with the help of the following script
cat export.csv | awk -F';' '{print "\"" $4 "\"; Deutschland"}' | sed "1d" | sort -u | iconv -f ISO-8859-15 -t UTF-8 > to_geocode.csv
NL
- Data from NWA (Netherlands Food & Consumer Product Safety Authority)
Inspiration