Changes

Jump to navigation Jump to search
no edit summary
Line 1: Line 1: −
[[Category:OpenWineFacts]]
+
[[Project:ProductOpener]] > [[Product Opener documentation]]
<pre>
+
[[Category:ProductOpener]]
<en:Alcoholic beverages
+
== Introduction ==
en:Wines
  −
es:Vinos
  −
fr:Vins, vin
     −
<en:Beverages
+
Global taxonomies are used in Product Opener to "tag" products. For instance Open Food Facts uses taxonomies for categories, countries, labels, additives etc.
en:Non-Alcoholic beverages
  −
fr:Boissons sans alcool, boisson sans alcool
     −
<en:Wines
+
The taxonomies are multilingual (each tag value can have translations in different languages) and hierarchical (tags have parents, and applying a tag to a product also applies the parent tags).
<en:Non-Alcoholic beverages
  −
en:Non-Alcoholic wines, Alcohol-free wines
  −
fr:Vins sans alcool, vins déalcoolisés
     −
<en:Wines
+
== Taxonomy files ==
en:Bordeaux, Bordeaux wines
  −
fr:Bordeaux
     −
<en:Wines
+
Taxonomies are kept in the taxonomies directory:
en:French wines, Wines from France
  −
fr:Vins français, vin français
     −
<en:French wines
+
<pre>
fr:Vins de pays, vin de pays
+
stephane@ks3095298:~/product-opener/taxonomies$ ls
 +
additives.result.sto  categories.result.txt  countries.txt
 +
additives.result.txt  categories.txt        labels.result.sto
 +
additives.txt          countries.result.sto  labels.result.txt
 +
categories.result.sto  countries.result.txt  labels.txt
 +
</pre>
   −
<en:Wines
+
* categories.txt is the source definition of the categories taxonomy
en:Red wines
+
* categories.result.txt is the resulting normalized definition after compilation in the same format as the source categories.txt
fr:vins rouges, vin rouge
+
** entries are normalized (e.g. the en:value is preferred to specify parents)
 +
** entries can be sorted in a different order
 +
* categories.result.sto in the resulting binary representation of the taxonomy, in Perl's native format
   −
<en:Wines
+
== Taxonomy source format ==
en:White wines
  −
fr:vins blancs, vin blanc
     −
<en:Wines
+
See [[Global taxonomies]]
en:Wines from Italy, Italian wines
  −
fr:Vins italiens, vin italien, vins d'Italie, vin d'Italie
     −
<en:Wines
+
== Taxonomy source files ==
en:Wines from Portugal, Portuguese wines
  −
fr:Vins portugais, Vins du Portugal, Vin portugais, Vin du Portugal
     −
<en:Wines from Portugal
+
Open Food Facts taxonomy source files are on the wiki so that contributors can improve them (adding tags, adding translations, reorganizing tags): http://en.wiki.openfoodfacts.org/Global_taxonomies
pt:Denominação de Origem Controlada
     −
<pt:Denominação de Origem Controlada
+
== Compilation ==
pt:Vinho verde (DOC)
     −
<pt:Denominação de Origem Controlada
+
Before taxonomies can be used in Product Opener, they need to be compiled in a binary format. The compilation has two goals:
pt:Porto (DOC)
     −
<pt:Denominação de Origem Controlada
+
# Generate as many synonyms as possible for each tag
pt:Douro (DOC)
+
#* This is so that end users can enter any form of a tag and still have it matched to the canonical tag
 +
#* e.g. "Yaourts fraise" and "Yaourt aux fraises"
 +
# Compute the hierarchy
 +
#* for each tag, knowing which tags are parents and childrens
 +
 +
The compilation code is in the build_tags_taxonomy() function of Tags.pm. This function is called by script/build_tags_taxonomy.pl
   −
<pt:Denominação de Origem Controlada
+
=== Compilation steps ===
pt:Távora Varosa (DOC)
     −
<pt:Denominação de Origem Controlada
+
The compilation steps are:
pt:Dão (DOC)
+
* read translations and synonyms
 +
* compute synonyms for each tag (using synonyms for some of the words)
 +
* add more synonyms (remove stopwords, generate simple singular/plural forms)
 +
* compute the hierarchy
   −
<pt:Denominação de Origem Controlada
+
Note: if the taxonomy is big and/or contain many synoym
pt:Bairrada (DOC)
     −
<pt:Denominação de Origem Controlada
+
=== Compilation script ===
pt:Beira Interior (DOC)
     −
<pt:Denominação de Origem Controlada
+
==== Testing ====
pt:Encostas de Aire (DOC)
+
<pre>
 +
stephane@ks3095298:~/product-opener/scripts$ ./build_tags_taxonomy.pl categories
 +
</pre>
 +
This will compile the categories but generate only categories.results.txt from categories.txt
   −
<pt:Denominação de Origem Controlada
+
You should review the resulting txt to make sure unwanted synonyms or relations were not created.
pt:Alentejo (DOC)
     −
<pt:Denominação de Origem Controlada
+
===== empty names : =====
pt:Ribatejo (DOC)
     −
<pt:Denominação de Origem Controlada
+
An empty name " : " indicates there is an error in the taxonomy source file, usually a non-existing entry was entered as the parent.
pt:Lourinhã (DOC)
     −
<pt:Denominação de Origem Controlada
+
<pre>
pt:Alenquer (DOC)
+
< :
 +
< en:Wheat flours
 +
en:Wheat malt flours
 +
es:Harinas de trigo malteadas, Harinas de malta de trigo
 +
fr:Farines de malt de blé
 +
</pre>
   −
<pt:Denominação de Origem Controlada
+
Check the source file to see which entry is wrong:
pt:Torres Vedras (DOC)
     −
<pt:Denominação de Origem Controlada
+
<pre>
pt:Arruda (DOC)
+
<en:Wheat flours
 +
<fr:Malt flours
 +
en:Wheat malt flours
 +
es:Harinas de trigo malteadas, Harinas de malta de trigo
 +
fr:Farines de malt de blé
 +
</pre>
   −
<pt:Denominação de Origem Controlada
+
Here "fr:Malt flours" is non existing (and should be "en:Malt flours").
pt:Bucelas (DOC)
     −
<pt:Denominação de Origem Controlada
+
Note: the build_tags_taxonomy.pl script has been updated to report those errors.
pt:Colares (DOC)
     −
<pt:Denominação de Origem Controlada
+
==== Publishing ====
pt:Carcavelos (DOC)
     −
<pt:Denominação de Origem Controlada
+
<pre>
pt:Palmela (DOC)
+
stephane@ks3095298:~/product-opener/scripts$ ./build_tags_taxonomy.pl categories publish
 +
</pre>
 +
This will also store categories.result.sto
   −
<pt:Denominação de Origem Controlada
+
== Test ==
pt:Muscat de Setúbal (DOC)
     −
<pt:Denominação de Origem Controlada
+
Taxonomies are very powerful tools, but with great powers comes great risks. The synonyms, stopwords etc. can create unwanted synonyms.
pt:Lagos (DOC)
     −
<pt:Denominação de Origem Controlada
+
=== Testing tools ===
pt:Portimão (DOC)
     −
<pt:Denominação de Origem Controlada
+
We need testing tools. Please create some. :-)
pt:Lagoa (DOC)
     −
<pt:Denominação de Origem Controlada
+
== Restart Apache ==
pt:Tavira (DOC)
     −
<pt:Denominação de Origem Controlada
+
<pre>
pt:Madère (DOC)
+
</pre>
    +
After restarting Apache, editing a product will recompute all tags using the newest published taxonomies.
   −
<en:Wines
  −
en:Wines from Spain, Spanish wines
  −
fr:Vins espagnols, Vin espagnol, Vin d'Espagne, Vins d'Espagne
     −
<en:Wines
+
== Updating products ==
fr:Vins américains
     −
<fr:Vins américains
+
The scripts directory contains scripts that will recompute tags for all products in the database.
en:Wines from California, Californian Wines, California Wines
  −
fr:Vins californiens, vins de Californie, vin californien, vin de Californie
  −
wikidata:en:Q1134713
  −
 
  −
<en:Wines
  −
fr:Vins effervescents
  −
en:Sparkling wines
  −
wikidata:en:Q321263
  −
 
  −
<fr:Vins effervescents
  −
fr:Blanquette de Limoux
  −
wikidata:en:Q881685
  −
 
  −
<fr:Vins effervescents
  −
fr:Champagne, champagnes, vins de Champagne
  −
en:Champagne, champagnes
  −
 
  −
<fr:Vins effervescents
  −
fr:Clairette de Die, Clairettes de Die
  −
wikidata:en:Q1094848
  −
 
  −
<fr:Vins effervescents
  −
fr:vins mousseux, vin mousseux, mousseux
  −
 
  −
<fr:Vins effervescents
  −
fr:vins perlant, vin perlant, perlé
  −
 
  −
<fr:Vins effervescents
  −
fr:vins pétillant, vin pétillant, pétillant
  −
 
  −
<en:Wines
  −
fr:Vins mutés, vin muté
  −
en:Fortified wines
  −
wikidata:en:Q722338
  −
 
  −
<en:Wines from Portugal
  −
<fr:Vins mutés de type vin doux naturel
  −
fr:Porto, vins de Porto, vin de Porto
  −
 
  −
<en:Red wines
  −
<fr:Porto
  −
en:red ports
  −
fr:porto rouges, porto rouge
  −
 
  −
<en:Red ports
  −
fr:porto réductifs, porto réductif
  −
 
  −
<fr:Porto réductifs
  −
fr:Porto Crusted, Crusted
  −
 
  −
<fr:Porto réductifs
  −
fr:Porto Late Bottled Vintage, LBV
  −
 
  −
<fr:Porto réductifs
  −
fr:Porto Ruby
  −
 
  −
<fr:Porto réductifs
  −
fr:Porto Single Quinta Vintage
  −
 
  −
<fr:Porto réductifs
  −
fr:Porto Vintage, porto millésimé
  −
 
  −
<en:Red ports
  −
fr:portos oxydatifs, porto oxydatif
  −
 
  −
<fr:Portos oxydatifs
  −
fr:Porto Tawny, tawny
  −
 
  −
<en:White wines
  −
<fr:Porto
  −
en:White Ports
  −
fr:porto blancs, porto blanc
  −
 
  −
en:Sweet wines, natural sweet wines
  −
es:Vinos dulces
  −
fr:Vins doux, Vin doux, Vins doux naturels, Vin doux naturel
  −
 
  −
<en:Sweet wines
  −
<fr:Vins mutés
  −
fr:vins mutés de type vin doux naturel, VDN
  −
 
  −
<fr:Vins mutés de type vin doux naturel
  −
fr:Banyuls
  −
 
  −
<fr:Vins mutés de type vin doux naturel
  −
fr:Maury
  −
 
  −
<fr:Vins mutés de type vin doux naturel
  −
fr:Muscats, Muscat
  −
 
  −
<fr:Muscats
  −
fr:Muscat de Beaumes-de-Venise, Muscats de Beaumes-de-Venise
  −
 
  −
<fr:Muscats
  −
fr:Muscat de Frontignan, Muscats de Frontignan
  −
 
  −
<fr:Muscats
  −
fr:Muscat de Lunel, Muscats de Lunel
  −
 
  −
<fr:Muscats
  −
fr:Muscat de Mireval, Muscats de Mireval
  −
 
  −
<fr:Muscats
  −
fr:Muscat de Rivesaltes, Muscats de Rivesaltes
  −
 
  −
<fr:Muscats
  −
fr:Muscat de Saint-Jean-de-Minervois, Muscats de Saint-Jean-de-Minervois
  −
 
  −
<fr:Muscats
  −
fr:Muscat du Cap-Corse, Muscats du Cap-Corse
  −
 
  −
<fr:Vins mutés de type vin doux naturel
  −
fr:Rasteau
  −
 
  −
<fr:Vins mutés de type vin doux naturel
  −
fr:Rivesaltes
  −
 
  −
<fr:Vins mutés
  −
fr:mistelle, vins mutés de types mistelle, vins de liqueur, vin de liqueur
  −
 
  −
<fr:Mistelle
  −
fr:Pineau des Charentes, Floc de Gascogne, Macvin du Jura
  −
 
  −
<fr:Vins mutés
  −
fr:vins vinés, vin viné
  −
 
  −
<en:Wines
  −
fr:Vins rosés, vin rosé
      +
<pre>
 +
stephane@ks3095298:~/product-opener/scripts$ ./update_all_products_categories.pl
 
</pre>
 
</pre>

Navigation menu