Changes

Jump to navigation Jump to search
Add first draft of reconciliation algorithm
Line 11: Line 11:  
=== The value of synchronizing datasources ===
 
=== The value of synchronizing datasources ===
   −
Synchronizing and cross-referencing the ingredients included in OpenFoodFacts with other datasources can be of mutual benefit the the quality of all datasources involved.
+
Synchronizing and cross-referencing the ingredients included in OpenFoodFacts with other datasources can be of mutual benefit to the quality of all datasources involved.
    
To achieve progress, the goal should not be to achieve a single authoritative datasource - an objective that's unlikely to be either achievable or desirable across multiple projects and cultures with subjective and reasonable opinions of their own - but to enrich each datasource and to learn and provide additional (and ideally well-substantianted) supporting information that may help each participant in future.
 
To achieve progress, the goal should not be to achieve a single authoritative datasource - an objective that's unlikely to be either achievable or desirable across multiple projects and cultures with subjective and reasonable opinions of their own - but to enrich each datasource and to learn and provide additional (and ideally well-substantianted) supporting information that may help each participant in future.
   −
Sometimes cross-referencing an individual ingredient is straightforward: one OpenFoodFacts ingredient may map near-unambiguously to exactly one corresponding item in another datasource.  In practice however, various ambiguities can and do occur for various reasons:
+
Sometimes cross-referencing an individual ingredient is straightforward: one OpenFoodFacts ingredient may map near-unambiguously to exactly one corresponding item in another datasource.  In practice however, ambiguities can and do occur for various reasons:
    
   * Naming ambiguities and the effects of natural-language translation (lime, lemon, citron).
 
   * Naming ambiguities and the effects of natural-language translation (lime, lemon, citron).
Line 25: Line 25:  
=== Datasource: WikiData ===
 
=== Datasource: WikiData ===
   −
WikiData is a [https://wiki.openfoodfacts.org/Structured_Data structured data] initiative hosted by the [https://foundation.wikimedia.org/wiki/Home WikiMedia foundation], and since it is universal in scope, food ingredients are part of the domain that it covers.  Within WikiData, each data item is identified by a QID -- a unique number (integer) and prefixed by the letter 'Q'.  For example, tomato has the WikiData QID of [https://www.wikidata.org/wiki/Q23501 Q23501].
+
WikiData is a [https://wiki.openfoodfacts.org/Structured_Data structured data] initiative hosted by the [https://foundation.wikimedia.org/wiki/Home WikiMedia foundation], and since it is universal in scope, food ingredients are part of the domain that it covers.  Within WikiData, each data item is identified by a QID -- a unique number (integer) and prefixed by the letter 'Q'.  For example, ''tomato'' has a WikiData QID of [https://www.wikidata.org/wiki/Q23501 Q23501].
    
The data model -- the schema -- is flexible and extensible, and is curated and maintained by the community.  Categories of items can be created, and categorical properties with optional-or-expected values can be added and removed over time.  Data quality varies; naively speaking, more popular items are likely to achieve greater eventual accuracy, whereas less-popular items tend to receive less attention.
 
The data model -- the schema -- is flexible and extensible, and is curated and maintained by the community.  Categories of items can be created, and categorical properties with optional-or-expected values can be added and removed over time.  Data quality varies; naively speaking, more popular items are likely to achieve greater eventual accuracy, whereas less-popular items tend to receive less attention.
Line 37: Line 37:  
=== Process: how to reconcile two datasources ===
 
=== Process: how to reconcile two datasources ===
   −
TODO
+
Our objective here is to take two datasources - one that we will call the '''origin''' datasource, and one that we will call the '''destination''' datasource, and to improve at least one of them.
 +
 
 +
It may be possible to automate much of this process - however to begin with, this page describes a manual, iterative process that is time-consuming but allows for subjective decision-making and human resolution of ambiguities.
 +
 
 +
====== Preparation ======
 +
Before starting, it's worth comparing some essential statistics about the datasources:
 +
 
 +
<nowiki>*</nowiki> How many records (items) are included in each datasource?
 +
 
 +
<nowiki>*</nowiki> What can be used as a unique identifier for an item in each datasource?
 +
 
 +
<nowiki>*</nowiki> Does each datasource include a way to cross-reference (link to) a corresponding item in the other datasource?
 +
 
 +
<nowiki>*</nowiki> What methods are available to search the contents of each datasource?
 +
 
 +
The answers to these questions may influence the reconciliation process and how it is best achieved.
 +
 
 +
====== Algorithm ======
 +
 
 +
# Begin by taking a snapshot (a read-only copy) of the '''origin''' datasource, so that you can confirm that you have processed all of the items that it contained when you are finished.
 +
# For each item in the snapshot:
 +
## Extract the information required to search for the item in the '''destination''' datasource.
 +
## Search for the item in the '''destination'''.
 +
## Does the item exist?
 +
### Yes
 +
#### Add a cross-reference from this item to the '''destination''' item.
 +
#### Check for any metadata on the '''destination''' item that appears to be more up-to-date and/or accurate than this item, and where it seems safe and correct to do so, update this item with that higher-quality information.
 +
#### Does the '''destination''' item have an existing cross-reference back to this item?
 +
##### Yes
 +
###### Is the cross-reference to ''this'' item?
 +
####### Yes
 +
######## Success: continue
 +
####### No
 +
######## Potential conflict.
 +
##### No
 +
###### Provide a suggestion to the '''destination''' datasource to add the missing cross-reference.
 +
### No
 +
#### Provide a suggestion to the '''destination''' datasource to add the missing item.
 +
 
 +
 
 +
The most difficult scenario here is when a '''destination''' item already exists but refers to a different item in the '''origin''' dataset.  This often indicates some kind of difference in representation of ingredients, or a different hierarchical ordering of those ingredients.  It's difficult to proscribe an algorithm to resolve this automatically: some amount of common sense, investigation, research and question-asking are usually required to figure out a good solution.  If in doubt, write down a note about the ambiguity, and after you have finished working through all of the '''origin''' items, take a look at the notes collected.  There may be a pattern, and if so that may help to ask better questions about how to resolve the inter-datasource ambiguity more comprehensively.  Answers are not guaranteed: in some cases, further discussion, and potentially evolution of the relevant schemas, may be required.
 +
[[Category:Ingredients]]
10

edits

Navigation menu