Changes

Jump to navigation Jump to search
Add first draft of reconciliation algorithm
Line 37: Line 37:  
=== Process: how to reconcile two datasources ===
 
=== Process: how to reconcile two datasources ===
   −
TODO
+
Our objective here is to take two datasources - one that we will call the '''origin''' datasource, and one that we will call the '''destination''' datasource, and to improve at least one of them.
 +
 
 +
It may be possible to automate much of this process - however to begin with, this page describes a manual, iterative process that is time-consuming but allows for subjective decision-making and human resolution of ambiguities.
 +
 
 +
====== Preparation ======
 +
Before starting, it's worth comparing some essential statistics about the datasources:
 +
 
 +
<nowiki>*</nowiki> How many records (items) are included in each datasource?
 +
 
 +
<nowiki>*</nowiki> What can be used as a unique identifier for an item in each datasource?
 +
 
 +
<nowiki>*</nowiki> Does each datasource include a way to cross-reference (link to) a corresponding item in the other datasource?
 +
 
 +
<nowiki>*</nowiki> What methods are available to search the contents of each datasource?
 +
 
 +
The answers to these questions may influence the reconciliation process and how it is best achieved.
 +
 
 +
====== Algorithm ======
 +
 
 +
# Begin by taking a snapshot (a read-only copy) of the '''origin''' datasource, so that you can confirm that you have processed all of the items that it contained when you are finished.
 +
# For each item in the snapshot:
 +
## Extract the information required to search for the item in the '''destination''' datasource.
 +
## Search for the item in the '''destination'''.
 +
## Does the item exist?
 +
### Yes
 +
#### Add a cross-reference from this item to the '''destination''' item.
 +
#### Check for any metadata on the '''destination''' item that appears to be more up-to-date and/or accurate than this item, and where it seems safe and correct to do so, update this item with that higher-quality information.
 +
#### Does the '''destination''' item have an existing cross-reference back to this item?
 +
##### Yes
 +
###### Is the cross-reference to ''this'' item?
 +
####### Yes
 +
######## Success: continue
 +
####### No
 +
######## Potential conflict.
 +
##### No
 +
###### Provide a suggestion to the '''destination''' datasource to add the missing cross-reference.
 +
### No
 +
#### Provide a suggestion to the '''destination''' datasource to add the missing item.
 +
 
 +
 
 +
The most difficult scenario here is when a '''destination''' item already exists but refers to a different item in the '''origin''' dataset.  This often indicates some kind of difference in representation of ingredients, or a different hierarchical ordering of those ingredients.  It's difficult to proscribe an algorithm to resolve this automatically: some amount of common sense, investigation, research and question-asking are usually required to figure out a good solution.  If in doubt, write down a note about the ambiguity, and after you have finished working through all of the '''origin''' items, take a look at the notes collected.  There may be a pattern, and if so that may help to ask better questions about how to resolve the inter-datasource ambiguity more comprehensively.  Answers are not guaranteed: in some cases, further discussion, and potentially evolution of the relevant schemas, may be required.
 
[[Category:Ingredients]]
 
[[Category:Ingredients]]
10

edits

Navigation menu