Line 37:
Line 37:
=== Process: how to reconcile two datasources ===
=== Process: how to reconcile two datasources ===
−
TODO
+
Our objective here is to take two datasources - one that we will call the '''origin''' datasource, and one that we will call the '''destination''' datasource, and to improve at least one of them.
+
+
It may be possible to automate much of this process - however to begin with, this page describes a manual, iterative process that is time-consuming but allows for subjective decision-making and human resolution of ambiguities.
+
+
====== Preparation ======
+
Before starting, it's worth comparing some essential statistics about the datasources:
+
+
<nowiki>*</nowiki> How many records (items) are included in each datasource?
+
+
<nowiki>*</nowiki> What can be used as a unique identifier for an item in each datasource?
+
+
<nowiki>*</nowiki> Does each datasource include a way to cross-reference (link to) a corresponding item in the other datasource?
+
+
<nowiki>*</nowiki> What methods are available to search the contents of each datasource?
+
+
The answers to these questions may influence the reconciliation process and how it is best achieved.
+
+
====== Algorithm ======
+
+
# Begin by taking a snapshot (a read-only copy) of the '''origin''' datasource, so that you can confirm that you have processed all of the items that it contained when you are finished.
+
# For each item in the snapshot:
+
## Extract the information required to search for the item in the '''destination''' datasource.
+
## Search for the item in the '''destination'''.
+
## Does the item exist?
+
### Yes
+
#### Add a cross-reference from this item to the '''destination''' item.
+
#### Check for any metadata on the '''destination''' item that appears to be more up-to-date and/or accurate than this item, and where it seems safe and correct to do so, update this item with that higher-quality information.
+
#### Does the '''destination''' item have an existing cross-reference back to this item?
+
##### Yes
+
###### Is the cross-reference to ''this'' item?
+
####### Yes
+
######## Success: continue
+
####### No
+
######## Potential conflict.
+
##### No
+
###### Provide a suggestion to the '''destination''' datasource to add the missing cross-reference.
+
### No
+
#### Provide a suggestion to the '''destination''' datasource to add the missing item.
+
+
+
The most difficult scenario here is when a '''destination''' item already exists but refers to a different item in the '''origin''' dataset. This often indicates some kind of difference in representation of ingredients, or a different hierarchical ordering of those ingredients. It's difficult to proscribe an algorithm to resolve this automatically: some amount of common sense, investigation, research and question-asking are usually required to figure out a good solution. If in doubt, write down a note about the ambiguity, and after you have finished working through all of the '''origin''' items, take a look at the notes collected. There may be a pattern, and if so that may help to ask better questions about how to resolve the inter-datasource ambiguity more comprehensively. Answers are not guaranteed: in some cases, further discussion, and potentially evolution of the relevant schemas, may be required.
[[Category:Ingredients]]
[[Category:Ingredients]]