11
edits
Changes
Jump to navigation
Jump to search
Line 37:
Line 37: − TODO+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Ingredient datasource synchronization (view source)
Revision as of 22:41, 2 November 2023
, 22:41, 2 November 2023Add first draft of reconciliation algorithm
=== Process: how to reconcile two datasources ===
=== Process: how to reconcile two datasources ===
Our objective here is to take two datasources - one that we will call the '''origin''' datasource, and one that we will call the '''destination''' datasource, and to improve at least one of them.
It may be possible to automate much of this process - however to begin with, this page describes a manual, iterative process that is time-consuming but allows for subjective decision-making and human resolution of ambiguities.
====== Preparation ======
Before starting, it's worth comparing some essential statistics about the datasources:
<nowiki>*</nowiki> How many records (items) are included in each datasource?
<nowiki>*</nowiki> What can be used as a unique identifier for an item in each datasource?
<nowiki>*</nowiki> Does each datasource include a way to cross-reference (link to) a corresponding item in the other datasource?
<nowiki>*</nowiki> What methods are available to search the contents of each datasource?
The answers to these questions may influence the reconciliation process and how it is best achieved.
====== Algorithm ======
# Begin by taking a snapshot (a read-only copy) of the '''origin''' datasource, so that you can confirm that you have processed all of the items that it contained when you are finished.
# For each item in the snapshot:
## Extract the information required to search for the item in the '''destination''' datasource.
## Search for the item in the '''destination'''.
## Does the item exist?
### Yes
#### Add a cross-reference from this item to the '''destination''' item.
#### Check for any metadata on the '''destination''' item that appears to be more up-to-date and/or accurate than this item, and where it seems safe and correct to do so, update this item with that higher-quality information.
#### Does the '''destination''' item have an existing cross-reference back to this item?
##### Yes
###### Is the cross-reference to ''this'' item?
####### Yes
######## Success: continue
####### No
######## Potential conflict.
##### No
###### Provide a suggestion to the '''destination''' datasource to add the missing cross-reference.
### No
#### Provide a suggestion to the '''destination''' datasource to add the missing item.
The most difficult scenario here is when a '''destination''' item already exists but refers to a different item in the '''origin''' dataset. This often indicates some kind of difference in representation of ingredients, or a different hierarchical ordering of those ingredients. It's difficult to proscribe an algorithm to resolve this automatically: some amount of common sense, investigation, research and question-asking are usually required to figure out a good solution. If in doubt, write down a note about the ambiguity, and after you have finished working through all of the '''origin''' items, take a look at the notes collected. There may be a pattern, and if so that may help to ask better questions about how to resolve the inter-datasource ambiguity more comprehensively. Answers are not guaranteed: in some cases, further discussion, and potentially evolution of the relevant schemas, may be required.
[[Category:Ingredients]]
[[Category:Ingredients]]