Difference between revisions of "Server-side product indexing and search"
Jump to navigation
Jump to search
(progress report) |
|||
| Line 1: | Line 1: | ||
== Summary == | == Summary == | ||
| − | Server-side product | + | Server-side product indexing and search is the 2nd of the 4 sub-tasks of the [[Project:Personalized_Search]] funded by the NGI0 Discovery Fund managed by NlNet. |
This page documents the progress made in Q2 2020. | This page documents the progress made in Q2 2020. | ||
| Line 15: | Line 15: | ||
# The app uses the user preferences stored locally to personalize the search results | # The app uses the user preferences stored locally to personalize the search results | ||
| − | == | + | == Initial research and specifications (completed in May and June 2020) == |
At the start of the project, we evaluated the different options for server-side product indexing and search: | At the start of the project, we evaluated the different options for server-side product indexing and search: | ||
| Line 22: | Line 22: | ||
* [[Server-side product indexing and search - Technical Specs]] | * [[Server-side product indexing and search - Technical Specs]] | ||
| − | == New Search API == | + | == New Search API (completed in August and September 2020) == |
The existing Open Food Facts search API is outdated and hacky (it was built on top of the OFF web site search form and is unnecessarily convoluted) and does not support some of the requirements for the Personal Search project (in particular being able to retrieve a given set of products using their barcodes). | The existing Open Food Facts search API is outdated and hacky (it was built on top of the OFF web site search form and is unnecessarily convoluted) and does not support some of the requirements for the Personal Search project (in particular being able to retrieve a given set of products using their barcodes). | ||
| − | We | + | We have create a new a new [[Open Food Facts Search API Version 2]] that is simpler but also more powerful. |
| − | + | Key features of the new Search API: | |
| + | * Simplified parameter specification | ||
| + | * Tag parameters (e.g. categories, labels, ingredients) can be searched in any language | ||
| + | * Support for AND, NOT and OR queries for tags fields (e.g. product with all those labels, not those labels, or one of those labels) | ||
| + | * Allow to sort results by popularity of products (most scanned products) | ||
| + | * New /api/v2/search (JSON) and /search (OFF web site) endpoints that accept the same parameters | ||
| − | We | + | == New Product Attributes for all search features (completed from August to October 2020) == |
| + | |||
| + | We have created new [[Product Attributes]] that allow clients (like apps but also the OFF web site) to easily filter and rank search results according to the user preference, and to explain to users how well the products match their preferences. | ||
| + | |||
| + | Key features of product attributes: | ||
| + | * All search criteria / features (e.g. nutritional quality, if a product is vegan, contains a specific allergen etc.) are computed individually on the server side, and then made available to clients in the same normalized format | ||
| + | ** Clients can implement local filtering and ranking with [[Client-side libraries for personalized product filtering and ranking|a very simple algorithm]]. All the complexity is of the different search features is handled on the server side. | ||
| + | ** Clients can get localized names, descriptions etc. for each search feature in the language they choose. | ||
| + | ** New search features (e.g. a new diet, a new environmental or social score) can be made available to clients without any code change on the clients. | ||
| + | |||
| + | == Next milestone: Client-side libraries for personalized product filtering and ranking == | ||
| + | |||
| + | To make it even easier to use the new Search API and Product Attributes, we are developing [[Client-side libraries for personalized product filtering and ranking]] in Javascript and DART (for Flutter apps). | ||
[[Category:Project:Personalized_Search]] | [[Category:Project:Personalized_Search]] | ||
[[Category:ProductOpener]] | [[Category:ProductOpener]] | ||
Revision as of 10:50, 12 October 2020
Summary
Server-side product indexing and search is the 2nd of the 4 sub-tasks of the Project:Personalized_Search funded by the NGI0 Discovery Fund managed by NlNet.
This page documents the progress made in Q2 2020.
Overview
Diagram source: https://vecta.io/app/edit/-M2XyVv8ZoaLNrW-zQoT
- The Open Food Facts mobile app (and 3rd party apps) make generic search requests that do not contain user preferences
- The server returns a big number of generic results
- The app uses the user preferences stored locally to personalize the search results
Initial research and specifications (completed in May and June 2020)
At the start of the project, we evaluated the different options for server-side product indexing and search:
- Server-side product indexing and search - Functional Specs
- Server-side product indexing and search - Technical Specs
New Search API (completed in August and September 2020)
The existing Open Food Facts search API is outdated and hacky (it was built on top of the OFF web site search form and is unnecessarily convoluted) and does not support some of the requirements for the Personal Search project (in particular being able to retrieve a given set of products using their barcodes).
We have create a new a new Open Food Facts Search API Version 2 that is simpler but also more powerful.
Key features of the new Search API:
- Simplified parameter specification
- Tag parameters (e.g. categories, labels, ingredients) can be searched in any language
- Support for AND, NOT and OR queries for tags fields (e.g. product with all those labels, not those labels, or one of those labels)
- Allow to sort results by popularity of products (most scanned products)
- New /api/v2/search (JSON) and /search (OFF web site) endpoints that accept the same parameters
New Product Attributes for all search features (completed from August to October 2020)
We have created new Product Attributes that allow clients (like apps but also the OFF web site) to easily filter and rank search results according to the user preference, and to explain to users how well the products match their preferences.
Key features of product attributes:
- All search criteria / features (e.g. nutritional quality, if a product is vegan, contains a specific allergen etc.) are computed individually on the server side, and then made available to clients in the same normalized format
- Clients can implement local filtering and ranking with a very simple algorithm. All the complexity is of the different search features is handled on the server side.
- Clients can get localized names, descriptions etc. for each search feature in the language they choose.
- New search features (e.g. a new diet, a new environmental or social score) can be made available to clients without any code change on the clients.
Next milestone: Client-side libraries for personalized product filtering and ranking
To make it even easier to use the new Search API and Product Attributes, we are developing Client-side libraries for personalized product filtering and ranking in Javascript and DART (for Flutter apps).