Changes

Jump to navigation Jump to search
1,223 bytes added ,  15:29, 19 June 2023
no edit summary
Line 1: Line 1:  
== Overview ==
 
== Overview ==
This document serves as a technical proposal for a new search API.
+
This document serves as a technical proposal for a new search API - V3 ([[Open Food Facts Search API Version 2|see V2 here]]).
 +
 
 +
 
 +
'''IMPORTANT:''' our goal have shifted a bit on this project. See [https://docs.google.com/document/d/1mibE8nACcmen6paSrqT9JQk5VbuvlFUXI1S93yHCK2I/edit Search-a-licious roadmap architecture notes]
    
=== Goals ===
 
=== Goals ===
Line 40: Line 43:     
=== Data ===
 
=== Data ===
The core datatype will be the ''Product''.
+
The core datatype will be the ''Product''. This data will be stored sparsely (ie, fields that are not set will not be stored in the index) to reduce space.
    
To enable API cases such as partial text search, a rich autocomplete, and the possibility of eventually serving as a unified read layer, [https://static.openfoodfacts.org/data/data-fields.txt all fields] will be added to the index. Only product names, brands and categories will be indexed for autocomplete queries.
 
To enable API cases such as partial text search, a rich autocomplete, and the possibility of eventually serving as a unified read layer, [https://static.openfoodfacts.org/data/data-fields.txt all fields] will be added to the index. Only product names, brands and categories will be indexed for autocomplete queries.
Line 74: Line 77:  
* [https://openfoodfacts.github.io/api-documentation/#3SEARCHRequests Require documentation]
 
* [https://openfoodfacts.github.io/api-documentation/#3SEARCHRequests Require documentation]
 
* [https://github.com/openfoodfacts/openfoodfacts-server/tree/111e0afdbac3c20ea34652b0b413be58be6dfae5/conf/nginx Need to be exposed in the Nginx configuration]
 
* [https://github.com/openfoodfacts/openfoodfacts-server/tree/111e0afdbac3c20ea34652b0b413be58be6dfae5/conf/nginx Need to be exposed in the Nginx configuration]
* Be prefixed with ''/v3'' (redirecting at the Nginx layer)
+
* Be prefixed with ''/v3'' (redirecting at the Nginx layer, but not using the v3 prefix at the Search Service layer)
    
The proposed API definition is below. Note that the requests are represented as Python objects as used in FastAPI - in reality, this is a JSON payload:
 
The proposed API definition is below. Note that the requests are represented as Python objects as used in FastAPI - in reality, this is a JSON payload:
Line 92: Line 95:  
     field: str
 
     field: str
 
     value: str
 
     value: str
     # One of eq, ne, like, without
+
     # One of eq, ne, like
 
     operator: str = 'eq'
 
     operator: str = 'eq'
   Line 99: Line 102:  
     field: str
 
     field: str
 
     value: float
 
     value: float
     # One of eq, ne, lt, gt, without
+
     # One of eq, ne, lt, gt
 
     operator: str = 'eq'
 
     operator: str = 'eq'
   Line 106: Line 109:  
     field: str
 
     field: str
 
     value: datetime.datetime
 
     value: datetime.datetime
     # One of eq, ne, lt, gt, without
+
     # One of lt, gt
 
     operator: str = 'eq'
 
     operator: str = 'eq'
   Line 132: Line 135:  
   
 
   
 
=== API Discussion ===
 
=== API Discussion ===
The barcode GET API is included to demonstrate how this service could easily replace our existing APIs, but is not intended to be used for the moment.  
+
The barcode GET API is included to demonstrate how this service could easily replace our existing read APIs, but is not intended to be used for the moment.  
    
The remaining APIs have several commonalities:
 
The remaining APIs have several commonalities:
   −
* Only fields with a value are returned in the JSON response, to keep response size down without needing to specify fields manually
   
* An optional ''response_fields'' parameter is provided, to limit the fields in the response further
 
* An optional ''response_fields'' parameter is provided, to limit the fields in the response further
 
* POST is used, to support a complex request body
 
* POST is used, to support a complex request body
Line 144: Line 146:  
* ''like'' operator, which does not need to match to exact fields, but rather will match by use of the [https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-snowball-tokenfilter.html snowball token filter] in Elasticsearch if the field supports it.
 
* ''like'' operator, which does not need to match to exact fields, but rather will match by use of the [https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-snowball-tokenfilter.html snowball token filter] in Elasticsearch if the field supports it.
 
* ''gt'', ''lt'' (greater than, less than) operators are provided.
 
* ''gt'', ''lt'' (greater than, less than) operators are provided.
* To maintain backwards compatibility, a ''without'' operator is provided, in which case the value will be ignored. Note that the value is still required, so a dummy value should be used (it was feared that making this optional would lead to client side bugs). This should be well documented.
+
* There is no support for an OR operator (which is supported in the [[Open Food Facts Search API Version 2|V2 API]]), however clients can perform this logic themselves if they wish through multiple API calls.
 +
 
 +
=== Relationship With Existing APIs ===
 +
The autocomplete API does not overlap with the other APIs. However, the new proposed ''/search'' API has a lot of conceptual overlap with the current search APIs. It is proposed to encourage users to migrate to this new V3 API (and to use it when writing new code) through guidance in the API docs.
 +
 
 +
Furthermore, once the new API has proven its stability, the legacy ''search.pl'' should be able to be switched over (with a translation layer to map between old and new API request and response norms). The v2 API can also be migrated, but will require a solution to supporting the OR operator (potentially dropping support, using another translation layer with multiple requests, or modifying the API).  
 +
 
 +
This work is considered out of scope for this proposal.
    
== Work Plan ==
 
== Work Plan ==
   −
* Completed (locally):
+
* Completed
 
** Product document definition
 
** Product document definition
 
** Bulk import script
 
** Bulk import script
 
** API definition
 
** API definition
** API implementation (partly implemented)
+
** API implementation
 
** Docker, Elasticvue configuration
 
** Docker, Elasticvue configuration
 +
** API tests
 +
** Redis Integration
 
* TODO:
 
* TODO:
** Proposal alignment
+
** Deploy (without any traffic)
** OFF-search repo creation, initial commit
  −
** Finish API implementation, unit tests
  −
** Redis reader
  −
** Deploy (without any traffic) - will need assistance for this, as well as Nginx configuration
   
** Redis writer on the Perl side
 
** Redis writer on the Perl side
 
** Final testing
 
** Final testing
 
** Document API
 
** Document API
 
** Ongoing monitoring
 
** Ongoing monitoring
 +
 +
== Future Work ==
 +
There appears to be a lot of opportunities to utilize this infrastructure further:
 +
 +
* Incorporate the autocomplete API in to the main search bar
 +
* Use the new search API to handle searches on the primary website (should result in improved ranking, matching, lower DB load)
 +
* Robotoff could use the same cluster
 +
* Current search APIs could be switched over (with a translation layer)
 +
 +
[[Category:Search]]
199

edits

Navigation menu