Line 1: |
Line 1: |
| == Overview == | | == Overview == |
| This document serves as a technical proposal for a new search API - V3 ([[Open Food Facts Search API Version 2|see V2 here]]). | | This document serves as a technical proposal for a new search API - V3 ([[Open Food Facts Search API Version 2|see V2 here]]). |
| + | |
| + | |
| + | '''IMPORTANT:''' our goal have shifted a bit on this project. See [https://docs.google.com/document/d/1mibE8nACcmen6paSrqT9JQk5VbuvlFUXI1S93yHCK2I/edit Search-a-licious roadmap architecture notes] |
| | | |
| === Goals === | | === Goals === |
Line 40: |
Line 43: |
| | | |
| === Data === | | === Data === |
− | The core datatype will be the ''Product''. | + | The core datatype will be the ''Product''. This data will be stored sparsely (ie, fields that are not set will not be stored in the index) to reduce space. |
| | | |
| To enable API cases such as partial text search, a rich autocomplete, and the possibility of eventually serving as a unified read layer, [https://static.openfoodfacts.org/data/data-fields.txt all fields] will be added to the index. Only product names, brands and categories will be indexed for autocomplete queries. | | To enable API cases such as partial text search, a rich autocomplete, and the possibility of eventually serving as a unified read layer, [https://static.openfoodfacts.org/data/data-fields.txt all fields] will be added to the index. Only product names, brands and categories will be indexed for autocomplete queries. |
Line 92: |
Line 95: |
| field: str | | field: str |
| value: str | | value: str |
− | # One of eq, ne, like, without | + | # One of eq, ne, like |
| operator: str = 'eq' | | operator: str = 'eq' |
| | | |
Line 99: |
Line 102: |
| field: str | | field: str |
| value: float | | value: float |
− | # One of eq, ne, lt, gt, without | + | # One of eq, ne, lt, gt |
| operator: str = 'eq' | | operator: str = 'eq' |
| | | |
Line 106: |
Line 109: |
| field: str | | field: str |
| value: datetime.datetime | | value: datetime.datetime |
− | # One of eq, ne, lt, gt, without | + | # One of lt, gt |
| operator: str = 'eq' | | operator: str = 'eq' |
| | | |
Line 136: |
Line 139: |
| The remaining APIs have several commonalities: | | The remaining APIs have several commonalities: |
| | | |
− | * Only fields with a value are returned in the JSON response, to keep response size down without needing to specify fields manually
| |
| * An optional ''response_fields'' parameter is provided, to limit the fields in the response further | | * An optional ''response_fields'' parameter is provided, to limit the fields in the response further |
| * POST is used, to support a complex request body | | * POST is used, to support a complex request body |
Line 144: |
Line 146: |
| * ''like'' operator, which does not need to match to exact fields, but rather will match by use of the [https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-snowball-tokenfilter.html snowball token filter] in Elasticsearch if the field supports it. | | * ''like'' operator, which does not need to match to exact fields, but rather will match by use of the [https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-snowball-tokenfilter.html snowball token filter] in Elasticsearch if the field supports it. |
| * ''gt'', ''lt'' (greater than, less than) operators are provided. | | * ''gt'', ''lt'' (greater than, less than) operators are provided. |
− | * To maintain backwards compatibility, a ''without'' operator is provided, in which case the value will be ignored. Note that the value is still required, so a dummy value should be used (it was feared that making this optional would lead to client side bugs). This should be well documented.
| |
| * There is no support for an OR operator (which is supported in the [[Open Food Facts Search API Version 2|V2 API]]), however clients can perform this logic themselves if they wish through multiple API calls. | | * There is no support for an OR operator (which is supported in the [[Open Food Facts Search API Version 2|V2 API]]), however clients can perform this logic themselves if they wish through multiple API calls. |
| | | |
Line 156: |
Line 157: |
| == Work Plan == | | == Work Plan == |
| | | |
− | * Completed (locally): | + | * Completed |
| ** Product document definition | | ** Product document definition |
| ** Bulk import script | | ** Bulk import script |
| ** API definition | | ** API definition |
− | ** API implementation (partly implemented) | + | ** API implementation |
| ** Docker, Elasticvue configuration | | ** Docker, Elasticvue configuration |
| + | ** API tests |
| + | ** Redis Integration |
| * TODO: | | * TODO: |
− | ** Proposal alignment
| + | ** Deploy (without any traffic) |
− | ** OFF-search repo creation, initial commit
| |
− | ** Finish API implementation, unit tests
| |
− | ** Redis reader
| |
− | ** Deploy (without any traffic) - will need assistance for this, as well as Nginx configuration | |
| ** Redis writer on the Perl side | | ** Redis writer on the Perl side |
| ** Final testing | | ** Final testing |
Line 180: |
Line 179: |
| * Robotoff could use the same cluster | | * Robotoff could use the same cluster |
| * Current search APIs could be switched over (with a translation layer) | | * Current search APIs could be switched over (with a translation layer) |
| + | |
| + | [[Category:Search]] |