Difference between revisions of "Project:Open-Prices"

From Open Food Facts wiki
Jump to navigation Jump to search
(add Slack channel. add link to GDPR requests)
 
(39 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 +
 +
 +
 
The goal of this project is to store prices of food products and to make them available to the public, through a REST API and web interface.
 
The goal of this project is to store prices of food products and to make them available to the public, through a REST API and web interface.
  
 
The data is licensed under OdBL 1.0, images (proof) are licensed under CC-BY-SA 4.0.
 
The data is licensed under OdBL 1.0, images (proof) are licensed under CC-BY-SA 4.0.
 +
 +
== Important links & documents ==
 +
 +
* App: https://prices.openfoodfacts.org/app/
 +
* API: https://prices.openfoodfacts.org/api/docs
 +
* Github repo: https://github.com/openfoodfacts/open-prices
 +
* Basic documentation: https://openfoodfacts.github.io/open-prices/
  
 
== Data sources ==
 
== Data sources ==
  
−
We currently consider the following data sources:
+
We currently consider the following price data sources:
  
−
* crowdsourced price data, with if possible a proof (either a store receipt, or a picture of the product with its price tag). Prices obtained through [[GDPR request]] to stores are also welcome.
+
{| class="wikitable"
−
* prices provided by retailers, through the API
+
|+ Caption text
−
* prices obtained from users on the web using a web extension
+
|-
 +
! Source !! Type !! Addition method !! Details
 +
|-
 +
| Receipts & price tags || Crowdsourced || Web or Mobile || Filling in a simple form. With a proof (either a store receipt, or a picture of the product(s) with its price tag(s))
 +
|-
 +
| GDPR requests || Crowdsourced || API || Obtained through [[GDPR request]] to stores (fidelity cards :)
 +
|-
 +
| Shop imports || Retails || API || Provided directly by retailers
 +
|-
 +
| Web extension || Crowdsourced || API || Current status: just an idea
 +
|}
  
 
Crowdsourcing the data collect has the advantage of mobilizing the community around the project and of getting traction and visibility. We want to reach a critical mass of data to make the project useful, and we think that crowdsourcing is the best way to achieve this goal. Scan parties in stores can be organized to collect data.
 
Crowdsourcing the data collect has the advantage of mobilizing the community around the project and of getting traction and visibility. We want to reach a critical mass of data to make the project useful, and we think that crowdsourcing is the best way to achieve this goal. Scan parties in stores can be organized to collect data.
  
−
We currently don't allow scraping as a data source ''for a first version'', as it bears some legal concerns and technical challenges to overcome:
+
We currently '''don't allow scraping''' as a data source ''for a first version'', as it bears some legal concerns and technical challenges to overcome:
−
 
 
 
* the database protection law in Europe may prevent us from extracting the price from the website, if we don't have the authorization of the website owner
 
* the database protection law in Europe may prevent us from extracting the price from the website, if we don't have the authorization of the website owner
 
* the price can be different depending on the user selected store. Furthermore, some retailers (in the UK for example) start to adapt their prices depending on the user profile (if they are identified as a regular customer, or if they are a new customer). This makes it difficult to have a consistent price for a product.
 
* the price can be different depending on the user selected store. Furthermore, some retailers (in the UK for example) start to adapt their prices depending on the user profile (if they are identified as a regular customer, or if they are a new customer). This makes it difficult to have a consistent price for a product.
Line 20: Line 39:
 
== Current status ==
 
== Current status ==
  
−
A POC developed during OFFdays 2023 is currently running here : https://open-prices.osc-fr1.scalingo.io ([https://github.com/raphodn/open-prices-poc Github code])
+
=== Tech Stack ===
 +
 
 +
The project is deployed on https://prices.openfoodfacts.org. It is built with:
 +
 
 +
{| class="wikitable"
 +
|-
 +
! Name (& link) !! Technology !! Code !! Extra info
 +
|-
 +
| [https://prices.openfoodfacts.org/api/docs Backend] || Python <s>FastAPI</s> Django API || https://github.com/openfoodfacts/open-prices ||
 +
|-
 +
| [https://prices.openfoodfacts.org Frontend] || Javascript Vue.js 3 + Vuetify || https://github.com/openfoodfacts/open-prices-frontend ||
 +
|}
 +
 
 +
=== OFF integrations ===
 +
 
 +
{| class="wikitable"
 +
|-
 +
! Name (& link) !! Description !! Code !! Extra info
 +
|-
 +
| Mobile integration || Integration in the official OFF mobile app || https://github.com/openfoodfacts/smooth-app ||
 +
|-
 +
| Openfoodfacts-explorer || || ||
 +
|}
 +
 
 +
We're not displaying the price on Open Food Facts website yet, as it has to be thoroughly considered first (the website attracts a lot of traffic).
  
−
A migration is in progress to host the code on https://github.com/openfoodfacts/open-prices and deploy the service on https://openprices.org (on Open Food Facts server).
+
=== Known re-uses ===
  
−
We're not displaying the price on Open Food Facts website yet, as it has to be thoroughly considered first (and as Open Food Facts website attracts a lot of traffic).
+
{| class="wikitable"
 +
|-
 +
! Name (& link) !! Description !! Code !! Main contributor
 +
|-
 +
| [https://dmayaux.shinyapps.io/open_prices Location map] || A world map of all the contributions in Open Prices || || Damien M
 +
|-
 +
| [https://leprixdescarottes.fr/ Price of carrots] || Simple price index of fruits and vegetables (in french) || https://github.com/TTalex/leprixdescarottes || Alex B
 +
|-
 +
| [https://www.doyouwantyourreceipt.info Do you want your receipt?] || Comparison between 2 shops || https://github.com/qloridant/shop_prices_comparaison || Quentin L
 +
|-
 +
| [https://project.med3d.eu/fooddata/price_plot.html?ean=5000159451666 Price graph] || || ||
 +
|}
  
 
== Challenges ==
 
== Challenges ==
Line 31: Line 85:
 
* Find a way to collect efficiently prices (probably through a mobile app)
 
* Find a way to collect efficiently prices (probably through a mobile app)
  
−
== Documentation and communication ==
+
== 🎯 Roadmap ==
 +
* [[Artificial Intelligence/Open Prices]]
 +
 
 +
== Usage ideas ==
 +
 
 +
{| class="wikitable"
 +
|- style="background-color:#c0c0c0;"
 +
! Description
 +
! Usage
 +
|-
 +
| For a given product, price trends
 +
|
 +
|-
 +
| For a given product, price comparison between stores
 +
|
 +
|-
 +
| For a given product, price comparison for the same store chain (geographic price differences)
 +
|
 +
|-
 +
| For a given product, list of stores where it is sold
 +
|
 +
|-
 +
| Better compare similar products (in addition to Nutri-score, NOVA & Eco-score) - price per kilo - Should I take the national brand, the retailer brand, the budget retailer brand ?
 +
|
 +
|-
 +
| See if ingredient composition align linearly with price (eg: more meat = pricier)
 +
|
 +
|-
 +
| Support shrinkflation investigation (compare products after a change of size/price)
 +
|
 +
|-
 +
| Support inflation investigation (evolution of prices in time)
 +
|
 +
|-
 +
| See if quality labels (eg organic) have an effect on prices
 +
|
 +
|-
 +
| Leverage the other Open Food Facts databases (products, beauty, pet food) to store prices for products other than food
 +
|
 +
|-
 +
| Help gather information about which products are available in your nearby store, as a side effect of price collection
 +
|
 +
|-
 +
| Keep a history of your purchases via your uploaded receipts
 +
|
 +
|-
 +
| Help gather OpenStreetMap POI information and add missing shops
 +
|
 +
|}
 +
 
 +
== How can I help Open Prices ? ==
 +
 
 +
* Adding prices
 +
* Feedback, bugs, ideas (Slack, Github, weekly Wednesday meeting)
 +
* Spread the word to potential third-party uses
 +
* GDPR: requesting your loyalty card data ([[GDPR request]]) + helping document the loyalty card data process
 +
 
 +
== 📚 Documentation ==
 +
 
 +
* https://openfoodfacts.github.io/open-prices/
 +
* https://github.com/openfoodfacts/open-prices/wiki
 +
 
 +
== FAQ ==
 +
 
 +
=== What happens when a shop goes out of business and gets removed from OSM? ===
 +
 
 +
Discussion: https://openfoodfacts.slack.com/archives/C02Q6KM7F/p1724084895885059
 +
 
 +
=== Is it possible to upload a pdf receipt with the new version of the OFF app? ===
 +
 
 +
Discussion: https://openfoodfacts.slack.com/archives/C02Q6KM7F/p1723559332138529
 +
 
 +
=== i've uploaded a receipt and added a couple of the items from it, but may have to finish before i've gotten to all of them - is it possible to resume at a later point or would i have to re-upload the receipt and go through it again? ===
 +
 
 +
Discussion: https://openfoodfacts.slack.com/archives/C02Q6KM7F/p1722881061582649
 +
 
 +
=== How can I find stores in Open Prices just with OSM node? ===
 +
 
 +
* https://openfoodfacts.slack.com/archives/C02Q6KM7F/p1717767803140249
 +
 
 +
=== Other ===
 +
 
 +
see https://github.com/openfoodfacts/open-prices/discussions
 +
 
 +
== Edge cases ==
 +
=== Recycling in Norway ===
 +
In Norway the price tags for beverages packaged in PET plastic bottles don't include the recycling deposit in the price. That's usually displayed with small text separate from the main price. Because of this it can be hard to make out from product shelf photos what the total price is. Up till now I've not added up the total price of product + deposit (except for when adding receipts), but the deposit is usually the same every time based on the drink's volume. Less than 500ml is 2 NOK and more than that is 3 NOK. Should there maybe be a separate field for the recycle deposit to not confuse potential future OCR bots trying to extract prices from the images, or should I start including the deposit in the total price going forward?
 +
https://openfoodfacts.slack.com/archives/C02Q6KM7F/p1722185150520809
 +
 
 +
== Public data about prices ==
 +
=== France ===
 +
* An resource I find really interesting: https://rnm.franceagrimer.fr/prix?M3027:12MOIS
 +
France Agri Mer (a French national agency) publishes regularly prices of vegetables and fruits, in bulk or in stores.
 +
=== Israel ===
 +
 
 +
== Papers to investigate ==
 +
* https://www.semantic-web-journal.net/system/files/swj3700.pdf
 +
* https://dl.acm.org/doi/abs/10.1145/3605098.3636200
 +
* https://www.linkedin.com/feed/update/urn:li:activity:7198558427755765760/ A maybe interesting idea: instead of the receipt, take a photo of product list + prices on the screen of self checkout automaton
 +
 
 +
== Marketing & Reads ==
 +
* https://www.reddit.com/r/france/comments/1cyrov7/mon_supermarch%C3%A9_me_vole_%C3%A9pisode_2/
 +
* https://www.ft.com/content/0ea5410f-da90-42dc-8be4-62bbdbd47762?accessToken=zwAAAY-qfgM4kc8OpUEP2pBC3NOL5GK729R3Yg.MEYCIQD3lF9nUmbAqCEp-Qw5uY3S22fYpzH0is-xkluZ9V_8SAIhAKvUGDKvM5oPHuel4ACDMVT1qWblekCH4HQF4qCaN-O5&segmentId=e95a9ae7-622c-6235-5f87-51e412b47e97&shareType=enterprise&shareId=1984a8fa-da57-40e2-a531-46926df0db05
 +
*  https://vm.tiktok.com/ZGeyRoQkX/
 +
* https://www.bbc.com/news/business-65833619
 +
* Inflation, product-price comparisons, relevés de prix (the first 30 minutes) (French TV)
 +
* https://actu.6play.fr/information-et-magazines/capital-discounters-vs-supermarches-diffusion-du-7-janvier-2024-sur-m6
 +
* https://x.com/GoogleTrends/status/1722324130308235485?t=kK49uMFhPLX3SdVf3VoCmA&s=31
 +
* Interesting thread & info by the French stats office (Insee): https://twitter.com/InseeFr/status/1719300014810235024
 +
* Le Monde: "Le couple s'organise pour rentabiliser au maximum les achats. Après Aldi, ils filent à Lidl, puis à Action. C'est la tournée hebdomadaire des enseignes discount. « C'est le parcours du combattant tous les samedis », résument Pascal et Danièle Landouzy. Ils ne sont pas les seuls à passer d'un magasin à l'autre à la recherche des meilleurs prix."
 +
* Very interesting feedbacks and information in this thread about prices in Austria, Germany, Slovenia, ...
 +
https://mastodon.gamedev.place/@badlogic/111071396799790275
 +
* Distributors ripping off the packaging to hide the advised price from Danone products: https://www.olivierdauvers.fr/wp-content/uploads/2023/09/20230829_202151.jpg
 +
* Price gaps between healthy and unhealtyh in the UK: https://openfoodfacts.slack.com/files/U02QC4WE8/FKXMQGR5J/the_growing_price_gap_between_more_and_less_healthy_foods_-_analysis_of_a_novel_longitudinal_uk_dataset.pdf
 +
* Insee: https://www.insee.fr/fr/metadonnees/source/s1308#documentation - Source - Indice des prix à la consommation / IPC | Insee
 +
* Are Vegetables Actually Cheaper Than Shite Food? - VICE: https://www.vice.com/en_uk/article/are-vegetables-actually-cheaper-than-shite-food?utm_source=vicefbuk&utm_campaign=global
 +
* http://www.lesechos.fr/economie-france/conjoncture/021840877689-alimentation-logement-habillement-les-ecarts-de-prix-paris-regions-1213948.php#xtor=CS1-25
 +
=== SDKs ===
 +
* https://github.com/aleene/OFFPricesAPIs
 +
 
 +
=== Forum thread ===
 +
* https://forum.openfoodfacts.org/t/introducing-open-prices/501
 +
 
 +
=== Others ===
 +
* new app with users who crowdsource prices https://www.radins.com/shopping/courses/actualites/reprice-lapplication-qui-permet-de-comparer-les-prix-en-supermarche/52523
 +
*  (https://heisse-preise.io/)
 +
* Where do you locate food sold in trains ?
 +
* 2016: Numbeo has a crowdsourced database of prices and charges a yearly subscription for its API : https://www.numbeo.com/common/api.jsp
 +
Example : https://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=France&country2=Russia&city1=Paris&city2=Moscow&tracking=getDispatchComparison
 +
0.76 € for a water bottle in Paris in a store. 0,62€ in Moscow
 +
We could offer the same kind of data the only difference is that they have… 355,205 contributors
 +
* 2016: http://www.procen.ru a Russian noncommercial site for people to enter prices and search them
 +
* https://play.google.com/store/apps/details?id=be.heyman.android.inflacheck
 +
* https://play.google.com/store/apps/details?id=com.skerou.android "Skerou est la première application qui permet à chaque consommateur de garder la trace de ses achats, grâce à une technologie révolutionnaire.
 +
Prenez une simple photo de votre ticket de caisse de supermarché (de supermarché uniquement, on s'occupe juste de ces courses). Et hop, vous retrouvez un peu après l'ensemble de vos achats, en images.
 +
A terme, l'historique de vos achats vous permettra, outre de les télécharger (après tout ce sont vos données à vous), de préparer vos listes de courses à partir de vos achats passés, et aussi de valider des promotions qui vous rapportent du cash sur votre compte en banque."
 +
 
 +
== Get in touch ==
 +
{{Box
 +
| 1    =  Slack channel
 +
| 2    =  [https://openfoodfacts.slack.com/messages/C02Q6KM7F/ #prices]
 +
}}
 +
 
 +
* Slack channel: [https://app.slack.com/client/T02KVRT1Q/C02Q6KM7F #prices]
 +
 
 +
== 📆 Weekly meeting on Wesdnesdays ==
 +
* Weekly meeting: every Wednesday at 2PM (CET), feel free to join if you're interested in the project: https://meet.google.com/oin-hiqp-tmd
 +
 
 +
== 🎨 Design & Mockups ==
 +
* Visuals & marketing assets folder for Open Prices on Canva: https://www.canva.com/folder/FAF65Zi0H8w
 +
* Mockups on Figma: https://www.figma.com/design/1mEjKuxKHxop3X60R6dr5s/OpenFoodFacts-%7C-Open-Price?node-id=1-29&t=dr1IHqIIHAcKFyqr-0
 +
* Price addition mobile mockups on Figma: https://www.figma.com/design/nFMjewFAOa8c4ahtob7CAB/Mobile-App-Design-(Quentin)?node-id=5816-22697&t=UoETOgDYDten9odq-0
 +
* Price visualization mockups on Figma: https://www.figma.com/design/nFMjewFAOa8c4ahtob7CAB/Mobile-App-Design-(Quentin)?node-id=5787-19012&t=UoETOgDYDten9odq-0
 +
 
 +
== Timeline ==
  
−
Slack channel: [https://app.slack.com/client/T02KVRT1Q/C02Q6KM7F #prices]
+
{| class="wikitable"
 +
|- style="background-color:#c0c0c0;"
 +
! Date
 +
! Description
 +
|-
 +
| 08/11/2023
  
−
Weekly meeting: every Wednesday at 2PM (CET), feel free to join if you're interested in the project: https://meet.google.com/oin-hiqp-tmd
+
[[Category:Project]]
 +
[[Category:Prices]]

Latest revision as of 21:31, 12 November 2024


The goal of this project is to store prices of food products and to make them available to the public, through a REST API and web interface.

The data is licensed under OdBL 1.0, images (proof) are licensed under CC-BY-SA 4.0.

Important links & documents

Data sources

We currently consider the following price data sources:

Caption text
Source Type Addition method Details
Receipts & price tags Crowdsourced Web or Mobile Filling in a simple form. With a proof (either a store receipt, or a picture of the product(s) with its price tag(s))
GDPR requests Crowdsourced API Obtained through GDPR request to stores (fidelity cards :)
Shop imports Retails API Provided directly by retailers
Web extension Crowdsourced API Current status: just an idea

Crowdsourcing the data collect has the advantage of mobilizing the community around the project and of getting traction and visibility. We want to reach a critical mass of data to make the project useful, and we think that crowdsourcing is the best way to achieve this goal. Scan parties in stores can be organized to collect data.

We currently don't allow scraping as a data source for a first version, as it bears some legal concerns and technical challenges to overcome:

  • the database protection law in Europe may prevent us from extracting the price from the website, if we don't have the authorization of the website owner
  • the price can be different depending on the user selected store. Furthermore, some retailers (in the UK for example) start to adapt their prices depending on the user profile (if they are identified as a regular customer, or if they are a new customer). This makes it difficult to have a consistent price for a product.

Current status

Tech Stack

The project is deployed on https://prices.openfoodfacts.org. It is built with:

Name (& link) Technology Code Extra info
Backend Python FastAPI Django API https://github.com/openfoodfacts/open-prices
Frontend Javascript Vue.js 3 + Vuetify https://github.com/openfoodfacts/open-prices-frontend

OFF integrations

Name (& link) Description Code Extra info
Mobile integration Integration in the official OFF mobile app https://github.com/openfoodfacts/smooth-app
Openfoodfacts-explorer

We're not displaying the price on Open Food Facts website yet, as it has to be thoroughly considered first (the website attracts a lot of traffic).

Known re-uses

Name (& link) Description Code Main contributor
Location map A world map of all the contributions in Open Prices Damien M
Price of carrots Simple price index of fruits and vegetables (in french) https://github.com/TTalex/leprixdescarottes Alex B
Do you want your receipt? Comparison between 2 shops https://github.com/qloridant/shop_prices_comparaison Quentin L
Price graph

Challenges

  • On receipt images, there may be some personal data (name, fidelity card number, etc.). We need to find a way to remove this data before publishing the image.
  • Find a way to collect efficiently prices (probably through a mobile app)

🎯 Roadmap

Usage ideas

Description Usage
For a given product, price trends
For a given product, price comparison between stores
For a given product, price comparison for the same store chain (geographic price differences)
For a given product, list of stores where it is sold
Better compare similar products (in addition to Nutri-score, NOVA & Eco-score) - price per kilo - Should I take the national brand, the retailer brand, the budget retailer brand ?
See if ingredient composition align linearly with price (eg: more meat = pricier)
Support shrinkflation investigation (compare products after a change of size/price)
Support inflation investigation (evolution of prices in time)
See if quality labels (eg organic) have an effect on prices
Leverage the other Open Food Facts databases (products, beauty, pet food) to store prices for products other than food
Help gather information about which products are available in your nearby store, as a side effect of price collection
Keep a history of your purchases via your uploaded receipts
Help gather OpenStreetMap POI information and add missing shops

How can I help Open Prices ?

  • Adding prices
  • Feedback, bugs, ideas (Slack, Github, weekly Wednesday meeting)
  • Spread the word to potential third-party uses
  • GDPR: requesting your loyalty card data (GDPR request) + helping document the loyalty card data process

📚 Documentation

FAQ

What happens when a shop goes out of business and gets removed from OSM?

Discussion: https://openfoodfacts.slack.com/archives/C02Q6KM7F/p1724084895885059

Is it possible to upload a pdf receipt with the new version of the OFF app?

Discussion: https://openfoodfacts.slack.com/archives/C02Q6KM7F/p1723559332138529

i've uploaded a receipt and added a couple of the items from it, but may have to finish before i've gotten to all of them - is it possible to resume at a later point or would i have to re-upload the receipt and go through it again?

Discussion: https://openfoodfacts.slack.com/archives/C02Q6KM7F/p1722881061582649

How can I find stores in Open Prices just with OSM node?

Other

see https://github.com/openfoodfacts/open-prices/discussions

Edge cases

Recycling in Norway

In Norway the price tags for beverages packaged in PET plastic bottles don't include the recycling deposit in the price. That's usually displayed with small text separate from the main price. Because of this it can be hard to make out from product shelf photos what the total price is. Up till now I've not added up the total price of product + deposit (except for when adding receipts), but the deposit is usually the same every time based on the drink's volume. Less than 500ml is 2 NOK and more than that is 3 NOK. Should there maybe be a separate field for the recycle deposit to not confuse potential future OCR bots trying to extract prices from the images, or should I start including the deposit in the total price going forward? https://openfoodfacts.slack.com/archives/C02Q6KM7F/p1722185150520809

Public data about prices

France

France Agri Mer (a French national agency) publishes regularly prices of vegetables and fruits, in bulk or in stores.

Israel

Papers to investigate

Marketing & Reads

https://mastodon.gamedev.place/@badlogic/111071396799790275

SDKs

Forum thread

Others

Example : https://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=France&country2=Russia&city1=Paris&city2=Moscow&tracking=getDispatchComparison 0.76 € for a water bottle in Paris in a store. 0,62€ in Moscow We could offer the same kind of data the only difference is that they have… 355,205 contributors

Prenez une simple photo de votre ticket de caisse de supermarché (de supermarché uniquement, on s'occupe juste de ces courses). Et hop, vous retrouvez un peu après l'ensemble de vos achats, en images. A terme, l'historique de vos achats vous permettra, outre de les télécharger (après tout ce sont vos données à vous), de préparer vos listes de courses à partir de vos achats passés, et aussi de valider des promotions qui vous rapportent du cash sur votre compte en banque."

Get in touch

Slack channel

📆 Weekly meeting on Wesdnesdays

🎨 Design & Mockups

Timeline

Date Description
08/11/2023
Retrieved from "https://wiki.openfoodfacts.org/index.php?title=Project:Open-Prices&oldid=25105"