Difference between revisions of "Events system"
Jump to navigation
Jump to search
(Created page with "= Introduction = This is a brainstorm page, to find a good solution to monitor events in Open Food Facts. = What kind of events do we want to track = == Info events == Som...") |
(→Info events: ideas) |
||
| (3 intermediate revisions by one other user not shown) | |||
| Line 2: | Line 2: | ||
This is a brainstorm page, to find a good solution to monitor events in Open Food Facts. | This is a brainstorm page, to find a good solution to monitor events in Open Food Facts. | ||
| + | |||
| + | * *We need an event and action tracking and management system for the Platform for Producers: to monitor imports, verify new users etc.* | ||
| + | * We can also consider other use cases to see if we can create a solution that could work for them too | ||
= What kind of events do we want to track = | = What kind of events do we want to track = | ||
| Line 12: | Line 15: | ||
** e.g. an user registered, a product was added or edited... | ** e.g. an user registered, a product was added or edited... | ||
** could be useful to have recent changes filterable by facets | ** could be useful to have recent changes filterable by facets | ||
| − | * | + | * '''Processes: some process started, failed, or completed''' |
** -> this is the most pressing need, for which we don't have a good solution today | ** -> this is the most pressing need, for which we don't have a good solution today | ||
** Regular maintenance tasks (e.g. daily processing of something) | ** Regular maintenance tasks (e.g. daily processing of something) | ||
| Line 18: | Line 21: | ||
** User requested tasks | ** User requested tasks | ||
*** Imports and exports on the Producers Platform | *** Imports and exports on the Producers Platform | ||
| + | ** Prospective data check reveals problem. | ||
| + | * Data Import | ||
| + | ** eg USDA Branded foods API is checked and reveals new info | ||
| + | * Errors | ||
| + | ** Software errors | ||
| + | *** MongoDB down | ||
| + | ** Product quality issues | ||
== Events that require an action == | == Events that require an action == | ||
| Line 45: | Line 55: | ||
* Sentry | * Sentry | ||
| + | ** Perl client: https://docs.sentry.io/clients/perl/ | ||
* Matomo | * Matomo | ||
* Minion | * Minion | ||
** We could turn maintenance tasks into Minion jobs | ** We could turn maintenance tasks into Minion jobs | ||
| + | * Nagios - nagios itself is having problems, but a nagios-like tool which: | ||
| + | ** has script returning GOOD, WARN, BAD | ||
| + | ** has a display | ||
== OFF events system == | == OFF events system == | ||
Latest revision as of 18:10, 8 September 2023
Introduction
This is a brainstorm page, to find a good solution to monitor events in Open Food Facts.
- *We need an event and action tracking and management system for the Platform for Producers: to monitor imports, verify new users etc.*
- We can also consider other use cases to see if we can create a solution that could work for them too
What kind of events do we want to track
Info events
Something happened.
- User actions
- e.g. an user registered, a product was added or edited...
- could be useful to have recent changes filterable by facets
- Processes: some process started, failed, or completed
- -> this is the most pressing need, for which we don't have a good solution today
- Regular maintenance tasks (e.g. daily processing of something)
- Adhoc maintenance tasks (e.g. re-processing of products to apply a new taxonomy)
- User requested tasks
- Imports and exports on the Producers Platform
- Prospective data check reveals problem.
- Data Import
- eg USDA Branded foods API is checked and reveals new info
- Errors
- Software errors
- MongoDB down
- Product quality issues
- Software errors
Events that require an action
Something needs to be done.
- User request that needs to be manually validated
- exporting products from the Producers Platform to the public database (at least for the first time)
Desirable features
- Event log: being able to see individual events
- Filtering by event type, country, product, organization, user etc.
- Integration with OFF
- It's not just about generating events, but also being able to display events directly into OFF
- e.g. show status of user generated tasks
- It's not just about generating events, but also being able to display events directly into OFF
- Aggregated metrics, graphs etc. of events
- Nice to have, but not the main focus.
- Event status for events that require an action (e.g. open / close)
- Notifications (by mail, Slack channel etc.) for specific types of events
- Public events and private events
Possible options
External events system
- Sentry
- Perl client: https://docs.sentry.io/clients/perl/
- Matomo
- Minion
- We could turn maintenance tasks into Minion jobs
- Nagios - nagios itself is having problems, but a nagios-like tool which:
- has script returning GOOD, WARN, BAD
- has a display
OFF events system
- Events collection in MongoDB
- Makes it easy to have lots of OFF specific facets for events
Notes
- We probably do not want to turn the events system in an analytics system
- Keep things (e.g. the events database) relatively small