We do not want to walk through seven lists every morning. Let the app
say what needs attention, and be quiet when there is nothing.
Almost every business system we come across has the same problem: somebody has to walk through seven lists every morning to see whether anything is wrong. So for a drinks wholesaler with an off licence we built an alerting layer into their purchasing app. Seventeen kinds of signal, recalculated every hour, and they disappear as soon as the cause is gone. It is running in production. The client stays anonymous and there are no amounts, prices or margins here.
Do not store what you can calculate
The first choice was the most important one: an alert is not an event we write down, it is a conclusion drawn from the current state.
Every hour, and on top of that after every synchronisation with the accounting and after every successful invoice upload, a task looks at the state of things and works out which alerts exist now. Each alert gets a key: the kind plus the thing it is about. That list is compared with what is in the database.
- A key that is new becomes an alert.
- A key that has disappeared gets a date on "resolved" and leaves the page.
- A key that was already there stays the same alert, but its contents are refreshed.
That last point is what keeps it calm. If an alert about prices is open and another invoice comes in, the count inside the alert changes, but no second mail goes out. You have already seen it.
Deduplication is not left to the code but to the database: a unique index on administration plus key, valid only while the alert is unresolved. The same alert cannot exist twice, and if a problem comes back later, a new one is allowed. One line in a migration instead of half a page of checks.
The rollout that nearly became a flood
This is where it went properly wrong in the test environment. Switch this on for an administration with three years of invoices in it and that administration has hundreds of open signals on day one. All of them justified, and all of them old. If those go out in one go, you have burned the system immediately.
The fix is simple and we would do it this way everywhere: on the first round per administration, and after importing old data, the existing state is recorded as "known". No mail, not in the counter, not on the page. Only what arises after that is an alert.
So you start with an empty screen that fills up with what actually happens. Anyone who wants the old state has the ordinary overviews for that.
Kinds, and who gets what
The seventeen kinds cover three things: the integration itself (sync failed, access expires within a week, daily limit nearly used up), the daily work (an invoice ready to process, a new order, an invoice that does not add up to the cent) and purchasing (an item more than 10 percent more expensive than last time, an item more than 30 percent cheaper, another supplier at least 5 percent lower).
Each kind has a severity: bad, watch out, or for information only. And every user chooses per kind and per channel whether they want to be told. The administrator keeps the whole matrix. That looks like a lot of switches for a small company, and the alternative is that everybody gets everything and so nobody looks any more.
Two kinds were taken back out on request, and that matters just as much as what went in. Payments and booking the goods receipt happen at this client through their own way of working, outside the app. So the app has no business squeaking about those. The cases stay in the code for alerts already in the database, but nothing is collected for them any more.
Two that do not resolve themselves
Not everything is a conclusion drawn from the state. One alert is about something a person has to do outside the app: an order where free bottles were booked at the full price, which somebody has to deduct on the purchase invoice. The app cannot watch that happen. So that alert stays open until somebody ticks it off.
One alert actively does something: while an "unusually low purchase price" is open, the app does not book that invoice into the accounting. Nine times out of ten that is a wrong unit, a case read as a single bottle, and then you want it to stop before it lands in the administration.
An alert is a task
Halfway through the build it turned out the question was not "what is going on" but "whose is this". So every alert can be assigned to someone. Each kind has a default person, you can pass an alert on with a message, or pick it up yourself with one button. That goes into the log. The page and the bell show what is yours, or nobody's, first.
Not mailing is a feature
Three things we built this way on purpose:
A round without news is silent. Quiet hourly runs do not reach the log and are cleaned up after a week. Otherwise your log fills up with "nothing new".
A failed mail never fails the task. It is logged and retried up to three times. If it still has not worked after two days, we let it go. Mailing a two day old alert after all helps nobody.
At most twenty five alerts in one message. Nobody reads more than that, and the rest is in the app.
Next to mail there is Telegram, switchable per user and per kind. Linking uses a one time code valid for a quarter of an hour, of which we only keep the hashed form. If somebody blocks the bot, we remove the link ourselves and that person sees it in the app. The round does not fail over it, and the mail goes out as usual. The daily summary at 07:00 stays mail, and is off by default: you have to want that one.
Common mistakes
Storing alerts as events. Then you have to clean up yourself, and somewhere you will forget. Recalculate them and let whatever no longer exists disappear.
Deduplicating in code. A unique index that only applies to open alerts does it better than any check in your application.
Mailing every existing signal at rollout. Record the starting state as known. Getting that right once is the difference between a system that gets used and a filter rule in somebody's mail program.
Mailing an alert again because the count changed. Refresh the open alert, send nothing.
Switching everything on for everybody. Choosing per person per kind per channel sounds like overkill, and it is not.
Creating an alert nobody can resolve. If there is no button leading to the place where you fix it, it should not be an alert.
What an integration on Exact Online can do is on connecting Exact Online. If you want something like this in your own system, have a look at custom software.