Patterns for Decoupling in Distributed Systems: Passage of Time Event

Replace cron jobs and scheduled commands, with an agnostic event to indicate the passage of time.

Published on 10 May 2019 by @mathiasverraes

Passage of Time Event

Replace cron jobs and scheduled commands, with an agnostic event to indicate the passage of time.

Problem

Many business processes involve some action or job or workload that needs to be performed at a future date. It can be a one-off action or a repeated action, it can be scheduled for a specific date (eg Christmas), a recurring date (the last weekday of the month) or a timeout (thirty days from now).

We want to keep the domain logic for the entire process nicely isolated in a single (micro)service, so that’s also where the logic for this action goes. But there is a small amount of logic that is not inside this service: the fact itself that the action needs to happen, and when it needs to happen.

Cron is the most troublesome: it only works for repeated actions, and offers little control to outside tools. In large systems, the cron file can be a huge mess. More modern solutions allow us to schedule using by calling an API from within our code, so that logic moves to our own service. But fundamentally, the scheduler is still a separate service that knows more than it should. In DDD lingo, the Ubiquitous Language of our business process is leaking. We’ve got more types of elements, and more moving parts in our system.

Solution

The mind switch is to think of the passage of time as just another Domain Event, exactly like all the other events. After all, if we define a Domain Event as a granular point in time where something happened that is relevant to the business, then certainly the next business day, month, or quarter, is extremely relevant.

In the new design, a cron or scheduler emits generic Passage of Time Events at regular intervals, such as a DayHasPassed {date} event at midnight, or a QuarterHasPassed {year, quarter}. All interested services listen for this event. They can react to it, by performing an action, by increasing a counter, or by querying some database and filter by date, to find items that have some work to be done.

Example

Time governs everything. It’s hard to look at a business and not find examples where the Passage of Time pattern is useful: billable hours, subscriptions, resource usage, rentals, accruing interest, payments, reporting, salaries, maintenance schedules, and everything that is cyclic.

On an invoice’s due date, we need to send a reminder to the client. In the old design, we can set up a cron job that calls CheckForOverdueInvoices. In the new design, the cron generates DayHasPassed every midnight. The InvoiceDebtCollection listens to DayHasPassed. Whenever this event arrives, InvoiceDebtCollection queries SELECT * FROM Invoice AS i WHERE DATEDIFF(i.dueDate, NOW()) >= 30. It can either send the reminder directly, but even better would be to emit a new InvoiceBecameOverdue event. Now, the service can listen to its own event and send the reminder. Other services can react to InvoiceBecameOverdue as well, for example to adjust a revenue projection or suspend an account.

Passage of Time Events can also be a little more domain-specific. The NASDAQ’s pre-market trading hours are from 04:00 to 09:30, followed by normal trading hours until 16:00, and after-hours trading until 20:00. On holidays there’s no trading. A service can produce events for each start and close, that can be consumed by many interested services.

Benefits

It’s very explicit. In the above example, an event log would show:

CustomerWasInvoiced
DayHasPassed
DayHasPassed
...
InvoiceBecameOverdue
ReminderWasSent
AccountWasSuspended

It makes writing tests for business processes with a time element very elegant:

Scenario: If no payment is received after 15 days, we suspend the account
Given CustomerWasInvoiced
  And DayHasPassed 
  And DayHasPassed 
  And (...) 
 When DayHasPassed
 Then InvoiceBecameOverdue
  And AccountWasSuspended 
  
Scenario: If payment is received in time, we do nothing
Given CustomerWasInvoiced
  And DayHasPassed 
  And DayHasPassed 
 When PaymentWasMade
 Then Nothing        

(You’d probably want to add a shortcut so you don’t need to write the DayHasPassed line 15 times.)

More importantly, it’s a nicely reactive approach. When a service sends a Command to another service, it needs to know that this other service accepts that Command. When all we do is send generic Passage of Time Events, the scheduler doesn’t need to know who’s listening, how the consumers should react, or if there’s even any service left that listens. All decisions and domain knowledge lie with the reciever. That’s great decoupling.

It’s also temporal decoupling: The scheduler can put this Passage of Time Event on a queue, and it doesn’t matter if the consumers are available at this time to handle the event. For all it cares, a consumer could be down for a few days, and then simply catch up and process all the DayHasPassed events on the queue.

In Event Sourcing, you simply store the DayHasPassed events in the event store, so you can play back entire histories exactly as they happened over time, with no dependency on an external source.

Implementation

The Passage of Time Events use the exact same format and protocol as your other Domain Events. In code, you can make classes for them, which are exactly like the classes for other Domain Events. And you send them over the same messaging infrastructure as everything else. This makes this pattern very cheap in terms of implementation.

Weaknesses

There is some domain knowledge in the other direction: The domain logic to calculate when something needs to happen, such as “every 10th of the month”, has now moved effectively from the cron or scheduler, to the service. In practice, it’s not a big deal, as you can find time libraries that can do the work for you. On the upside, “every 10th of the month except in weekends, during full moons, or during the yearly office party” is something no scheduler will do for you anyway.

It’s worth noting that you don’t want to use the Passage of Time Event pattern for realtime systems. We can easily spare the 365 events per year for DayHasPassed but for systems that deal with minutes or seconds or less, it’s not feasible. Luckily for programmers, in most businesses, the word “immediately” means by “by the end of business day”, or “by the end of the week”, or “before the end of the quarter following the month where the thing happened”.

Patterns for Decoupling in Distributed Systems

  1. Domain Queries
  2. Summary Event
  3. Completeness Guarantee
  4. Passage of Time Event
  5. Fat Event
  6. Explicit Public Events
  7. Segregated Event Layers

Intrigued? I teach workshops about DDD for Messaging Architectures.

Follow @mathiasverraes on Twitter.



Upcoming

Have a look at our upcoming workshops
Past agenda entries...

Blog

2019

2016

2015

2014

2013

2012

2011

Creative Commons License This work by Mathias Verraes is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License.