A Primer on Eventing and Domain Events
Generally speaking an event is a record of something happening. Software will record something to a log and that's the event. They must be
- Write only
- With a Timestamp
- An event type
- Metadata related to the specific type
Events can be used in two ways
- Real time systems
- Back end analysis
Events present a particular difficulty in that an events can interact and override each other. Such as for an ecommerce site, an order may come in as an event, but then an order cancel event may also come and together there is no order.
But there was an order. For a period of time this order existed in the system, between the time between order made and order cancelled.
Systems that process events in real time need to be able to handle these types of cases, and back end systems that run a reporting should include the order during the time the order existed.
So why do I as a software engineer or data professional care about events and how do I start to deal with this type of dataset?
What is an Event?
An event is simply a record of something happening. This may be something a user did, the system did, or more trivially some logging if database access, security events, or exceptions.
Typically they are stored and transmitted in a variety of methods including
- Ad hoc storage to Database Tables
- Logging to flat files or Log aggregation systems
- Sent to a queuing system such as Kafka
Examples
Some typical examples are
Order was Placed - What items were ordered, when and by which customer.
Database was queried - A log that may be captured and used by an engineer to debug a production system.
Product Recommendations successfully re-generated and available for real time use.
Domain Events?
Domain Events are more specific they are geared towards significant and important events within a system that can be used to understand the main goals and calculate key KPIs.
For example, a user placing an order would absolutely be a domain event, but querying a database would not be a domain event. These are very specific to the team responsible for implementing and tracking these events and although some things like orders are likely to be applicable from company to company, every company is going to set these up differently.
As a side note, domain thinking by itself can help a company break their system up into manageable pieces where each is responsible for a single domain.
Domain events come into play because a domain must send data out for other teams to use, and teams don't care about "querying a database", but absolutely care about "a user placing an order". This approach to eventing allows emitting of events so that other systems can listen proactively to necessary events without having to also sort through a bunch of unrelated data.
Web Engineering
One of the biggest impacts to using domain events widely is that web teams can implement their underlying databases however they want to.
A common pattern with reporting at web companies is that the web engineering team transmits in the entire database downstream for use in analytics and reporting. I've seen this mechanism in a few ways, but the problem is that if the web team want to make a simple change - add a column, or rename a table, or split a table, they have to negotiate with downstream uses of that table.
Events help get away from this anti-pattern by enabling downstream users to rely on contractually obligated events being emitted. These events are separate from the database, allowing an application team to optimize their queries and table structures without having to negotiate with other teams.
This could be both an opportunity and a bit daunting to implement.
For the example above with orders a web team may start to implement a domain event when an order occurs and then find out that orders occur in multiple places across the system. On the one hand it can be hard to implement the domain event but on the other hand, large complex systems can be cleaned up and unified in such a way that the entire domain or "orders"is more easily understood and easier to maintain.
Data Engineering
Data engineering or analytics engineering teams often benefit greatly from eventing because it helps them to fulfill their obligations to both reporting and data science teams.
One of the core requirements on a data engineering team is to know the state of the system at any given point in time for the past few years. This becomes exceptionally complex when the source of your data is operational databases that have updates applied. What did the table look like yesterday? You have to know what the table looked like yesterday, and if it has updates, then teams will often need to store the entire table from yesterday.
This resents an enormous cost either in development time to clean this up or in storage costs to store multiple copies of databases.
Events are a solution to this. A streaming set of things happening in the system can just be stored in their raw form. Since there are no updates, there's no worry of storing duplicate data, nor is there a huge complexity in cleaning up the data set for use in reporting or machine learning.
Data Science
A core part of building an understandable ML model is to know what was the state of the system into the past. Eventing is one methodology of allowing data science teams to both have high quality data sets where the state of the system is known for every point in time, and to allow the ML models to operate on streaming data.
Does your LLM need to know the customer's most recent order? Just store, and use when it's necessary. Does your model need to know how many times a user has refunded an order in the last month, just a trivial count.
This allows data science teams to build models that are low in leakage because they are able to understand exactly what data came in exactly when.
This does increase the complexity on building features for an ML model. Many times I've had conversations with people over what exactly is your training set, no, really what exactly is the training set? Data over the last 6 months is the not the same as data for all users from the last 6 months who have relevant data. Building models off of streaming data means teams have to think very carefully up front about features in order to ensure that they're fulfilling the requirements of the model and aligning with it's future implementation.
These types of eventing systems present both an opportunity to build high quality models for data science teams but also a Less straightforward method of building feature sets.
Data Analytics
A common problem on reporting teams is when data changes upstream. Web team renames a column and suddenly the entire reporting suite is broken. Analytics teams traditionally struggle with being held accountable for changes that are outside of their purview.
If you are an analyst, how many time in the last month has one of your reports broken due to upstream changes?
Events help them with this because The very nature of eventing forces teams to think deeply about what data is being produced and what reporting needs. Analytics gets a seat at that table to define events, instead of just being a downstream user of the data they are the stakeholder.
Conclusion
I barely mentioned Kafka. Often Kafka and eventing and domain events are conflated, but they're really different and Kafka is just one system that could be used for moving events. Eventing is a way of thinking about data flows and a methodology of scaling a engineering organization to many independent, encapsulated teams.
At your company, ask yourself,
- Have we built systems that are flexible and enable teams to work independently and at scale?
- Can we build systems to allow advanced machine learning use cases to run robustly and at scale?
- If we think of all our dependencies, how can we enable those teams and projects to work independently and at scale?
There is a funny bit here, these types of systems are old and companies still struggle to even understand whether or not they want to adopt them. I tried to illustrate the justification for that as opposed to the mechanism that a lot of books and blog posts focus on.