Kafka topic must host the same domain of data? The Kafka topic myth

Same domain of data goes into one Kafka topic, and Kafka topic should have only one domain, are two conclusions I often saw some people talk about. The backgroud I believe is from relational database.

Ordering Problems

The second one : Kafka topic should have only one domain, is easier to justify and it’s not ture due to ordering requirement.

When putting different event types in different kafka topics, the events are not in order.

In stream processing application, ordering is very important for performance and correctness of the output. Becuase the application needs to produce new event, based on ordering of different event types.

Of course windowing and watermark help to resolve it, it has to sacrifice lots of performance to re-order the events.

Confluent, the Kafka company, answers it in a blogpost https://www.confluent.io/blog/put-several-event-types-kafka-topic/

Single Writer Principle in Kafka

The first one is the same domain of data should go into one Kafka topic.

Let’s see a concret example. Two microservices share the same domain of data, do you let the two microseservices use the same database table? if the application does, it may make sense to let the two microservices to produce to the same topic.

If the two microserivce doesn’t share the same mongo collection, or the same database table, why do you want the two microservices to write to the same topic, even though the data schema is the same?

If following per microservice per topic, separating topics brings simpler monitoring, schema evolution and scaling.

Confluent calls it Single Writer Principle in Kafka https://www.confluent.io/blog/build-services-backbone-events/