The virtual conference is free and I will share the talk I recommend
Kafka as Your Data Lake — is it Feasible?
I once worked in a company that uses Kafka as soure of truth, although I have to say, not completely because at that time kafka cannot support S3 as storage like recently. This talk summaries some usage patten, for example using KSQL/Spark/Presto. It recaps and summarizes what I saw in a very comprehensive manner and give a demo. I’m quite sure the slides would be useful in the future during architecture discussion whenever necessary.
https://www.slideshare.net/gschmutz/kafka-as-your-data-lake-is-it-feasible
Spring Kafka Beyond the Basics: Lessons Learned on Our Kafka Journey at ING Bank
It’s a short talk and introduces the already implemented ErrorHandler in Spring-Kafka that allows you to log and/or send the message which generates DeserializationError to a dead letter topic.
Tradeoffs in Distributed Systems Design: Is Kafka The Best?
It allows you to figure out the design choice made in Kafka, comparing to other systems. What’s the pros and cons for 1. putting the partitions in the same broker 2. Less metadata for data storage position information and why Kafka decided so.
I Don’t Always Test My Streams, But When I Do, I Do it in Production
If you already uses Kafka Streams or consumer, you only need to watch the last 10 minutes. You will see running KafkaContainer&SchemaRegistryContainer in IDE instead of using docker-compose to run Kafka and schema registry and more components. Thus, your tests can run in parallel. What a big advantage! TestTopologyDriver you might already use before, but ksql test runniner is new to me.
https://github.com/confluentinc/demo-scene/tree/master/testing-streams
