Database to database streaming with Apache Kafka and Exasol
This article shows how to efficiently feed data flows into a relational database system in near real time. This means that the data is loaded and processed when it occurs and is quickly available to the company for analytical purposes. This approach is known as real-time / near real-time processing. In contrast to (near) real-time loads are batch loads, in which data is loaded and processed at time X (for example, daily or weekly) in correspondingly large quantities. So instead of working with data that can be over 24 hours old, (near) real-time processing allows you to use very recent data. This enables companies to react faster and significantly increases the timeliness of the information content. Data from different source systems can be loaded and processed via the same stream.
Specifically, we will show how Exasol can work together with the streaming platform Kafka. In the following use case, the integration of Exasol as a sink and source system, i.e. as a target or source system, for Kafka is presented and implemented in a test example.