Myujth

Kafka MirrorMaker is a basic approach to mirror Kafka topics from source to target brokers. Unfortunately, it doesn't fit my requirements to be configurable enough.

My requirements are very simple:

• the solution should be JVM application


• if destination topic doesn't exist, creates it


• solution should have the ability to add prefixes/suffixes to destination topic names


• it should reload and apply configurations on the fly if they're changed

According to this answer there are several alternative solutions to do this stuff:

• 
  MirrorTool-for-Kafka-Connect


• 
  Salesforce Mirus (based on Kafka Connect API)


• Confluent's Replicator


• Build my own application (based on Kafka Streams functionality)

Moreover, KIP-382 was created to make Mirror Maker more flexible and configurable.

So, my question is what's the killer features of these solutions (comparing to others) and finally what's the better one according to provided requirements.

I see you are referring to my comment there...

As for your bullets

the solution should be JVM application

All the listed ones are Java based

if destination topic doesn't exist, creates it

This is dependent on the Kafka broker version supporting the AdminClient API. Otherwise, as the MirrorMaker documentation says, you should create the destination topic before mirroring, otherwise you either get (1) denied to produce because auto topic creation is disable (2) problems seeing "consistent" data because a default configured topic was created.

That being said, by default, MirrorMaker doesn't "propogate" topic configurations on its own. When I looked, MirrorTool similarly did not. I have not looked throughly at Mirus.

Confluent Replicator does copy configurations and it will use the AdminClient.

Replicator, MirrorTool, and Mirus are all based on Kafka Connect API. And KIP-382 will be as well

Build my own application (based on Kafka Streams functionality

Kafka Streams can only communicate from() and to() a single cluster.

You might as well just use MirrorMaker because it's a wrapper around Producer/Consumer already, and supports one cluster to another. If you need custom features, that's what the MessageHandler interface is for.

solution should have the ability to add prefixes/suffixes to destination topic names

Each one can do that, but MirrorMaker requires extra/custom code. See example by @gwenshap

reload and apply configurations on the fly if they're changed

That's the tricky one... Usually, you just bounce the Java process because most configurations are only loaded at startup. The exception being whitelist or topics.regex for finding new topics to consume.

KIP-382

Hard to say that'll be accepted. While it was thoughly written, and I personally think it's reasonably scoped, it somewhat defeats the purpose of having Replicator for Confluent. The AvroConverter for Confluent Schema Registry is open sourced, and with large majority of Kafka commits and support happening out of Confluent, it's a conflict of interest as, AFAICT, that's the only missing piece between Replicator and the other Connect implementations... Case in point, MirrorTool had a KIP, but it was seemingly rejected on the mailing list with the explanation of "Kafka Connect is a pluggable ecosystem, and anyone can go ahead and install this mirroring extension, but it shouldn't be part of the core Kafka Connect project", or at least that's how I read it.

What's "better" is a matter of opinion, and there are still other options (Apache Nifi or Streamsets come to mind). Even using kafkacat and netcat you can hack together cluster syncing.

If you need Avro support and are paying for an enterprise license, then you might as well use Replicator.

One thing that might be important to point out, that if you are mirroring a topic that is not using the DefaultPartitioner, then data will be reshuffled into the DefaultPartitioner on the destination cluster if you don't otherwise configure the destination Kafka producer to use the same partition value or partitioner class as the source Kafka Producer.