Builing Kafka Connect For Syslog

Recently at work, I needed to install a connector for Kafka to receive syslog data. A quick Google search showed that Confluent has a paid connector to use with Kafka Connect, but we were looking for something more free-as-in-beer. Github has a couple of old repositories for such a connector, but these are old forks or clones that haven’t been touched in years. And worse, no one has compiled the JAR and stuck it in as a Release, so all we have is the source code. Let’s dive in to building the JAR that can be installed and quickly used on my already existing Kafka Connect server.

Step zero, install maven and a JDK. We’re not getting far without the prereqs. Now that we’re set there, clone one of the old repos from Github. Let’s start by trying to build as-is and see what we get. Run mvn clean package and you will get a fair amount of action, followed by kafka-connect-syslog-back/src/main/java/com/github/jcustenborder/kafka/connect/syslog/SyslogSourceTask.java:[19,45] package io.confluent.kafka.connect.utils.data does not exist. Note that com/github/jcustenborder/kafka/connect/connect-utils/0.2.86/connect-utils-0.2.86.jar was downloaded and provides the appropriate package. All we need to do is tell our .java file to look for the differently-namespaced package already downloaded as a POM depedency.

Modify the complaining file to:

//import io.confluent.kafka.connect.utils.data.SourceRecordConcurrentLinkedDeque;
import com.github.jcustenborder.kafka.connect.utils.data.SourceRecordConcurrentLinkedDeque;

You could delete or modify the bad import line, but I prefer to avoid deleting where possible. Let’s save this change and run mvn clean package once more. Hmmm… io.confluent.kafka.connect.utils.config does not exist is pretty similar to our previous error. This package is imported in three files: UDPSyslogSourceConfigTest.java, TCPSyslogConfigTest.java, and SSLTCPSyslogSourceConfigTest.java. In each of these three files, modify the import lines like so:

//import io.confluent.kafka.connect.utils.config.MarkdownFormatter;
import com.github.jcustenborder.kafka.connect.utils.config.MarkdownFormatter;

Trying mvn clean package once more should download all dependencies successfully and complete the preliminary build only to have the tests fail. Luckily, tests are for losers so we’ll rerun it as mvn -DskipTests clean package and the JAR will build successfully and be dropped in the target folder. Awesome!

This may suffice for some, but when I went to deploy this JAR I ran into a new issue: Kafka Connect complained about not being able to find Java packages that were installed to /etc/java. After a bit of putzing with my CLASSPATH, I realized the safest way to ensure they were available (and not not found again if a different user with a different CLASSPATH attempts to start Kafka Connect) would be to bake them into the JAR itself. Adding the below code into the <project> element of pom.xml will do what you want:

<build>
    <plugins>
    <plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-assembly-plugin</artifactId>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>single</goal>
      </goals>
      <configuration>
        <archive>
          <manifest>
            <mainClass>${fully.qualified.main.class}</mainClass>
          </manifest>
        </archive>
        <descriptorRefs>
          <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
      </configuration>
    </execution>
  </executions>
</plugin>
</plugins>
</build>

And now you can run mvn -DskipTests clean package one last time and take a look at your beautiful new JAR in your target folder which has the dependencies included and will drop right in to Kafka Connect to be used right away.

Because I’m such a nice person, I am hosting the file I just walked through building in the event that anyone has difficulty building it themselves or chooses not to do so. kafka-connect-syslog-0.2-SNAPSHOT-jar-with-dependencies.jar