Kafka Developer

Skills set: • Administration of Confluent Kafka infrastructure • Kafka brokers, Producer, Consumers, zookeepers, Kafka connect, schema registry, KsqlDB, Kafka streams. • Monitoring and setup using Prometheus, Grafana and confluent control Centre • Scripting using Shell and Python • Infrastructure as Code- Terraform • CI/CD using Jenkins • Knowledge on GCP/AWS/Azure Cloud • Managed/Custom Kafka connectors configuration • Kafka development experience with Microservices/java/j2ee frameworks. Roles and responsibilities: • Leading and Architecting Confluent Kafka application • Creating Kafka topics and providing required access to the Producer Application to publish messages to the Kafka topic. • Providing appropriate access to the consumer group to consumer messages from Kafka topic for Consumer Application. • Diagnosing, Troubleshooting, and solving Kafka issues and maintaining the documentation for the same. • Developing middleware integration solution for enrichment or transformation • Deploying Source and Sink connectors on self-managed Virtual Machines (VM) based on the client’s needs • Build data pipelines with Kafka connect/connectors to deploy and support Kafka Data stream processing. • Setup monitoring tool Prometheus, Grafana and confluent control Centre • Configure Grafana dashboard to monitor Confluent Kafka cluster and providing access to the requested users. • Setup alerts for Kafka Consumer Lags and other issues. • Responsible for implementation and ongoing administration of Confluent Kafka infrastructure.