Senior Infrastructure Engineer - Middleware Operations - Kafka

Overview

Wells Fargo is seeking a Senior Infrastructure Engineer for its Middleware Operations team, focused on supporting and engineering enterprise Apache Kafka and Confluent Platform environments.

What you'll do

• Design, deploy, upgrade, and operate enterprise-scale Apache Kafka and Confluent Platform environments.
• Configure and manage KRaft controllers, brokers, Schema Registry, REST Proxy, Kafka Connect, ksqlDB, and Cluster Linking.
• Deploy and operate Confluent Platform on OpenShift Container Platform using the Confluent Operator.
• Provision and administer Kafka topics, ACLs, producers, consumers, partitions, retention policies, quotas, and replication settings.
• Configure Kafka security using TLS, certificates, authentication, authorization, secrets management, and key rotation practices.
• Monitor and optimize platform performance, including JVM garbage collection, heap utilization, partition distribution, ISR health, disk capacity, throughput, and consumer lag.
• Troubleshoot production issues such as message loss or duplication, offset errors, broker failures, disk saturation, latency, throughput degradation, and consumer lag.
• Design and maintain platform health dashboards and alerts using Prometheus, Grafana, Datadog, Splunk, or comparable enterprise monitoring tools.
• Automate provisioning, monitoring, recovery, regression testing, and operational activities using Python, Bash, Ansible, or similar technologies.
• Conduct capacity planning, performance benchmarking, and throughput optimization using tools such as KafkaPerf and kcat.
• Support incident, problem, change, and recovery management for business-critical middleware services, including root-cause analysis and preventive remediation.
• Develop infrastructure specifications, test solutions, support complex system upgrades, and contribute to application and infrastructure requirement validation.
• Establish long-term solutions for recurring production issues and identify opportunities to improve availability, resilience, and recovery time.
• Support risk and control initiatives by following technology policies, operating procedures, compliance requirements, and established work standards.
• Collaborate with application teams, engineering teams, cybersecurity partners, product managers, and infrastructure stakeholders to resolve issues and deliver platform objectives.
• Create and maintain technical documentation, standard operating procedures, recovery procedures, architecture diagrams, and operational runbooks.
• Participate in an on-call support rotation and provide implementation support for production releases and critical risk initiatives.

What you'll need

• 4+ years of Technology Infrastructure Engineering, Middleware Engineering, Site Reliability Engineering, or equivalent experience demonstrated through work, education, training, or military experience.
• Minimum 4 years of hands-on experience supporting large-scale middleware environments, including Apache Kafka and Confluent Platform.
• Strong production experience with Kafka brokers, KRaft, Schema Registry, REST Proxy, Kafka Connect, ksqlDB, and Cluster Linking.
• Hands-on experience deploying and operating Kafka on OpenShift using the Confluent Operator.
• Experience managing topics, ACLs, partitions, replication, retention, quotas, idempotent producers, and consumer groups.
• Strong troubleshooting experience covering consumer lag, broker crashes, offset issues, message loss or duplication, disk saturation, latency, and throughput problems.
• Experience implementing Kafka security using TLS, certificates, authentication, authorization, and enterprise key-management practices.
• Experience configuring Prometheus and Grafana dashboards for platform monitoring and health reporting.
• Proficiency in automation and scripting using Python, Bash, Ansible, or comparable technologies.
• Experience supporting highly available, enterprise production environments.
• Understanding of technology risk, control, compliance, and change-management practices within the Banking, Financial Services, and Insurance sector.

Nice to have

• Experience with KRaft migrations and upgrades from ZooKeeper-based Kafka environments.
• Knowledge of performance testing and benchmarking using KafkaPerf, kafkacat, or kcat.
• Experience with Confluent Cloud, Amazon MSK, or Azure Event Hubs for Apache Kafka.
• Knowledge of schema evolution, compatibility modes, and Avro, JSON, or Protobuf serialization.
• Experience with additional messaging technologies such as Solace, IBM MQ, or RabbitMQ.
• Understanding of enterprise networking concepts, including routing, BGP, multicast, load balancing, DNS, and certificate management.
• Experience with capacity management, workload forecasting, and throughput optimization.
• Experience with Prometheus, Grafana, Datadog, Splunk, or similar monitoring and observability platforms.
• Confluent Certified Developer for Apache Kafka or Confluent Certified Administrator certification.
• ITIL 4 Foundation or higher certification.

Details

• Ability to provide production implementation and operational support outside standard business hours when required.
• Ability to participate in an on-call rotation for critical incidents and service restoration.
• Ability to work in a controlled enterprise environment with strong risk, security, audit, and documentation requirements.
• Location: Hyderabad, India.