Software Diagnostics & Performance Test Engineer

Overview

Investigate complex software failures and performance problems in SOTI's products through deep root-cause analysis, experimentation, and performance test design.

What you'll do

• Investigate complex and ambiguous software failures and performance problems.
• Perform deep root-cause analysis across application, system, network, database, and infrastructure layers.
• Analyze application logs, system logs, stack traces, thread dumps, heap dumps, metrics, traces, and profiling data.
• Analyze network traffic and protocols to identify latency, connectivity, retransmission, timeout, and communication issues.
• Investigate issues that occur only under specific workloads, concurrency levels, configurations, environments, or long-running conditions.
• Understand product architecture and critical customer workflows deeply enough to reason about failure modes.
• Learn the product in depth, including core workflows, customer usage patterns, deployment models, and performance-sensitive areas.
• Design experiments to isolate variables and prove or disprove hypotheses.
• Develop diagnostic scripts, instrumentation, and small tools when necessary to accelerate investigations.
• Perform exploratory testing and deliberately challenge system assumptions to discover weaknesses and failure modes.
• Investigate performance degradation, scalability limitations, resource exhaustion, latency spikes, and throughput bottlenecks.
• Create meaningful Performance Test Cases based on product architecture, customer workflows, expected load patterns, scalability goals, and known risk areas.
• Review Performance Test Cases created by other engineers to ensure coverage, clarity, workload relevance, data validity, and alignment with product behavior.
• Execute Performance Test Cases, analyze results, identify bottlenecks or regressions, and communicate findings with clear evidence and recommended actions.
• Correlate evidence across multiple components of distributed systems.
• Work with developers to identify the actual root cause rather than simply identifying the component where the symptom appeared.
• Clearly communicate technical findings, evidence, root cause, impact, and recommended corrective actions.
• Build and maintain diagnostic knowledge, troubleshooting methodologies, and internal technical documentation.
• Mentor other engineers in systematic debugging and root-cause analysis.

What you'll need

• 5+ years of software engineering, performance engineering, systems engineering, reliability engineering, production troubleshooting, or a closely related discipline, preferably in a product-based software company.
• Programming/debugging ability in at least one major language such as Java, C++, C#, Python, etc.
• Strong ability to read and understand unfamiliar application code.
• Demonstrated experience troubleshooting complex software problems.
• Strong understanding of operating-system concepts: processes, threads, memory, CPU, file I/O, sockets, synchronization, resource utilization.
• Strong understanding of networking fundamentals: TCP/IP, HTTP/HTTPS, DNS, connection management, latency, timeouts, retransmissions.
• Experience analyzing logs and traces across multiple components.
• Experience with application profiling and/or performance analysis.
• Hands-on experience creating, reviewing, and executing Performance Test Cases for complex software products.
• Experience diagnosing concurrency, latency, resource contention, or scalability issues.
• Ability to interpret performance test results, compare them with baselines, identify regressions, and connect findings back to product behavior or code-level causes.
• Ability to formulate hypotheses and systematically validate them with evidence.
• Exceptional curiosity about how software systems work.
• Strong interest in learning the product deeply rather than treating testing as a black-box activity.
• Strong analytical and problem-solving skills.
• Comfortable working with incomplete or ambiguous information.
• Willingness to challenge assumptions.
• Persistence when the first explanation isn't the real root cause.
• Ability to communicate complex technical findings clearly.
• Comfortable collaborating with developers without simply handing problems back to them.
• Strong ownership of difficult investigations.