Linux Performance Developer

Linux Performance Developer

Key Responsibilities:

1. System Performance Analysis

• Analyze system-level performance on Linux platforms across CPU, memory, disk I/O, and network.
• Use performance monitoring tools (e.g., perf, ftrace, oprofile, bpftrace, systemtap, vmstat, iotop, top, htop, sar, latencytop) to identify bottlenecks.
• Profile applications and system services to pinpoint inefficiencies.

2. Benchmarking and Optimization

• Design, implement, and run synthetic and real-world benchmarks (e.g., Phoronix, stress-ng, lmbench, fio).
• Optimize kernel subsystems (scheduler, memory management, file systems, network stack) and user-space components.
• Tune performance parameters (sysctl, cgroups, kernel boot args, IRQ affinity, NUMA policies).

3. Performance Regression Detection

• Integrate performance testing into CI pipelines.
• Use tools like kselftest, LTP, BPF, and flamegraphs to detect regressions and improve stability under load.

4. Collaboration with Kernel and User-Space Teams

• Work closely with kernel developers to suggest enhancements or fix issues in upstream drivers, kernel modules, or subsystems.
• Collaborate with application teams to guide performance improvements or align workloads with system capabilities.

5. Tool Development

• Build custom performance profiling or tracing tools using eBPF, Python, C/C++, or shell scripting.
• Maintain dashboards and telemetry infrastructure (Grafana/Prometheus/InfluxDB/etc.).

6. Power and Thermal Optimization

   (in mobile/embedded use cases)
• Analyze and reduce CPU wakeups, idle power usage, and dynamic frequency scaling behavior (cpufreq, cpuidle, thermal governor tuning).

Required Expertise:

• Strong Linux Internals Knowledge

• Deep understanding of kernel subsystems (scheduler, memory, CPU topology, virtual memory, etc.).
• Good grasp of system calls, IPC, and process life cycle.

• Proficiency in Tools

• perf, ftrace, bpftrace, strace, valgrind, gprof, flamegraph, systemtap, numactl.

• Languages

• C (for kernel modules or low-level profiling), Python or Go (for automation), Shell scripting (for tooling and monitoring setup).

• Experience with Virtualization/Containers

• Performance tuning for KVM, Xen, Docker, LXC, and Kubernetes environments.
• Understanding of cgroups, namespaces, and resource quotas.

• Experience in SoC/Embedded Devices

• ARM performance counters, PMU tuning, kernel power governors, and device-specific optimizations.

• Monitoring and Metrics

Familiar with telemetry stacks (Grafana, Prometheus, Telegraf), exporting and interpreting metrics.