Cloud Observability and Performance Engineer

The Role:

We are looking for a Cloud Observability and Performance Engineer to join our Chaos Cloud Engineering team. In this role, you will design and implement observability, monitoring, and performance strategies for cloud-hosted microservices that manage and orchestrate endpoint security agents at scale. This position is critical to ensuring the reliability, visibility, and performance optimization of our backend systems that power cloud-based security operations for millions of endpoints worldwide.

Responsibilities: 

• Design, build, and maintain end-to-end observability for distributed cloud services (telemetry, logging, tracing, alerting). 
• Develop and optimize metrics pipelines and dashboards (e.g., Prometheus, Grafana, OpenTelemetry, Datadog).    
• Ensure high performance, availability, and scalability of agent management systems in production.    
• Collaborate with development, SRE, and security teams to troubleshoot production issues using observability tooling.    
• Define and implement SLOs, SLIs, and performance benchmarks for cloud components and services.    
• Instrument code and services to expose business-relevant metrics and latency bottlenecks.    
• Automate performance regression testing and anomaly detection.    
• Support proactive incident detection and real-time monitoring strategies across multi-cloud environments.
• Design, implement, and own a performance testing framework to validate system throughput, latency, and scalability under load.
• Define baseline performance thresholds and use observability tooling to monitor and validate results.    
• Provide root cause analysis and performance tuning recommendations.    

Skills and Qualifications:

• 5+ years of professional work experience in observability, site reliability, or cloud performance roles.    
• Strong experience with monitoring and observability stacks (e.g., Prometheus, Grafana, ELK, OpenTelemetry, Datadog, AWS CloudWatch).    
• Proficiency in cloud platforms (e.g., AWS, GCP, Azure) and cloud-native services (e.g., ECS, EKS, Lambda).  
• Experience with containerization and orchestration tools (e.g., Docker, Kubernetes).  
• Hands-on experience designing and implementing performance or load testing frameworks for distributed systems.
• Ability to define and validate throughput baselines, latency thresholds, and system limits under real-world traffic scenarios. 
• Solid knowledge of distributed systems, microservices, and performance debugging.    
• Proficiency in Python, Scala, or other language(s) for tooling and automation.    
• Familiarity with CI/CD pipelines, infrastructure as code (e.g., Terraform), and version control (Git).    
• Ability to participate in an on-call rotation to support observability infrastructure and assist with incident investigations.

• Experience with endpoint security platforms or agent-based systems.    
• Familiarity with SIEM, security analytics, or cloud threat detection pipelines.    
• Background in networking performance, TLS handshake optimization, or load balancing.    
• Experience with SLA/SLO-driven operational excellence in high-scale environments.    
• Knowledge of additional languages, such as Go.
• Experience integrating performance tests into CI/CD pipelines and visualizing results using tools like Grafana, Datadog, or similar.
• Familiarity with tools such as k6, Locust, JMeter, Gatling, or custom-built performance testing solutions.

• Work on cutting-edge cloud infrastructure supporting global security products.    
• Be a core part of a team building resilient systems at internet scale.    
• Competitive salary, flexible work culture, and continuous learning opportunities.

 Halcyon offers the following benefits to eligible employees:

• Comprehensive healthcare (medical, dental, and vision) with premiums paid in full for employees and dependents.

• 401k plan with a generous employer contribution.

• Short and long-term disability coverage, basic life and AD&D insurance plans.

• Medical and dependent care FSA options.

• Flexible PTO policy.

• Parental leave.

• Generous equity offering.

The Company reserves the right to modify or change these benefits programs at any time, with or without notice.​

Base Salary Range: $150,000 - $185,000

Bonus Target: 10%