Job Brief:
The Data Acquisition team operates at the core of our platform, engineering efficient systems that empower all our clients through all our products. We design and run large scale, highly distributed pipelines capable of scheduling, orchestrating, and executing tens of millions of data collection tasks daily across a diverse and constantly evolving landscape of sources. This involves tackling complex challenges in high-throughput web crawling, adaptive extraction, intelligent batching, fault tolerant processing, and seamless distribution of mission critical datasets. We are seeking a Software Engineer who thrives on building resilient, scalable systems, solving real world data engineering puzzles, and pushing the boundaries of automation and efficiency in large-scale information retrieval.
Essential Function & Responsibilities:
- Design and implement advanced capabilities to detect and counter anti-bot protection methods.
- Engineer complex network requests that accurately mimic real user behavior and environments.
- Develop network services leveraging HTTP(S) protocol versions 1.1, 2, and 3.
- Implement and manage egress proxy networks with MITM traffic interception and routing optimization.
- Extend Chrome DevTools Protocol (CDP) with custom commands to support specialized use cases.
- Build and maintain scalable WebSocket infrastructures optimized for short-lived, high-volume connections.
- Design distributed caching systems to support high throughput containerized worker pools.
- Deploy and manage geographically distributed proxies with automated failover and load balancing.
- Optimize compute, storage, and traffic costs for high-volume cloud-based services.
- Proficiency with data parsing, transformation, and storage technologies.
- Unwavering focus on delivering measurable value to customers
Location:
This is a remote position open to candidates based in the United States.
Compensation:
The salary for this role is $130,000-160,000. Please note that our salary ranges are based on current market data and the offered compensation may vary based on experience, skills, location, and other factors permitted by law.
Wayvia is an equal opportunity employer that is committed to inclusion and diversity.
Top Skills
Similar Jobs
What you need to know about the Boston Tech Scene
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
