Senior Software Engineer (Big Data), Massive Graph Processing
LiveRamp is the leading data connectivity platform for the safe and effective use of data. Our platform powers people-based customer experiences that improve the relevance of marketing and allow consumers to connect with the brands and products they love. We thrive on solving fascinating technical problems and enabling our customers’ success—and we’re always looking for smart, kind, compassionate people to help us blaze a trail.
Mission: LiveRamp makes it safe and easy for businesses to use data effectively.
We on the Identity Engineering team maintain the graph processing engine that connects the trillions of different identifiers for consumers and the devices they use online. Help us process this trillion edge graph as quickly and efficiently as possible.
- Collaborate with a team of supportive and passionate software engineers, product, infrastructure and other internal partners remotely to tackle challenging problems, plan and execute on development projects that build, enhance and customize our high-performance data ingest, partitioning, graph and edge building systems.
- Foster a positive environment of integrity, operational excellence, initiative and empower the team to achieve a high level of technical productivity, reliability, and simplicity by developing in Java and leveraging tools like Docker, Kubernetes, Terraform, Gradle, Jenkins, GCP, Hadoop and Spark.
- Design with privacy and security-first principles and err on the side of shipping fast and iterating.
Your team will:
- Find feedback to be important in helping us grow as individuals and as a team; we’re always looking for chances to kindly share positive and constructive feedback and we aim to accept feedback with grace.
- Follow a Scrum-based devops model with a few guiding principles:
- Support, trust and motivate the people involved
- Continuous software delivery and process improvement
- Minimize barriers to development and leverage automation
- Work on:
- Edge Ingestion and Partitioning Framework: We could never process all trillion edges at once and luckily we don’t have to. Instead we process subgraphs that contain specific types of edges. Our edge ingestion and partitioning leverages LiveRamp’s Seek MSJ framework to efficiently incorporate new data into existing edge stores.
- Path Computation as a Service: We provide a service to other LiveRamp engineering teams for finding specific types of paths within our massive graph. It services 80,000 requests, 10PB data per day and this is possible due to its use of highly scalable Spark jobs and intelligent caching mechanism.
- Project categories: besides product requests and enhancements, we work on projects in performance and cost optimization, security and legal compliance, reliability and quality initiatives, as well as architectural refreshes such as POC in restructuring our data to be more streaming/real time friendly, new REST API for relevant services, migration of apps to Spark, Parquet, and other new technologies that could make our processing faster, more scalable, maintainable and cost effective.
- Have 3+ years of experience writing and deploying object oriented production code.
- Have 2+ years of experience using cloud based platform such as GCP, AWS, Azure or similar technologies
- Strong ability to break down complex problems to their essential components, design and implement elegant solutions for them.
- Have a passion for building large-scale distributed systems and are comfortable writing maintainable and high performance code.
- Comfortable evaluating and adapting to the latest tools and technologies.
- You love mentoring junior engineers and deploying software that conforms to best practices.
- Have a startup personality and enjoy working as part of a team: smart, ethical, friendly, hard-working, and productive.
- Excellent communication and presentation skills.
- Bachelor of Science in Computer Science or related degree.
- Experience with GCP products such as BigTable, BigQuery, Dataproc, PubSub or other data tools like Spark, Kafka, Hive, or Apache Beam
- Experience with development tools such as Terraform, Kubernetes, Helm, or Gradle
- Experience with designing and implementing interfaces and infrastructure for large scale systems and RESTful APIs
- People: work with talented, collaborative, and friendly people who love what they do.
- In-Office Food: enjoy catered meals, boundless snacks, and the occasional food truck.
- Fun: we host in-person and virtual events such as game nights, happy hours, camping trips, and sports leagues.
- Work/Life Harmony: flexible paid time off, remote work opportunities, and paid parental leave.
- Whole Health Package: medical, dental, vision, and disability insurance. Plus mental health support (via Talkspace) and fitness reimbursement up to $100 per month.
- Savings: our 401K matching plan helps you plan ahead.
- Remote Work Support: a comprehensive program to assist you in setting up a home office that works for you
- Location: Work remotely from home.
More about us:
To all recruitment agencies: LiveRamp does not accept agency resumes. Please do not forward resumes to our jobs alias, LiveRamp employees or any other company location. LiveRamp is not responsible for any fees related to unsolicited resumes.
#LI-Remote (Recruiters - remove this tag if the role is NOT a remote position)