DataRobot is leveraging the power of Hadoop to scale our ground-breaking predictive analytics platform and is seeking experienced Hadoop Automation Engineers to develop and support integrations with customer environments and their Hadoop infrastructures.

As a Hadoop Platform Engineer you’ll be working with the most popular enterprise Hadoop solutions to enable scalable predictive modeling capabilities on Hadoop clusters. You will apply your knowledge to improve Hadoop experience for DataRobot’s customers, developers, and integrators.

The Challenges:

• High-load environments: we run Hadoop in the cloud and scale up to meet peak demand.
• Secured environments: our clients are very serious about enterprise data security. Our Hadoop infrastructure has to reflect that by supporting cutting-edge security protocols and technologies..
• Automation capabilities: every new environment is an exciting challenge. We need to make sure that even the most complex environments can be reproduced in our testing pipeline.
• Automation reliability: we run dozens of Hadoop pipelines every day to ensure the safety and stability of all of our iterations. We continuously improve the automation of Hadoop deploys to verify that the new code doesn’t introduce any regressions.
• Problem-solving: If our support team can’t resolve a request, our duty is to tackle the challenge and prove our extraordinary proficiency with Hadoop, all of its components, and features.

Mission-Critical Tasks:

• Improving Hadoop components: we’re always looking to improve high-load performance, refine hardware failure management, and sharpen operations support.
• Better test coverage: DataRobot supports a breadth of environments. Our job is to continuously expand the test coverage for these environments. Nothing should slip through the cracks.
• Realistic bare metal testing: emulating customer environments is paramount. We need to uncover potential issues before our customers find them.

Responsibilities

• Develop integration tools that will allow DataRobot to run smoothly on multiple Hadoop platforms.
• Develop automated build and test systems to deliver a stable, reliable product.
• Provide third-tier support for customer installations of DataRobot, jumping in for the most technically challenging problems.
• Pairing with and mentoring colleagues.

Main Requirements

• Understanding of what distributed computing is and when it should be used
• 3+ years of experience with Linux system configuration and administration.
• Experience with creating Python applications.
• Experience with creating Bash scripts.
• Excellent written and verbal communication skills.
• Great technical troubleshooting skills.

Bonus

• Experience with Hadoop Security (Kerberos, LDAP, etc).
• Experience with Hadoop components such as HDFS, YARN, Spark, and ZooKeeper.
• Experience as a technical lead.
• Experience with automation tools such as Ansible, Chef, or Puppet.
• Experience with Scala (or Java).
• Experience with systems and application performance tuning.
• Experience with AWS, Docker, Jenkins, or similar technologies