Build and maintain scalable, secure data pipelines and AI-ready datasets using Azure Databricks, Snowflake, Python, and Spark; support RAG solutions, integrate Azure AI services, implement data governance, CI/CD, and production monitoring for enterprise generative AI initiatives.
This is a remote position.
Job SummaryWe are seeking a skilled Data Engineer to support enterprise AI and Generative AI initiatives by building scalable, secure, and AI-ready data platforms. The ideal candidate will have strong expertise in Azure Databricks, Snowflake, Python, and SQL, along with experience integrating modern Azure AI services into enterprise data ecosystems.
In this role, you will partner with Data Engineers, AI Engineers, Data Scientists, and business stakeholders to develop high-quality data pipelines, enable Retrieval-Augmented Generation (RAG) solutions, and prepare enterprise data for AI-powered applications using Microsoft's AI ecosystem.
- Design, develop, and maintain scalable data pipelines using Azure Databricks, Snowflake, and Azure Data Factory.
- Build ELT/ETL pipelines to ingest data from ERP, CRM, manufacturing, supply chain, and other enterprise applications.
- Develop AI-ready datasets that support machine learning and Generative AI use cases.
- Integrate structured, semi-structured, and unstructured data into centralized data platforms.
- Support Retrieval-Augmented Generation (RAG) solutions by preparing document repositories, metadata, embeddings, and search indexes.
- Collaborate with AI Engineers to expose enterprise data securely to LLM-powered applications.
- Optimize data models, transformations, and query performance for analytics and AI workloads.
- Implement data quality, governance, lineage, monitoring, and security best practices.
- Build reusable data transformation frameworks using Python, SQL, Spark, and dbt (if applicable).
- Develop REST API integrations to support AI services and enterprise applications.
- Participate in architecture discussions, code reviews, CI/CD implementations, and Agile ceremonies.
- Support production deployments, monitoring, troubleshooting, and performance tuning.
- 6+ years of experience in Data Engineering.
- Strong hands-on experience with Azure Databricks and Snowflake.
- Proficiency in Python, SQL, and PySpark.
- Experience building scalable ELT/ETL pipelines using Azure Data Factory or similar orchestration tools.
- Strong understanding of data warehousing, dimensional modeling, and data lake architectures.
- Experience working with REST APIs and integrating cloud-based services.
- Familiarity with Git, Azure DevOps, and CI/CD pipelines.
- Experience working in Agile/Scrum environments.
- Experience supporting enterprise AI or Generative AI initiatives.
- Working knowledge of Azure Machine Learning (Azure ML).
- Experience with Microsoft AI Foundry (Azure AI Foundry) for AI solution development and orchestration.
- Experience integrating Azure OpenAI Service into enterprise applications.
- Knowledge of Azure AI Search for enterprise search and Retrieval-Augmented Generation (RAG) solutions.
- Exposure to Microsoft Copilot Studio for developing AI-powered copilots and conversational experiences.
- Understanding of vector search, embeddings, prompt engineering, and LLM integration.
- Experience preparing enterprise data for AI model training, inference, and knowledge retrieval.
Similar Jobs
Artificial Intelligence • Cloud • Computer Vision • Hardware • Internet of Things • Software
Design, build, and operate production AI and data systems for marketing: LLM/agent orchestration, retrieval pipelines, data pipelines, CDP and warehouse architecture, automation of manual workflows, data quality and reliability, stakeholder collaboration, and mentoring. Ship production-grade Python and SQL solutions and evaluate/monitor AI systems.
Top Skills:
Agent OrchestrationAPIsBigQueryClaude CodeCursorCustomer Data Platform (Cdp)Data WarehouseDatabricksDbtEltETLHightouchLlmsMarketing AutomationPythonRetrieval PipelinesSalesforceSegmentSnowflakeSQLWeb Analytics
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Lead design, build, and deploy of large-scale data platforms for LLMs, RAG, and agentic AI systems. Hands-on coding, architecting fault-tolerant pipelines, establishing MLOps/DataOps best practices, mentoring engineers, and operationalizing research into production across Exabyte-scale distributed systems.
Top Skills:
AirflowAWSBigQueryDaskDevsecopsDockerFlinkGCPGoJvmKafkaKubeflowKubernetesLangchainLlamaindexLlmsMlflowOciPulsarPythonRetrieval-Augmented Generation (Rag)RustSagemakerSnowflakeSparkVertex Ai
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Design, build, and deploy AI/ML and generative AI solutions and full-stack applications. Develop frontend experiences, backend services, microservices, APIs, and cloud-native data integrations. Implement RAG, LLMs, vector DBs, dashboards, reporting, and self-service analytics to support quality, patient safety, and workflow automation. Provide technical leadership, mentor engineers, and apply CI/CD, testing, observability, security, and performance best practices.
Top Skills:
AgentsApache SupersetAutomated TestingAWSAzureAzure Ai ServicesCi/CdDatabricksEvent-Driven ArchitectureGCPGenerative AiJavaJavaScriptLangchainLlmsMicroservicesMicrosoft FabricObservabilityOpenaiPower BIPrompt EngineeringPythonRagReactRest ApisSemantic KernelSnowflakeSpring BootSQLTypescriptVector Databases
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
