Design, implement, and maintain PySpark applications to automate large-scale financial data reconciliations. Build transformation and matching algorithms, integrate with rules engines, analyze data gaps, and collaborate with analysts and architects to ensure data quality and system resilience.
Job Title: PySpark Data Reconciliation Engineer
Summary:
We're seeking a skilled PySpark Data Reconciliation Engineer to join our team and drive the development of robust data reconciliation solutions within our financial systems. You will be responsible for designing, implementing, and maintaining PySpark-based applications to perform complex data reconciliations, identify and resolve discrepancies, and automate data matching processes. The ideal candidate possesses strong PySpark development skills, experience with data reconciliation techniques, and the ability to integrate with diverse data sources and rules engines.
Key Responsibilities:
Data Reconciliation Development:
- Design, develop, and test PySpark-based applications to automate data reconciliation processes across various financial data sources, including relational databases, NoSQL databases, batch files, and real-time data streams.
- Implement efficient data transformation, matching algorithms (deterministic and heuristic) using PySpark and relevant big data frameworks.
- Develop robust error handling and exception management mechanisms to ensure data integrity and system resilience within Spark jobs.
Data Analysis and Matching:
- Collaborate with business analysts and data architects to understand data requirements and matching criteria.
- Analyze and interpret data structures, formats, and relationships to implement effective data matching algorithms using PySpark.
- Work with distributed datasets in Spark, ensuring optimal performance for large-scale data reconciliation.
Rules Engine Integration:
- Integrate PySpark applications with rules engines (e.g., Drools) or equivalent to implement and execute complex data matching rules.
- Develop PySpark code to interact with the rules engine, manage rule execution, and handle rule-based decision-making.
Problem Solving and Gap Analysis:
- Collaborate with cross-functional teams to identify and analyze data gaps and inconsistencies between systems.
- Design and develop PySpark-based solutions to address data integration challenges and ensure data quality.
- Contribute to the development of data governance and quality frameworks within the organization.
Qualifications and Skills:
- Bachelor's degree in Computer Science or a related field.
- 5+ years of hands-on experience in big data development, preferably with exposure to data-intensive applications.
- Strong understanding of data reconciliation principles, techniques, and best practices.
- Proficiency in PySpark, Apache Spark, and related big data technologies for data processing and integration.
- Experience with rules engine integration and development
- Strong analytical and problem-solving skills, with the ability to translate business requirements into technical solutions.
- Excellent communication and collaboration skills to work effectively with business analysts, data architects, and other team members.
- Familiarity with data streaming platforms (e.g., Kafka, Kinesis) and big data technologies (e.g., Hadoop, Hive, HBase) is a plus.
Similar Jobs
Edtech • Social Impact
Own new-logo acquisition for higher education: generate pipeline, run full sales cycles with CS faculty and administrators, close campus partnerships, manage HubSpot pipeline and forecasts, represent CodePath at conferences, and meet quarterly new-partner and enrollment targets. Expect regular travel to events and campus meetings.
Top Skills:
Hubspot
Edtech • Social Impact
Embed with teams to audit workflows, build and productionize LLM/agent automations (using Claude and integrations), create reusable assets and guardrails, coach teams to self-sufficiency, and measure outcomes to prioritize future engagements.
Top Skills:
AsanaClaudeClaude CodeGoogle WorkspaceHubspotLlm ApisLow-Code/No-Code Agent FrameworksMcp IntegrationsNotionSlackZendesk
Artificial Intelligence • Professional Services • Business Intelligence • Consulting • Cybersecurity • Generative AI
Lead and manage internal investigations, develop and implement compliance policies, advise on regulatory requirements, analyze operational risks, communicate findings to stakeholders, coach and lead teams, and support compliance program implementation and training to strengthen internal controls and ethical standards.
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
