Data Migration Lead (Data Engineering)

Job Description

Join our team as a Data Migration Lead and take the lead in an exciting migration of data workloads from SAS to Databricks for a major brand client. If you are passionate about Databricks, Spark, and cloud-based data engineering , this is your opportunity to drive a high-impact, transformative project . Immediate start available! Job Type: Contract (with potential for full-time permanent position). About INGRITY: INGRITY is a Microsoft Data and AI Solution Partner dedicated to delivering transformative data and AI-driven solutions. We collaborate with some of the best ASX-listed companies and medium-sized businesses, working closely with Microsoft to create measurable value for our customers. Our success is built on innovation, customer advocacy, and a strategic partnership with Microsoft. At INGRITY: Imagine the Impossible: It's not just an aspiration for our clients; we expect the same from our team. INGRITY isn't your typical workplace; we thrive on the fusion of Intelligence & Grit. Prioritising personal growth, open communication, continuous learning, career progression, and accountability, we're on the lookout for individuals like you. Do you seek roles fostering personal growth, allowing you to excel? Do you flourish in a culture that values open communication and diverse perspectives? Are you passionate about expanding your skills, eager to join a company that invests in employee development? Are integrity and accountability at the core of your values? If you answered yes to any of these questions, we invite you to explore opportunities with us and join a team committed to data smart excellence. About the job: We are seeking a skilled and experienced Data Engineering/ Data Migration Lead with extensive Databricks expertise to lead the migration of data workloads from SAS to Databricks . The ideal candidate will possess a deep understanding of data engineering principles and a strong grasp of data processing, cloud-based data platforms, and ETL optimisation within Databricks. This role requires collaboration with cross-functional teams to ensure a seamless transition, optimise data pipelines, and enhance our data infrastructure. This position is available to start immediately. Key Responsibilities: Lead the migration of data processing and analytics workloads from SAS to Databricks , ensuring a seamless transition with minimal business disruption. Re-engineer SAS-based transformations and business logic into Databricks using Spark SQL, PySpark, or Scala . Design, develop, and optimise data pipelines and ETL workflows in Databricks for scalability and performance. Work closely with data scientists, analysts, and business stakeholders to align data infrastructure with business needs. Implement best practices for data governance, security, and quality assurance within Databricks. Troubleshoot and resolve migration challenges , ensuring data integrity and operational efficiency. Provide technical leadership and mentorship to the team. Document migration strategies, workflows, and processes for future reference. Requirements: 5+ years of experience in leading Data Engineering/Data Migration projects, with a strong emphasis on Databricks . Proven expertise in Databricks , including cluster management, Delta Lake, and performance optimization . Strong hands-on experience in Spark (PySpark, Scala, or SQL) for data processing. Deep knowledge of ETL processes, data modelling, and data warehousing concepts . Experience with cloud platforms (Azure, AWS) and their data services , particularly in relation to Databricks . Excellent problem-solving and troubleshooting skills. Strong communication skills , with the ability to explain technical concepts to non-technical stakeholders. Additional Information: Eligibility: Candidates must have full working rights in Australia to be considered for this role. Job Type: Contract #J-18808-Ljbffr