Data Engineering/ Data Migration Lead

Job Description

Join our team as a Data Engineering / Data Migration Lead and lead the exciting migration of data workloads from SAS to Databricks for a major brand client - start immediately! Job Type : Contract (with potential for full-time permanent position) About INGRITY INGRITY is a Microsoft Data and AI Solution Partner dedicated to delivering transformative data and AI-driven solutions. We collaborate with some of the best ASX-listed companies and medium-sized businesses, working closely with Microsoft to create measurable value for our customers. Our success is built on innovation, customer advocacy, and a strategic partnership with Microsoft. At INGRITY Imagine the Impossible : It's not just an aspiration for our clients; we expect the same from our team. INGRITY isn't your typical workplace; we thrive on the fusion of Intelligence & Grit. Prioritising personal growth, open communication, continuous learning, career progression, and accountability, we're on the lookout for individuals like you. If you seek roles fostering personal growth, flourish in a culture that values open communication and diverse perspectives, are passionate about expanding your skills, and have integrity and accountability at the core of your values, we invite you to explore opportunities with us and join a team committed to data smart excellence. About the job We are seeking a skilled and experienced Data Engineering / Data Migration Lead to join our team and lead the migration of data processing and analytics workloads from SAS to Databricks for one of our major brand clients. The ideal candidate will possess a deep understanding of data engineering principles and have hands-on experience with both SAS and Databricks platforms. This role requires collaboration with cross-functional teams to ensure a seamless transition, optimize data pipelines, and enhance our data infrastructure. This position is available to start immediately. Key Responsibilities : Lead the planning and execution of migrating data and analytics workloads from SAS to Databricks. Recreate existing SAS data transformations and business logic in Databricks using Spark SQL, PySpark, or Scala. Design, develop, and optimize data pipelines and ETL processes in Databricks to ensure high performance and scalability. Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and ensure alignment with business goals. Implement best practices for data governance, security, and quality assurance throughout the migration process. Troubleshoot and resolve issues related to data migration and processing, ensuring data integrity and accuracy. Provide technical leadership and mentorship to team members. Document processes, workflows, and migration strategies for future reference and compliance. Requirements : 5+ years of solid experience in leading data engineering / data migration projects, specifically with a focus on migrating from SAS to Databricks platforms. Proven expertise in SAS and Databricks, with a strong understanding of both environments. Proficient in programming languages such as Python, Scala, or SQL, with a solid understanding of Spark. Strong knowledge of ETL processes, data modelling, and data warehousing concepts. Experience with cloud platforms (such as Azure and AWS) and their data services, particularly in relation to Databricks. Excellent problem-solving skills, attention to detail, and the ability to work independently and as part of a team. Strong communication skills, with the ability to convey technical concepts to non-technical stakeholders. Additional Information : Eligibility : Candidates must have full working rights in Australia to be considered for this role. #J-18808-Ljbffr