Data Platform Owner

• Design and maintain standards for data ingestion, transformation and serving in our cloud data eco-systems  

• Coordinates oversight into existing data pipelines, including the implementation of solutions that will simplify and increase performance. 

• Ensures team follows best practices and execute architected techniques and solutions for data collection, management and usage to aid in company-wide data governance and management framework. 

• Work closely with data analysts and scientists, and database and systems administrators to identify data solutions.  

• Evaluate new data sources for quality and attribution to support product requirements. 

• Documents new/existing pipelines, Data Sets and Data Sets lineage. 

• Drive cost optimization & monitoring strategies across cloud  environments and Databricks. 

• Define strategies for data retention (Archival & Deletion) as per compliance and legal guidelines. 

• Sets team priorities to ensure task completion. 

• Participate in the formulation of departmental objectives, plans, and policies. 

• Acts as a liaison between data engineering, business units, business analysis personnel, and IT infrastructure. 

• Establishes platform engineering best practices. 

• Interprets customer needs and assesses requirements; works with team members to aid in understanding tasks to support the production of solutions. 

• Engages in recruiting and training new staff. 

• Provides concise status updates and challenges to senior management. 

Additionally, The Ideal Candidate May Also Have: 
 

• Must be familiar with Databricks  data intelligence platform (Administration, workspace management, jobs/pipelines, clusters, delta lake, cost opitimization). 

• Must be familiar and hands on delta lake concepts, as well as the ability to fine tune delta lake performance. 

• Must be familiar with the Databricks Declarative Pipeline framework. 

• Familiar with cloud services (AWS preferred, including S3, IAM, EC2, Lambda, Glue, and VPC). 

• Strong communication skills, with the ability to present complex technical solutions in a manner that can be interpreted by a general audience.  This includes proficiency in PowerPoint, Visio, Drawio, etc. 

• Familiar with the software development life cycle as it relates to data integration and data quality. 

• Have good understanding of Data Governance and experience on Data Governance tools like Alation, DataHub, Collate or others. 

• Experience in designing and building production data pipelines from ingestion to consumption within a hybrid data architecture, using PySpark, PySql, Java, Python, Scala, C#, etc. 

• Experience in Designing and implementing scalable and secure data processing pipelines. 

• Strong experience in common data warehouse modelling principles including Kimball, Inmon. 

• Strong experience with data processing frameworks such as Apache Spark and Hadoop 

• Ensuring data quality and consistency through data cleaning, transformation, and integration processes. 

• Must Have Knowledge of Dev-Ops processes (including CI/CD) and Infrastructure as code. 

• Experience in Managing Monitoring and troubleshooting data-related issues within the cloud environment to maintain high availability and performance. 

• Experience establishing data lineage frameworks. 

• Collaborating with data scientists, business analysts, and other stakeholders to understand data requirements and implement appropriate data solutions. 

• Implementing data security measures, including encryption, access controls, and auditing, to protect sensitive information. 

• Automating data pipelines and workflows to streamline data ingestion, processing, and distribution tasks. 

• Knowledge of BI tools like Tableau and Microsoft BI Stack (SSRS/SSAS (Tabular with DAX & OLAP with MDX) SSIS) is desirable. 

• Keeping abreast of the latest AWS features and technologies to enhance data engineering processes and capabilities. 

• Experiece on AWS Sagemaker Unified Studio is a plus. 

• Documenting data procedures, systems, and architectures to maintain clarity and ensure compliance with regulatory standards. 

• Providing guidance and support for data governance, including metadata management, data lineage, and data cataloging. 

• Knowledge of FinOps / cloud cost optimization. 

• Experience implementing Unity catalogue. 

• Skills in navigating interpersonal dynamics, resolving team conflicts while maintaining a positive work culture.