Databricks Developer

Empresa: Cognizant Technology Solutions
Provincia: Madrid
Población: 
Descripción: As a Databricks Developer, you will design and build the enterprise data pipelines that power analytics, reporting and AI initiatives for a leading company in the energy sector. Join a fully remote data engineering team working hands-on with cutting-edge Lakehouse technology. HIGH-IMPACT DATA PROJECTS LATEST LAKEHOUSE TECH FULLY REMOTE LEARNING and GROWTH Don´t tick every box? If you meet around 70% of the requirements above, we´d still encourage you to apply. ABOUT THE ROLE We are looking for a highly skilled Data Engineer with 5+ years of experience to design, build and optimize enterprise data pipelines on the Databricks Lakehouse platform for a leading energy sector company. In this role, you will be the hands-on technical driver responsible for transforming raw data into high-quality, actionable datasets. You will build and maintain a Medallion architecture, optimize Spark workloads, and ensure the data infrastructure seamlessly supports advanced analytics, BI dashboards and emerging Generative AI applications. KEY RESPONSIBILITIES Data Pipeline Engineering • Design, build and maintain scalable, robust ETL/ELT pipelines using Python, SQL and Apache Spark within the Databricks environment. • Implement and manage a robust Medallion architecture (Bronze, Silver, Gold layers) to process and refine data from diverse sources. • Develop and maintain the Gold semantic layer specifically optimized for high-performance consumption by BI tools (e.g., Power BI). Platform Optimization and Architecture • Optimize Databricks workloads, cluster configurations and Spark queries to ensure high performance and cost efficiency. • Work extensively with open table formats, specifically Delta Lake and Apache Iceberg, to ensure ACID compliance, time travel and efficient data storage. • Execute complex data migrations, including transitioning legacy workloads from traditional cloud data warehouses (e.g., AWS Redshift) into the Databricks Lakehouse. Data Governance and Automation • Implement data governance and access control policies at the table, row and column levels using Databricks Unity Catalog. • Automate deployment processes and pipeline orchestration using Databricks Workflows, CI/CD pipelines (e.g., GitHub Actions, Azure DevOps) and tools like Terraform. • Embed data quality checks and monitoring directly into pipelines to ensure strict Master Data Management (MDM) standards are upheld. AI and Advanced Analytics Support • Collaborate closely with Data Scientists and AI Engineers to provision clean, structured data for machine learning model training and inference. • Support the data foundations required for GenAI frameworks, autonomous agents and AI observability platforms. REQUIRED SKILLS and EXPERIENCE • 5+ years of dedicated data engineering experience in an enterprise environment. • Expert-level proficiency in Python and SQL. • Extensive hands-on experience with Data...
Tecnologías: Databricks, ETL, Python, SQL, spark
Tipo de Contrato: Indefinido
Salario: Sin especificar
Experiencia: Más de 5 años
Funciones: Big Data - Programador
Descubre más: https://www.tecnoempleo.com/databricks-developer-cognizant-technology-solution/etl-python/rf-80fe1ff3224d53040d42