Senior Data Engineer AWS & GCP at Vigil

This a Full Remote job, the offer is available from: Anywhere SUMMARY: As a Data Engineer, you will be responsible for designing, developing, maintaining, and optimising a data pipeline infrastructure using a proprietary data platform , which is based on Databricks. You will collaborate with cross-functional teams to design and implement scalable data solutions, ensuring efficient data ingestion, transformation, storage, and analysis. WHAT WILL YOU BE DOING: • Data Engineering Ownership: End-to-end creation and upkeep of robust data workflows and components, leveraging tools such as Databricks. This includes designing early-stage prototypes and deploying large-scale data acquisition, handling, and storage strategies. • ETL/ELT Workflow Management: Construct and refine data ingestion pipelines to seamlessly integrate varied data sources into the central platform. Ensure data consistency through validation, cleaning, and enrichment routines. • Stakeholder Collaboration: Partner with commercial teams and data stakeholders to gather and refine specifications for analytical products and visual reporting needs. • Architectural Design & Data Modelling: Engage with analysts and data experts to define efficient data models and architectural plans. Support dashboard and report development, while ensuring optimal data structuring for performance. • Pipeline Performance & Reliability: Continuously assess and fine-tune data workflows, addressing system inefficiencies, integration problems, and data fidelity concerns. • Quality Assurance in Data: Define expectations for data integrity and collaborate with QA to automate validation checks. Utilise monitoring tools to surface and track data quality metrics. • Data Controls & Compliance: Apply internal governance standards and safeguard sensitive data through access rules, encryption protocols, and retention strategies, aligning with organisational security frameworks. • Technical Enablement & Knowledge Sharing: Work closely with interdisciplinary teams to ensure data needs are met, while thoroughly documenting processes and solutions. Convey technical ideas in a way that’s accessible across departments. • Innovation & Process Evolution: Keep informed on developments within the data ecosystem. Advocate for and implement improvements in efficiency, tooling, and automation. Participate in internal knowledge communities. • Agile Delivery Engagement: Contribute to delivery efforts by actively participating in sprint planning, stand-ups, and backlog refinement. Help define and deliver technical tasks, aligning work with CI/CD best practices. WHAT WE ARE LOOKING FOR: • Approximately 4 years of experience in data engineering for mid-level roles • 6 or more years of experience for senior-level data engineering positions • Between 1 to 4 years of experience building data products using platforms such as Databricks, Spark, Cloudera, or HortonWorks • Skilled in Python (especially PySpark), Scala, and SQL • Experienced in designing and implementing scalable data pipelines for high-volume environments • Solid understanding of ELT/ETL practices and data integration strategies • Capable of writing robust production code with automated testing • Familiar with CI/CD tools such as GitHub Actions and Jenkins for deploying code • Hands-on experience with distributed data processing using Apache Spark • Proficient with cloud services including AWS, Azure, or Google Cloud and tools like S3, Glue, Lambda, Redshift, and BigQuery • Good knowledge of data modelling, relational databases, and SQL performance optimisation • Strong analytical and problem-solving abilities, with attention to troubleshooting details • Effective communicator with experience working in collaborative, cross-functional teams • Basic understanding of machine learning concepts such as classification, regression, A/B testing, and experimental design AWESOME BUT NOT REQUIRED: • Understanding of the UK media landscape, including over-the-top (OTT) and traditional broadcast advertising • Familiarity with concepts and trends in digital advertising and marketing analytics • Experience applying statistical approaches such as regression and classification, as well as designing and analysing A/B and other controlled experiments • Awareness of modern data architecture methodologies including Data Mesh, enterprise-level data frameworks, and business intelligence architecture • Solid grasp of data protection practices, regulatory compliance, and governance principles • Exposure to a variety of data management domains such as metadata handling, data quality enforcement, master data systems, and governance tooling • Hands-on experience with data visualisation platforms like Tableau, Looker, AWS QuickSight, and ThoughtSpot for building interactive reporting solutions WHAT’S IN IT FOR YOU? • Be part of our collegial environment where responsibility and authority are shared equally amongst colleagues and help create our company culture • A culture in which we don’t criticise failure but ensure we learn from our mistakes • An Agile environment where your ideas are welcome • The possibility to grow and experience different projects • Ongoing Training & Mentoring • The possibility to travel - ATTENTION! THIS POSITION IS FOR PORTUGAL OR BRAZIL BASED ONLY This offer from "Vigil" has been enriched by Jobgether.com and got a 82% flex score.

Want to see how well you match this job?