Data Integration Engineer in EPAM Systems

We are seeking a seasoned Data Integration Engineer to take charge of onboarding a new Data Provider.This role involves crafting new pipelines for Data Ingestion, Raw, Curated, and Consolidated stages using Databricks, employing existing templates as a foundation. The ideal candidate will leverage their expertise in Databricks and PySpark to enhance data processing and integration.ResponsibilitiesDesign and develop scalable data pipelines for Ingestion, Raw, Curated, and Consolidated phasesUtilize Databricks to build efficient and reliable data flowsImplement PySpark to process large datasets within the pipelinesCollaborate with data teams to establish pipeline architecture and best practicesOptimize data pipelines for maximum performance and scalabilityEnsure data integrity and quality throughout the pipeline processMonitor and troubleshoot any issues that arise in data pipelinesConduct regular system tests and updates to enhance functionalityDocument pipeline designs and modifications comprehensivelyRequirementsMinimum of 2 years of working experience in data pipeline engineering or a related fieldProficiency in Databricks and PySparkBackground in designing and implementing complex data pipelinesSkills in data processing and workflow managementQualifications in system architecture related to data managementCapability to debug and optimize pipeline performanceNice to haveFamiliarity with Amazon Web ServicesKnowledge of Apache AirflowUnderstanding of Python and SQL programming languagesWe offerWork on a flexible schedule remotely or from any of our comfortable offices or coworking spaces in UkraineReceive the necessary equipment to perform your work tasksChange projects and technology stacks within EPAMGain experience in various business domains (Insurance, E-commerce, Healthcare, Finance, Travelling, Media, Artificial Intelligence, and more)Consider relocation options in over 30 countries worldwideParticipate in volunteer, charity programs and communities (both technical and interest-based)You can plan your individual career path together with your manager.Receive regular feedback from colleaguesImprove your English for free with certified teachers (Speaking Clubs, client interview preparation courses, etc.)Get the opportunity to undergo free training and certification in AWS, GCP, or Azure CloudsUse the internal E-learn training program (18,200+ specialized training and mentoring programs)Access corporate accounts on LinkedIn Learning, Get Abstract and other partner resourcesStudy at EPAM Solution Architecture School with the instructors who are practicing architectsDevelop as a leader, join Delivery Management, Resource Management, Leadership Essentials school and moreParticipate in internal communities (500+ meetups, technical discussions, brainstorming sessions, online events and conferences annually)Vacation and sick leave (including a sick leave without a medical certificate)A wide range of Voluntary Medical Insurance programs providing both medical treatment and various preventive options (including sports activities)Medical insurance for family members at corporate ratesCompany support during significant life events (childbirth or adoption, marriage, etc.)Support for psychological comfort: discounts on services from mental health specialists or coaches, thematic trainingE-kids program - a free programming language training program for EPAMers' childrenEPAM strives to provide its global team of over 52,800+ professionals in more than 55 countries with opportunities for professional growth from day one of collaboration. Our colleagues are the source of EPAM's success, so we value cooperation, strive to always understand our clients' business and aim for the highest quality standards. No matter where you are, you will join a dedicated, diverse community that will help you realize your potential to the fullest. Show more Show less Посадовий рівень Помічник Тип зайнятості Повний робочий день Посадові обов’язки Інформаційні технології Галузі Розробка програмного забезпечення і ІТ-послуги та ІТ Консалтинг