Senior Data Engineer
Job description
We are seeking an experienced Data Engineer to design, build, and optimize scalable big data solutions. This role involves developing high-performance ETL pipelines using Apache Spark across major cloud platforms (Azure, AWS, or GCP), with a strong emphasis on data engineering best practices and system performance.
Key responsibilities
- Lead the end-to-end design and optimization of Databricks ETL pipelines to ensure high-quality data delivery. - Develop scalable data ingestion and transformation workflows by leveraging the full power of the Apache Spark framework. - Manage complex data lifecycles involving both structured and unstructured datasets from diverse originating sources. - Fine-tune Spark jobs to balance peak performance with cost-efficiency and long-term scalability. - Establish robust CI/CD patterns and version control standards for all Databricks-based workflows. - Collaborate with stakeholders across engineering, data science, and analytics to translate business needs into technical solutions. - Enforce rigorous security protocols and access management policies within the Databricks environment. - Diagnose and resolve complex bottlenecks related to big data processing and system scalability.
Requirements
Databricks Functional SAP knowledge on finance modules
Job overview
Number of positions
Salary range
Posted date
About Texplorers
Founded
Industry