DescriptionAbout Us: Join our innovative Digital Intelligence team at J.P. Morgan, where we leverage cutting-edge technology to drive data-driven decision-making and enhance business performance. We are seeking a talented and motivated Data Engineer - Databricks to join our team and contribute to our mission of transforming data into actionable insights.
Job Description:
As a Data Engineer - Databricks in our Digital Intelligence team, you will play a crucial role in designing, developing, and maintaining scalable data processing solutions using Databricks, Python, and AWS. You will collaborate with cross-functional teams to deliver high-quality data solutions that support our business objectives.
Job Responsibilities:
- Execute software solutions, design, development, and technical troubleshooting with the ability to think beyond routine or conventional approaches to build solutions or break down technical problems.
- Create secure and high-quality production code and maintain algorithms that run synchronously with appropriate systems.
- Produce architecture and design artifacts for complex applications while being accountable for ensuring design constraints are met by software code development.
- Gather, analyze, synthesize, and develop visualizations and reporting from large, diverse data sets in service of continuous improvement of software applications and systems.
- Proactively identify hidden problems and patterns in data and use these insights to drive improvements to coding hygiene and system architecture.
- Contribute to software engineering communities of practice and events that explore new and emerging technologies.
- Provide guidance to the immediate team of software engineers on daily tasks and activities.
- Set the overall guidance and expectations for team output, practices, and collaboration.
- Anticipate dependencies with other teams to deliver products and applications in line with business requirements.
- Manage stakeholder relationships and the team’s work in accordance with compliance standards, service level agreements, and business requirements.
Required Qualifications, Capabilities, and Skills:
- Bachelor’s degree in Computer Science, Information Technology, or a related field.
- Formal training or certification on software engineering concepts and applied experience.
- Hands-on experience in data mapping, data architecture, and data modeling on Databricks.
- Extensive experience in AWS, design, implementation, and maintenance of data pipelines using Python, PySpark on Databricks.
- Proficient in Python and PySpark, able to write and execute complex queries to perform curation and build views required by end users (single and multi-dimensional).
- Strong understanding of front-end and back-end technologies, with a focus on creating seamless user experiences.
- Extensive experience in Databricks data engineering, data warehousing concepts, ETL processes (Job Runs, Data Ingestion and Delta Live Tables, Spark Streaming).
- Experienced in standing up and maintaining EC2/ECS instances, S3, Glue, and Lambda services.
- Experience in building Notebooks with complex code structures and debugging failed jobs.
- Proven experience in performance and tuning to ensure jobs are running at optimal levels and no performance bottleneck.
- Proven ability to deliver high-quality features into production systems in a rapid-paced, iterative development environment.
Preferred Qualifications, Capabilities, and Skills:
- Experience with machine learning and data science workflows.
- Familiarity with data visualization tools and techniques.
- Knowledge of data governance and security best practices.
- Experience in carrying out data analysis to support business insights.