Position Title: Data Engineer
Reports Directly To: CRISP Insights Project Manager
Location: Columbia, MD
Hours Per Week: 40
Job Summary
The Data Engineer will be responsible for supporting HIT activities in the Baltimore/Washington region. The successful candidate will work to design, develop, and maintain a data analytics platform solution. CRISP is currently running an Azure Data Lake 2 / Azure Data Factory / Spark / Databricks infrastructure with approximately 300 production use cases that consolidate several streams of healthcare data. This role will primarily focus on the creation of new use cases through Databricks programming including communication, technical proficiency, collaboration, and willingness to learn new things. This role assists a senior data engineer to implement new ingestion pipelines and troubleshoot production issues.
This position will have a base salary.
This position requires the candidate to be in the Columbia office 1-2 days a week.
ESSENTIAL DUTIES AND RESPONSIBILITIES
Include the following. Other duties may be assigned.
Create new business data sources from the available data in the data platform:
· Work with source system owners to understand integration needs
· Work with business users to understand reporting needs
· Implement Databricks and Azure Data Factory components to transform the data into business facing datasets
· Perform unit and integration testing of business facing datasets
· Document business use cases
Maintain existing data sources in data platform pipeline:
· Monitor data pipeline job status and troubleshoot issues
· Work with source system owners and business owners to incorporate business changes into data pipeline (i.e., updating metadata file to add new object, update curated layer query, etc.)
· Perform automated data quality checks on data pipelines
· Perform continuous improvement
Qualifications
To perform this job successfully, the incumbent must be able to perform each essential duty satisfactorily. The requirements listed below are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.
Required Skills
· Experience in a production Data Lake or Data Warehouse environment
· Data Integration – Intermediate
· Data Lake Design Patterns – Familiar
· SQL – Intermediate
· Agile Methodologies – Familiar
· Ability to develop programming components to build complex business use cases
· Ability to prioritize and manage multiple job duties, including personal time management and organization skills
· Ability to diagram and understand technical workflows
Preferred Skills
· SQL - Intermediate
· Python - Intermediate
· Cloud Infrastructure (Azure) - Foundational
|
