Project scope
Categories
Cloud technologies Data analysis Data modelling Information technology Data scienceSkills
data quality mulesoft anypoint extract transform load (etl) data governance data lineage datadog governance open source technology quality assessment amazon redshiftData governance in legacy systems can be tough to implement because those systems are often inflexible and unmaintainable. TechInsights needs to rationalize and document its existing data flows/ETLs.
In this project, learners will become familiar with AWS Data Zone and the Open Source Great Expectations too to document the complete Data Lineage of several ETL flows and implement reliable data quality rules so that TechInsights can establish and entrench better Data Governance, quality, and monitoring.
AWS Data Zone champions and implements the OpenLineage standard. This industry standard seeks to wrangle the complex ETL industry (100s of companies) towards data governance. TechInsights wants to adopt this standard and the AWS Data Zone tool
Great Expectations is an OpenSource tool that ETL authors and maintainers to specify automated quality checks on ETL sources and targets to ensure data is complete and compliant.
Learners will
- Learn and Setup AWS DataZone
- Learn and Setup Great Expectations
- Describe several ETL flows in DataZone including their full lineage and demonstrate to stakeholders
- Determine how to implement Data Lineage for ETLs written in Mulesoft AnyPoint and Composer
- Determine how best to implement data quality checks in Great Expectations
- Implement simple ETLs in AWS ZeroETL with full monitoring in DataDog
Providing specialized, in-depth knowledge and general industry insights for a comprehensive understanding.
Sharing knowledge in specific technical skills, techniques, methodologies required for the project.
Providing access to necessary tools, software, and resources required for project completion.
Scheduled check-ins to discuss progress, address challenges, and provide feedback.
Supported causes
The global challenges this project addresses, aligning with the United Nations Sustainable Development Goals (SDGs). Learn more about all 17 SDGs here.
About the company
TechInsights supports innovation and decision making in the semiconductor industry through its independent research and analysis. We exist to support our customers in advancing the world around us. TechInsights is the semiconductor industry’s authoritative platform.