AWS Bedrock Knowledge Bases - AI Search on SQL and Graph Data

TechInsights aims to enhance its SaaS platform's search capabilities by integrating advanced AI/ML technologies. Our current Lucene-based search system, while effective, might be superseded or at least complemented by a disruptive GenAI solution. Like most companies, TechInsights stores important information in relational databases. This information is not currently indexed by our platform's search engine, making it difficult for our platform's users to find this information through our search interface. In this project, learners will develop AWS Bedrock Knowledge Bases in an agentic AI workflow to query our structured data stores in order to respond to natural language user queries using LLMs and the latest AI techniques. An AWS Bedrock Knowledge Base translates a natural language query into a SQL or Cypher query so that a human user can retrieve information that TechInsights has stored in a relational database without knowing SQL, Cypher or our storage schemas. Learners will Help select LLMs to use Develop AI Agents Develop and explain queries to the agent Fine-tune agent behaviour and performance Establish Bedrock Guardrails (leveraging AWS' Automated Reasoning) Work with SMEs to refine results Work with QA to validate results Calculate search statistics like precision and recall The Bedrock Knowledge Bases learners develop will underpin chat-based search and will also later underpin a Semantic Search feature will will add to our Lucene-based search engine. This project will provide a superb opportunity for learners to apply their knowledge of AI/ML and information retrieval systems in a real-world context. Learners will interact regularly with TechInsights' highly trained software Developers, Engineers, and Managers as well as product ownership/management to develop powerful state-of-the-art solutions.

Data Lineage In AWS Data Zone and ETL Governance in Great Expectations

Data governance in legacy systems can be tough to implement because those systems are often inflexible and unmaintainable. TechInsights needs to rationalize and document its existing data flows/ETLs. In this project, learners will become familiar with AWS Data Zone and the Open Source Great Expectations too to document the complete Data Lineage of several ETL flows and implement reliable data quality rules so that TechInsights can establish and entrench better Data Governance, quality, and monitoring. AWS Data Zone champions and implements the OpenLineage standard. This industry standard seeks to wrangle the complex ETL industry (100s of companies) towards data governance. TechInsights wants to adopt this standard and the AWS Data Zone tool Great Expectations is an OpenSource tool that ETL authors and maintainers to specify automated quality checks on ETL sources and targets to ensure data is complete and compliant.

Enhancing Search with AI/ML for TechInsights' Customers

TechInsights aims to enhance its SaaS platform's search capabilities by integrating advanced AI/ML technologies. The current Lucene-based search system, while effective, can be significantly improved to provide more relevant and intuitive search results for users. The project is broken into 4 phases. The first involves fine-tuning the existing search solution based on continuous analyst feedback to ensure it meets user needs. The second will develop AI/ML-based semantic search to better address user queries and deliver more relevant results. Further, by analyzing consumption behaviors, an additional AI/ML model will be developed to identify likely related content, enriching search results with valuable insights. Lastly, by training on our content, an AI/ML model will be developed that can identify and rank the relevance of related content drawn from among our data set. This project provides a superb opportunity for learners to apply their knowledge of AI/ML and information retrieval systems in a real-world context. Learners will interact regularly with TechInsights' highly trained software developers, Engineers and Architects as well as product and program management.

Calculate and Expose Software Development DORA Metrics in Tableau and Postgres

TechInsights' operates a professional software development team that has adopted the industry's standard performance metrics - the DORA metrics. In order for these performance metrics to be useful, they need to be live (kept up to date at high frequency) and they need to be accurate and tailored to our approach and team. The goal of this project is to calculate and visualize our KPIs in near real-time for ourselves and our stakeholders to consume from a web portal.