Member-only story
Agentic AI for Data Engineering
Reimagining Enterprise Data Management leveraging AI Agents
1. Introduction
The discussion around ChatGPT (in general, generative AI), has now evolved into agentic AI. While ChatGPT is primarily a chatbot that can generate text responses, AI agents can execute complex tasks autonomously, e.g., make a sale, plan a trip, make a flight booking, book a contractor to do a house job, order a pizza. Fig. 1 below illustrates the evolution of agentic AI systems.
Bill Gates recently envisioned a future where we would have an AI agent that is able to process and respond to natural language and accomplish a number of different tasks. Gates used planning a trip as an example.
Ordinarily, this would involve booking your hotel, flights, restaurants, etc. on your own. But an AI agent would be able to use its knowledge of your preferences to book and purchase those things on your behalf.
The key characteristics of agentic AI systems are their autonomy and reasoning prowess that allow them to decompose complex tasks into smaller executable tasks, and then orchestrate their execution in a way that can monitor, reflect, and adapt / self-correct the execution as and when needed. Given this,
agentic AI has the potential to disrupt almost every business process prevalent in an enterprise today.
In this article, we take the example of one such process from a software engineering point of view. While we all agree that good quality data is essential to providing the competitive edge to both generative AI and agentic AI solutions;
here we show how the data management process itself can be re-engineered leveraging agentic AI.
In particular, we show how agentic AI can be applied to two core data management processes: data cataloging and data…
