Generative AI tools such as ChatGPT, Claude, and Bard are making data engineering more accessible and more efficient.

• If you work with spreadsheets or business intelligence tools but aren't too familiar with Python or SQL, then generative AI can help you analyze data and build your own data pipelines and ETL/ELT processes.

• If you are a data engineer, then GenAI can help you focus your efforts on the problem domain and designing a data architecture while spending less time writing code that can be generated by a machine.

Generative AI and LLMs will not replace data engineers or data analysts but those who know how to use these AI tools will be able to build more capable and reliable data pipelines faster. They will also have access to a tool that can help you develop your Python, SQL, and data modeling skills by providing a variety of examples of functional code and help with error messages and troubleshooting processes that do not work as expected.

Learn Data Engineering Techniques as Well as Data Engineering Tools

In this course, you will learn how to break down data engineering problems into a series of tasks that can be automated using Python, SQL, and command line scripts generated by a large language model (LLM). Prompting an AI to "generate a data pipeline to do X, Y, and Z" will probably not get you the results you expect. LLMs are powerful tools, but they are not oracles. As with any tool, we need to understand what the tool is capable of and how to use the capabilities to meet our needs.

This course shows you how to think through a data engineering problem, incrementally build components of a solution, and combine those components into functional data pipelines.

This course is organized into several topics that cover the fundamental skills needed to begin work in data engineering using GenAI, including:

• Introduction to large language models, foundation models, and other AI topics related to data engineering. This course uses Claude AI from Anthropic, a large language model that is both well suited to data engineering code generation and free to use.

• Working with CSV and JSON files

• Data quality and data cleaning, including statistics and visualizations

• Extraction transformation and load (ETL)/ extraction, load, and transform (ELT) processes

• Relational and NoSQL databases

• Data modeling using dimensional data model patterns

• Working with JSON data in relational databases such as PostgreSQL

• Understanding more advanced components of the modern data stack, including Apache Airflow, Apache Spark, Great Expectations and dbt

The course begins with the most basic of data engineering tasks: working with files. You will learn how to quickly filter, transform, and find problems in data sets made up of comma-separated value (CSV) and JSON files. You'll also see how we can create samples from large data sets to efficiently experiment with different solutions to our data engineer needs. You will learn how to generate code that uses command line utilities like awk, a text processing and data extraction tool, and jq, a tool for parsing, filtering, and transforming JSON data. If you are not familiar with tools like awk and jq, that is no problem. In this course, you will learn how to describe what you want in a solution so the LLM can choose an appropriate tool for the job.