This weekend, as I was planning to write about how AI can accelerate the data pipeline process, I started by sketching out a typical ETL flow using the trusty pen and paper. In just a few minutes, I had diagrammed out how data moves through a medallion architecture, from the bronze layer of raw data to the gold layer of refined, analysis-ready data.

On a whim, I decided to snap a photo of my sketch with my phone and feed it into an AI image-to-code model, in this case Google’s Gemini, along with this prompt:

Generate the mermaid code to produce this flow chart which shows the ETL process of data moving through a medallion architecture data pipeline

ETL on Paper

To my amazement, with just that simple sketch and one-line prompt, the LLM was able to generate a nearly perfect mermaid code snippet to recreate my flow chart digitally:

graph LR
    subgraph Raw
        DB[(DB)]
    end
    subgraph Bronze
        DB -- Extract --> NormalizedTable1["Normalized Table"]
        DB -- Extract --> NormalizedTable2["Normalized Table"]
    end
    subgraph Silver
        NormalizedTable1 -- Join --> DenormalizedTable["Denormalized Table"]
        NormalizedTable2 -- Join --> DenormalizedTable
    end
    subgraph Gold
        DenormalizedTable -- Aggregate --> AggTable["Agg Table"]
    end

graph LR
    subgraph Raw
        DB[(DB)]
    end
    subgraph Bronze
        DB -- Extract --> NormalizedTable1["Normalized Table"]
        DB -- Extract --> NormalizedTable2["Normalized Table"]
    end
    subgraph Silver
        NormalizedTable1 -- Join --> DenormalizedTable["Denormalized Table"]
        NormalizedTable2 -- Join --> DenormalizedTable
    end
    subgraph Gold
        DenormalizedTable -- Aggregate --> AggTable["Agg Table"]
    end

Designer, architects, and engineers no longer need to be bogged down with the intricacies of diagramming tools or mermaid syntax. Instead, they can quickly iterate on ideas using nothing more than a pen and paper.

This got me thinking - in a world with such advanced AI and LLMs, perhaps the most powerful human-machine interface is the same one we’ve been using for centuries: pen and paper. There’s something to be said for the speed and flexibility of sketching out ideas on a piece of paper.

Coupled with AI superpowers that can almost instantly translate those sketches into working code, pen and paper becomes a surprisingly potent tool for modern data engineering. Sometimes, to make a leap forward, we need to take a step back and leverage the tools that have served us so well in the past.

So next time you’re designing a complex engineering design, you may want to start with pen and paper. Your productivity will thank you for it.