Fire. Agriculture. Bronze. Iron. The printing press. The steam engine. Penicillin. The internet.

All leaps for humanity that completely transformed an entire civilization, practically overnight — at least when compared to most other innovations.

Each of these was a true leap forward.

The modern implementation of databases sits among them.

Between the internet and AI?

The biggest invention in between — by far — is the database.

And it hasn’t remained static. The way we create and use databases is practically changing by the year.

And with the growing prevalence of AI?

Practically by the week.

In this series of articles, we want to take a look at each step of this journey.

From punch cards, through the invention of SQL and the rivalry between IBM and Oracle, the unprecedented scaling efforts of Google and Amazon, all the way to how AI is changing the databases landscape.

First up?

The very start.

A clunky approximation of data storage

Before the printing press, people used to copy books by hand. Probably wasn’t a great experience, but it was a necessary one.

Some books — like the Bible — could take up to and over a year to copy.

And of course, scribes could make mistakes, or even have to restart the process if they weren’t careful enough.

And vellum and parchment? Expensive for the time.

And the books could rot and deteriorate quite quickly if not stored properly. 

And stealing a book would be as easy as just reaching out, taking it, and leaving.

So:

• You can’t store data safely
• You can’t keep data secure
• You can’t copy data quickly
• You can’t retain data integrity

Pretty much the antithesis of what we have today.

But again, there was just no other way. This wider copying of books happened during the thirteenth century.

And 8 centuries later?

Almost the same.

Past the industrial revolution — almost the same

Say you’re a government official, and you want to figure out how many people live in your area. You go to the census bureau.

Birth certificates? The hospital.

Loans? The bank.

Payroll? Company warehouses.

You go to a place, you ask a clerk, and they check a veritable library of information for you.

This process was not just slow and clunky, but it also cost money — clerks would have to be trained and paid, money would be spent on the paper itself, and of course, entire warehouses had to be rented.

Even better? For census, for example, clerks would have to go door to door to gather the necessary information.

No cross-referencing either. Want to know how many Italian-born men over 40 worked in manufacturing across three states? A team of clerks would have to go through every relevant pile and tally manually.

Every new question one might have required entirely manual work from scratch.

Still not ideal.

Even as computers entered offices in the 70s and 80s, for most people the experience didn't change — you still went somewhere, asked someone, and waited. The database revolution hadn't reached the surface yet.

Until the modern database.

But to get to that, we have to go back again. 

It all starts, surprisingly, in a silk factory in Lyon.

Joseph Marie Jacquard kind of invents punch cards

Jacquard was born in Lyon, France, in 1752, and he would revolutionize the looming industry forever.

What’s looming?

In simple terms, it’s a machine that holds one row of threads in place so that another row of threads can be interwoven around them.

You use a loom to create cloth, carpets, textile — that kind of stuff.

The machines themselves are immensely complex, but that’s all you need to know to understand this article.

Jacquard was almost predestined to create the punch card.

His father, Jean Charles dit Jacquard, was already renowned as a master weaver. And his brother-in-law, Jean-Marie Barret, ran a printing and book selling business.

Somewhat ironically, Jacquard struggled to work at his father’s factories, and instead went for his brother-in-law’s printing and bookbinding business.

But when his father died in 1772, Jacquard inherited his father’s looming business.

His life over the next 30 years is historically contentious — he might’ve claimed to be a master weaver, might’ve bankrupted a few businesses, might’ve fought in the Rhine campaign of 1795, and might’ve lost his son.

But one thing is clear — right at the turn of the 19th century, he became an inventor.

All the devices he came up with revolved around looms, but it’s the one he invented in 1804 that we’ll be focusing on.

Because that one? It kind of used a binary system of punch cards.

A loom would be equipped with a long series of these punch cards, each row of dots representing a row of threads.

Once the punch card is used up?

Press on a pedal with your foot, and the next punch card automatically makes its way down so it can be used.

Rinse and repeat.

And eventually? It was fully automated.

Despite the use case, this really was the first new way to store data, allowing weavers to replicate complex designs easier, faster, and with consistent results.

Input → storage → output.

That’s data.

When Babbage was theorizing the first computer? He turned to Jacquard’s loom. Ada Lovelace recognized its importance too.

Here’s a great video on the topic:

 Oh, and there’s one more pioneer who was inspired by the Jacquard loom.

And he, incidentally, is the next person we’re talking about.

Herman Hollerith saves the US census

It’s 1890 and the US census is growing at a truly unprecedented pace.

Not just for the US itself, but for the world. Unprecedented rapid growth.

In 1880, the census had taken eight years to count. But the number of people (and other data points) were growing faster than the census could count.

An impossible task.

So how do you keep track of all those data points, growing rapidly, especially across the immense span of the US territory?

You can’t. Not really. And definitely not by hand, which is what people did at the time.

The previous cycle had taken eight years. This one would take ten. And by the end of it? They would already be behind by X amount.

Herman Hollerith had an idea.

Born in Buffalo, New York, he was a child of German immigrants. 

He eventually landed at MIT, and participated in the 1880 census himself.

And there, he saw the problem.

Remember Jacquard?

Hollerith did.

His solution was to take Jacquard's core insight — encode information as a pattern of holes in cards — and point it at a completely different problem. Not weaving patterns, but recording people.

Each person who needed to be recorded received a punch card.

Of course, clerks had to go around and give them out personally, but it was still a vast improvement.

The card collected essential information — age, sex, occupation, birthplace, marital status, etc. Even more niche stuff — like how many sheep you own. Yes, really. Anything that was statistically significant for a US government in 1890.

And of course, it was still binary — a hole for “yes”, no hole for “no”.

Now, the way it worked is also a bit complicated, so we’ll keep it simple.

First, a clerk would go around and interview people. Then, they’d used a handheld punch — also designed by Hollerith — to make the process as fast and as smooth as possible.

Once a punch card was ready, it would be tallied up using Hollerith’s census machine.

How does that work?

Here we go.

1. A card would be placed on the plate.
2. A press above it — equipped with spring-loaded pins — would be lowered onto the card.
3. Because of the springs, pins would only pass through the holes punched into the card.
4. Below the card are small cups of mercury — each cup corresponding to a potential hole in the punch card. Mercury conducts electricity.
5. Once a pin passes through a hole, it connects with the mercury — closing a circuit — and sending an electrical signal to an odometer. What’s an odometer?  It’s the same thing that’s in your car that shows your current speed.
6. Each odometer — 40 in total — would go all the way to 9,999.
7. Once an odometer goes past 9,999? It rolls over — back to 0.

If that still sounds a bit clunky, it’s because it was.

Operators would need to track every rollover and keep track of the actual numbers by hand.

But it still significantly improve speed, efficiency, and reliability.

But you might be wondering:

Ok, if there’s so many data points they used to collect… how come there’s only 40 odometers on the machine?

First pass — track population totals and age brackets. Record the numbers. Reset the dials. Run the same cards through again for a different set of categories. Occupation this time. Record. Reset. Run again for birthplace. And so on.

Despite not being an original Hollerith handheld punch, here’s a video of how a handhold punch works.

This time, the process of collecting data took five years. It took another four-and-a-half years to fully complete the 1890 census.

About the same time as the 1880 census — but there were about 16 million more people. Without Hollerith’s census machine, one can only imagine how long it would’ve taken.

Later, Hollerith would create a company that focused around his technology.

The company he creates? It’ll eventually grow into something you might be familiar with.

IBM.

Laying the foundations

Beginnings are often humble, but not this time.

The written word is one of those things that has been around forever - and still is, even in its primal form.

But the 19th century was a time of explosive, rapid, global innovation.

Production needed to be faster. Growth followed. Then counting growth needed to be faster.

And if you think about it, counting stuff and keeping track is one of the absolutely most quintessential things for a functioning society.

Today we have that.

All thanks to a Frenchman who wanted more consistent textile and a yankee who wanted to know what goes past 9,999.

So what's next?

In Part II, we'll be taking a detour through the history of magnetic tape storage. A technology that vastly improved the amount of data you could store on one medium.

After that, it's on to the main event — Charles Bachman, who created the blueprint for all future database management systems.

To this day.

See you in the next one!