I wish I was like you/easily amused

I sometimes like to browse through old text books because it reminds me people used to be able to communicate technical information. On-line articles, especially about technical matters, get bare bones editing. Not that editors are doing a smashing job in other areas. But there were also technical writers who knew how to write and how to work with engineers to create readable prose. And in some classrooms textbooks have disappeared altogether. Why? They are expensive and teachers can cobble together resources that are essentially free.

I was reading one text book and there was an interesting pattern. Updates to a database go into a transaction file. That file is sorted. The update process then creates a new master¹ file, meaning the source of truth, by reading the old source of truth an applying any updates found in the transaction file. If stored correctly, even on the hardware of 50 years ago, this would be fast.

At the end you have three files (four including the log that records the status of the update). You have the old source of truth, the transactions, and the new source of truth. The files are transferred off to tape backups (disk space was a pricey commodity in those days). The next day, the cycle repeats. After a while, you have a series of tapes that represent the state of the source of truth on each day. And the transactions that allowed you to create the next source of truth file.

Let’s say you discover a bug several days in that caused some transactions to be recorded incorrectly. You fix the bug, then load the source of truth from when the bug was introduced, and your replay all the transactions. You can even archive them onto different tapes so that you have the old data and the new data. (Tape was cheap at the time, so buying spools of tapes, given a good reason, was not a problem).

We spent a few years re-discovering this pattern in the 2010s as event based systems. That’s because people realized that just updating the data when a transaction came in was great, but it was hard to re-create what had happened. If a bug appeared, it was hard to go backward in time and “reapply” all the updates. It would be like getting the original eggs back from an omelette. Event based systems allowed you to take data and just play it forward, if you found a bug or decided to store the data in a different way. But even so, event based systems are not as capable as storing tapes.

What’s bad about this approach, of course, is it’s batch based. Meaning you accumulate the transactions in a file and apply them at the end of the day (for example). Meaning your customers and you can’t see the results of the transactions until the next day. And from the 1990s on the goal has been to convince executives that all their data has to be real time. So they can make decisions in the moment. But the never really do. We could still be processing data, batch style, and I think most everything would be fine. The only people it would hurt are those that need to adjust pricing several times a day based on demand and who the customer is. And those people can go fuck themselves.

1. This terminology is controversial and should be avoided, but if you look at older documentation – it is the term used. ↩︎