For the sake of argument, let’s assume you can create and to end software machines. Just assume this is true for now.

Right now AI generates code, which is reviewed by a human. The human prompts, provides context, and whatever additional injection, the AI chews on it for something between seconds to tens of minutes, and out pops a module. It could be an API, a library, a command line application, single class, part of a web app or whatever. The human reviews the product, reasoning symbolically over the code, looking for defects of logic rather than text. The LLM is a probabilistic text generator, which is fine, as most new code isn’t necessary novel. There exists a wealth of similar code (text) on which to draw. The human applies a different kind of reasoning, like a pair programming session where one developer knows the syntax and another the algorithm. Is there a race condition here? Could this resource be created once instead of with each invocation of a function? Do the tests check anything meaningful?

The ability to write code, and symbolic – not textual – mental models around that code, are what allow us to evaluate the code. The reason we spend time solving problems like the “dining philosopher’s problem,” or struggle through algorithms, is not to regurgitate that same text ten years later. It’s because we encounter novel problems which are similar to, but not exactly like, well understood problems learned in school. We mash together the novel and well understood elements not by cutting and pasting code, but by applying mathematical logic. Some of us express it more directly in actual mathematical symbols, or tools such as TLA+, but even drawing boxes on a whiteboard utilizes symbolic logic.

Eventually, that mental muscle will atrophy as we read code but produce little. Many AI heavy developers anecdotally report the loss of even basic skills. At some point the loss of skill will become enough of a problem, that code cannot be reliably reviewed. The likely response will be to adopt AI to review the code for the human. At this point it is easy to wonder why a human, with their need for a desk, noise-canceling head-phones, and their expectation they will be served lunch, needs to be paid like a software engineer. Are they an AI engineer? No, they are a machine operator. They are as pivotal to the production of a new piece of software as factory worker is important to stamping out the bit or bob their machine produces. This is what Cory Doctorow calls the reverse centaur, a human who is there because it’s not feasible to automate a part of the process. But the machine is clearly in charge.

What about AI engineers building fantastic AI models? There has been a need for other engineers to produce machines used in factories. Mechanical engineers, electrical engineers, chemical engineers, material scientists, etc. work to design and build better widget stamping machines. But these are nowhere near as common as widget stamping machine operators. And while they make the salary of maybe 5 widget stampers, they are not particularly well compensated with respect to software developers today. It’s likely we’ll need AI specialists to help build better AI machines, but the spectacle of exorbitant salaries is tempered by the realization their jobs will also be more automated. Why would one need hand-made bricks to build a brick factory? You would just buy bricks from another factory.

At this point you might well be wondering why an engineer in the future would make six figures on their first day of work¹. Operating the software machine is not about math or programming skills (which average people are excited about for five minutes but then discover learning to code is a non-addictive sleep aid). Operating the machine requires being trained on the machine (likely by the vendor) and feeding in the inputs for the required outputs. This is no longer a prestige or high salary job. Following instructions and not tinkering with the machine are more important to success than symbolic reasoning over concurrent threads of execution. And being the boss of a bunch of machine operators? That’s what we call a shop foreman today.

I’m not saying all programmers drop off the face of the earth. I’m just saying that the ratio of programmers to AI operators drops to something like the ratio of people who can hand-machine a part to those who at most stick a metal blank in a machine and push a button. There are people who build and repair the machines. They aren’t really artisanal craft people. They are machine makers. Often those person is paid somewhat more than the person feeding the machine blanks and pushes the button. Blank, after blank, after blank. But hand machining is not the same skill as feeding blanks into the machine and hand machining is too expensive. Nor does it guarantee of a better part, if you have a well calibrated and maintained machine.

There are plenty of examples of crafts that have been reduced to people feeding machines at various levels. No one spins yarn by hand except for artisanal craftspeople. A loaf of bread from a fancy bakery is more expensive than a mass-produced loaf in a grocery store. IKEA produces a Billy book-book case every five seconds. If any of the workers that operate the machines at IKEA are woodworkers, it is because woodworking is their after-work hobby. But the person operating the bread machine at a bread factory, the person feeding sheared wool into a spinner, or the person feeding sheet goods into a cutting machine at IKEA are not paid as skilled artisans. They are paid as factory workers. The output of craftspeople, like hand-spun wool, hand-made sweaters, or hand-machined Swiss watches, are usually the fetishes of the wealthy. I just don’t see anyone bragging about their artisanal, hand-code, shell script the same way as a watch.

And this world is a different world. In the past, we fixed or extended software because of the cost of re-writing the software. In this world, if you need a new iteration of a storefront for a retailer, you run it through the machine again and make new software. You keep running it through until the machine produces something you want. There is no need to make understandable, modifiable, high-quality software. It will be thrown away every iteration. There is no point in hiring a skilled weirdo, who hand-codes in their parents’ basement, to fix the site or make changes. Maybe in India a few people who hand-code exist to help make the machines, the way IKEA makes their products in other lower-wage countries with the requisite skills. But you wouldn’t pay developed world wages for those skills.

Don’t get me wrong, you are not going to get the average level of quality. You’re going to get shit. A Billy book-case falls apart in the presence of too much humidity because it’s particle board. I’ve had plenty of Billy book cases with flaking laminate or drooping shelves. Do I get rid of them? No, they just continue to be an eyesore until something else forces me to get rid of the bookcase. Just as bad and broken software will sit around until something motivates a broader change. Possibly with no one going into why it was broken other than to stitch together a vague notion from (possibly wrong) tracing and log messages. But no one is going to read all that code, much less edit and debug that code, because (practically) no one will be able to read it. Certainly not the five-figure code machine operator.

But I don’t believe we can build this software machine. I believe there is an upper limit to the complexity LLMs can incorporate in a purely text-based model. (It’s not even text-based, it’s token based, which is why counting letters in words has been such a challenge). LLMs cannot reason (although they can produce text that looks like reasoning). They cannot process categories of concepts, just text about categories of concepts. If it doesn’t already exist, or is not just within reach by mashing more text together, an LLM cannot imagine or create it. And most of the software it does create is not imaginative or novel. It is the same API, using the same authentication, using the same languages, and the same back-end as thousands of other similar examples, code repositories, and text-book descriptions. As long as you want a store-front that is largely like every other store-front on the web today, and prior to LLMs you would have mashed together from example code and Stack Overflow, you’re going to be replaced by a code machine operator. But then again, were you ever really contributing that much?

1. Rather than a specific number – just assume six-figure salary means well-compensated above the average wage. ↩︎

I have not had great experiences with AI development tools. I’m not a Luddite, but if the tool takes me more time than just doing it by hand, and I get the same results, it’s not a tool I want to use. In some cases, the code had subtle bugs or logic little better than the naive implementation. Or in other cases, the code was not modular and well laid out. Or the autocomplete took more work to clean up than it saved in typing. And in some cases it would bring in dependencies, but the version numbers would date back from the early 2020s. They were out of date and didn’t match to current documentation. For the most part the code worked, but I knew if I accepted the code as is, I would open myself (or whoever followed me) to maintenance issues at some later date. One might argue that the AI would be much better then, and could do the yeoman’s work of making changes, but I wasn’t sold on that idea. (And I’m still skeptical).

I would turn on the AI features, use them for a while, but eventually turn them off. I found it helped me with libraries with which I wasn’t familiar. Give me a few working lines of code and a config to get me going, and I’ll fix it up. It would save me a search on the internet for an out-of-date Stack Overflow article, I guess. I used prompt files and tried to keep the requests narrow and focused. But sometimes, writing a hundred lines of markdown and a four sentence prompt to get a function, didn’t seem scalable.

Well, pigs are flying and I found something that appears to be working. First, it involves a specific product. At the time of writing Claude Opus/Sonnet 4.5 seem to be quite good. Second, I have a better way of incorporating prompt files in my workflow (more on that below). Third, language matters. I found Claude gave me the same problems listed above when working on a Jakarta EE code base. But Rust is great. (Rust also has the advantage of being strongly typed and eliminating some of the issues I’ve had when working with Python and LLMs). Fourth, I apply the advice about keeping changes small and focused. Fifth, I refactor the code to make it crisp and tight. (More on that below). Sixth, ask the LLM for a quick code review. (More on that below).

The first topic I’ll expand on is changing my relationship with prompt files. Instead of attempting to create prompt files for an existing code base, I started writing a prompt file for a brand new project. I had Claude generate a starter and then added my edits. I believe in design (at least enough to show you’ve thought about the problem). This actually dovetails with my need to think through the problem before coding. I still find writing prompt files for an existing code base tedious. But, if I have to sit down and think about my architecture and what pieces should do, the prompt file is as good a place as any.

The other thing I want to cover is refactoring what the LLM hath wrought. Claude generated serviceable code. It was on par with the examples provided for the Rust libraries I was using. (Which also happen to be very popular with plenty of on-line examples). Claude would have had access to rich training data and pulled in recent versions (although I had to help a little). But the code is not quite structured correctly. In this case I needed to move it out of the main function and into separate modules. But mostly it was cut and past and let the compiler tell me what’s broken. Next, in the refactor, is to minimize the publicly exposed elements. Now I have code that’s cohesive and loosely coupled. The LLM by itself does not do a great job at this. Taste is more a matter for meat minds than silicon minds at this stage.

The final thing I want to touch on is using the LLM to review the code after refactoring. This gives me another data point on the quality of my code and where I might have had blind spots during the refactoring. I work with lots of meat minds and we review each others’ code on every commit. There are some good reviews and there are some poor reviews. And reviewing code is harder, if you’re not familiar with the specific problem domain. But the machine can give a certain level of review prior to submitting the PR for team review.

So that’s what I’ve found works well so far: green-field projects, in highly popular frameworks and languages, performing design tasks in the prompt file, using LLM best practices, refactoring the code after it’s generated, and a code review before submitting the PR. Auto-complete is still off in the IDE. And I’ll see if this will scale well as code accumulates in the project. But for now, this seems to produce a product with which I am satisfied.

[A small addendum on the nature of invention and why I think this works].

Peoples’ ideas are not unique. As one of my relations by marriage pointed out years go, when he moved above 190th street in Manhattan, there seemed to be a sudden run to the tip of the island to find “affordable” housing. In a city of millions of people, even if very few people have the same idea at the same time, the demand quickly gobbles up the supply. Humans build ideas up mostly from the bits and pieces floating around in the ether. Even “revolutionary” ideas are often traced back to maybe a interesting recombination of existing ideas. Moreover, people have sometimes been doing that “revolutionary” thing before but didn’t connect it to some other need or didn’t bother repacking the idea. What’s more important about “ideas” is not the property of the idea but the execution of the idea.

There is still something about invention, even if it is largely derivative, that the LLMs don’t appear to posses. Nor do they have the ability to reason about problems from logical principles, as they are focused on the construction of language from statistical relationships. Some argue that enough statistics and you can approximate the logical reasoning as well as a human, but I haven’t seen solid examples to date. The LLM doesn’t understand what to create, but it does summarize all the relevant examples necessary to support the work of realizing the idea. But even then, there are gaps that we have to fill in with human intention. Does this revolutionize coding for me? No, I estimate it makes me some percentage more productive, but in the 5-15% range. And of the time and effort necessary to realize an idea, coding is maybe 1/3 or less of the effort. And I worry that we’ll never get to the point this technology will be available at a price point that makes the provider a profit while being affordable to most people. After all, there’s a limit to how much you would spend for a few percentage points of additional productivity.

How does a company that’s hemorrhaging money get to profitability, when they offer a free service? You can create tiers or pay walls to funnel users to paying. This model is popular in the SaaS world, where the free version is a loss leader for future sales. But it isn’t a suitable model for every service. The other avenue to monetization is to show advertisements. It isn’t black and white, with some paid services, like Hulu, still show advertisements. The degree to which advertising is permitted is the degree to which the consumers (businesses or individuals) push back on the advertising.

Strictly speaking, Google and Meta are communication service providers on the SP 500 index. Practically all their money comes from advertising and sponsored content. Amazon and Microsoft are also making significant money from advertising and sponsored content. Your feeds on services like Linked In, X, Facebook, Tik-Tok, YouTube and so on are becoming a gruel of actual content and advertisements, either direct ads through the platform or “creators'” own advertising. New and improved with AI slop to foster more interaction and create more engagement. More of our economy is based on putting an ad in front of someone’s eyeballs than you would imagine. It’s easy to spot some advertising, such as a commercial about a laxative in the middle of a Football game. It’s harder to spot other ads, such as an influencer that doesn’t disclose a payment for a “product review.” The adage that if you aren’t paying for it, you’re the product, is ever more true. Have you thought, for five minutes, how the startups offering free access to LLMs are going to make money?

After thinking about it, I realized companies like OpenAI are better positioned to make money than we realize. First, the injection of cash has turbo-charged their data gathering. There is more investor money to harvest more and more data. I suspect this is also where the first moats for legacy chat-bots will happen, inking deals with content companies. New entrants won’t have the pockets or the bandwidth to negotiate a bunch of little deals to avoid getting sued. But that’s another issue. They are hoovering up everything. There is plenty of evidence they, or their agents, are ignoring any ‘ROBOTS.TXT’ entries that disallow scraping. When actual regulation arrives, it serves more as regulatory capture than creating equitable payments to the sources of content.

Second, we have come to accept that they can wrap your prompt in their secret prompt. These additions to your prompt are hidden, to arguably prevent circumvention. The stated reason to inject those prompts is to prevent leaking dangerous information, such as how to make explosives. They are also part of your terms of service. Attempting to circumvent or discover the prompts is a basis for canceling your account. The account that has your obsequious, pleasant friend on which you’ve come to rely. The point is we are now comfortable, or happily oblivious to, our prompt being wrapped in additional hidden prompts. The easiest way to hide advertising is to keep promotional material secret, like the safety prompts. And to make it a violation of the terms of service to avoid promotional prompting, like the safety prompting. You may even be aware that there is promotional prompting in general, but a specific prompt.

Another way is to selectively return supporting links. For example, if you ask about camping, cold weather clothing, or places to visit in Maine, you might get a link to LL Bean. This is relatively harmless, except that it is different from search, where you can move past the initial results. There is a push for search engines to move from search results to AI results. That may mean, in the future, that you only get the handful of links from the paid advertisers along with the chat response. There may be no button to show more results, or you may have to explicitly ask for more results. Combine that with the advertiser’s ability to modify the hidden prompts injected along with your prompt, and you might lose any awareness of other possibilities. And should the LLM lie about one retailer having the best price, or a particularly well-suited product, that’s chalked up to the hallucinations.

There is also the information you are divulging about yourself. Maybe you are spewing information you would never share on Facebook or even Google Search. For free users, the AI companies are likely to mine all prior conversations, building up a detailed profile. For paid users, mining may depend on the plan and the account, such as a corporate account versus an individual premium account. This is already happening through other social media, but the LLMs may have more detailed information about mental state or health. While it may be more a difference of degree than kind, the chats may have richer data. I suspect the need for vast amounts of storage is to handle the influx and processing of the data you are freely giving them about your internal emotional and psychological state.

What I fear, and may be more deeply concerning, invoving the ability of the LLM to prime you over time. In some sense, search is “one shot.” You type in a search, you get back results. Facebook and other social feeds have been shows to influence peoples’ opinion not on just products, but able to alter their mental health. Their advertising can be better concealed. You might have retweeted or re-posted what were ads in the past. To a degree people have unmasked some of the behavior. We might be more inured to it now, and therefore have a bit of a resistance, but the social media algorithmic rabbit hole is alive and well. We know to watch for “radicalizing” content. What we don’t know how to spot are radicalizing outputs from a chat bot.

LLMs and chat bots may catch us in a particularly vulnerable way. We have a bias to believe the computer’s response is a neutral, disinterested party. And the responses from the LLM are private and highly individual. Not like public feeds on various Apps. If a company that sees sufficient lifetime value in a customer, they may be willing to pay over multiple chats. Maybe a $100 for a couple of months of ‘pushing.’ Imagine if the opioid vendors had access to this technology. Paying a few dollars to push someone toward a prescription for their brand of opiate may be worth thousands of dollars per patient. And each future addict’s chats are essentially customized to that person. Remember, we have plenty of evidence that existing social media can shape opinion and even mental health. Show enough people “PSA” style ads about enough vague symptoms and people will, in fact, ask their doctor if that drug is right for them.

But the big hook is the outsourcing of your cognition. Human beings are inherently lazy. If an escalator is present, almost no-one takes the stairs. Go to the airport and watch people, without luggage, queue for the escalator. The stairs are almost empty and there is just one flight. But they will wait in a press of people. Having a tool that allows you to ‘just get the answer,’ is like your brain being given the option to take the escalator. Instead of thinking through even simple problems, you just pose the prompt to the chat bot. And just like muscle gets soft and atrophies with disuse, your ability to solve problems dwindles. It’s like the person who begins to take the escalator not because it’s a little easier, but because they are now winded when taking the stairs. Need a plan for a workout? This shouldn’t be that hard, but you can just ask the LLM. (Ignoring it may actually give you bad advice, or in a world of sponsored chats, push you toward products and services you don’t need). Need a date idea? Just ask the LLM. Is your back pain something to worry about? The LLM has a short answer.

At least reading search results might inadvertently expose you to a knowledgeable and objective opinion between ads. If I search on Google for US passport applications, the first link is actually a sponsored link to a company that will collect all my data and submit my passport application for me. Who is this company? I’ve never heard of them. It ends in a “.us” domain, making it seem US related, but who knows what they do with the data or how they store it. The second link is the state department, but the third link is not. The only reason the state department is there, is because they paid to sponsor a search result. But at least it’s there. And it’s also in the list of general results. Google, Facebook, Tik-Tok, and so on have a track record of taking advertiser money from almost anyone. Amazon’s sponsored content is sometimes for knock-off or counterfeit products. And some sites have absolutely no scruples on the ads they serve, ads which might originate from Google or Meta ad services.

The lack of scruples or selectivity demonstrated by other on-line services that take advertising, combined with the outsourcing of cognition, means you are exposing yourself to some of the shittiest people on the face of the earth. For every time you are pushed toward buying a Honda, you might also be pushed toward taking a supplement that is dangerous to your health. You will likely be unaware you are being marketed to, and in ways that are completely personal and uniquely effective on your psyche. In a state of mind where you’re being trained to expect an objective result, with additional prompts that are invisible to you for “safety,” and a technology whose operation is inscrutable, you have no idea why you are provided with a given answer. Is it your idea not to buy a car at all and just use ride share services every day? If the ride share services want the behavior to stick, they know it needs to feel like it was your idea. Is it your idea to really push your doctor for a Viagra prescription, even though you are an otherwise healthy, 24 year old male? You shouldn’t but those symptoms come to mind…

The possibilities for political advertising and opinion shaping are staggering. The LLM expected to give neutral answers is sponsored to return “right leaning” or “left leaning” answers for months before an election. Or it embeds language also used by framers of important electoral issues, to prime you for other messaging. Unlike the one-shot advertising in a search result, or the obvious ad on the page you ignore, the LLM is now doing your thinking for you. There will be people who will take the mental stairs because they know the LLM dulls their wits. But these will be fewer and fewer as LLMs get better and more common. With no evidence that on line advertisers find any customer objectionable, could Nick Fuentes be paying to inject your responses with pro-fascist content?

It will be impossible for you to determine what ideas are a product of your reason and research. You will still feel like you’re in control. You will still have your mind. But what goes through your mind will be even more carefully and accurately shaped. In a state were a few thousand votes can sway an election, how much would a campaign pay to advertise to specific voters, if they start seeing those voters adopt talking points and slogans from their LLM chats and social media posts? Would it be $500 per voter? Maybe you need to target 50,000 voters at a total cost of $25,000,000? That actually seems affordable, given the vast sums that are spent on some close elections. The free chat bot loses money. The “premium” plan at $20 per month loses money. Even the $200 a month plan loses money. But the advertising may be their pay-day. How much would you pay to get people to think the way you want them to think, each person believing this was the natural evolution of their own thinking. Casually using LLMs is essentially opening your mind to think other peoples’ thoughts.