Modern technology is fragile

Modern technology is mostly computer-based, and so is only a keystroke away from being unusable.

Machines in the past were built to perform particular tasks, and those tasks only. The next stage was incorporating adjustments that could modify their operations. Thereafter came sensors that could control the operation as a result of external events. While these could become quite complex, such as for a watch, their operations and interactions were well-defined for their whole life. While later machines included electrical actuation and sensors, or even simple electronic ones, these were enhancements, but did not change the machines' basic functionality.

With the development of programmable logic circuitry, and particularly with the invention of integrated circuits, machines were made programmable, allowing their behaviour to be modified after they were made. They were controlled by programs, firstly set by switches, progressively abstracting the language to be less concerned about the logistics of the various machine-level interactions towards controlling the status and data of the machine and its environment.

Despite some attempts at alternatives, most machines are programmed with English-based languages in specially-formatted text files. The formatting embeds control structures like loops and decisions. While modern development environments provide picklists of valid options, the text is usually still editable at any point. This means that any control structure or value could be damaged by an inadvertent keystroke. While the processor that turns the text into usable code can pick up incorrect formatting and some logical errors, some unwanted, but syntactically-valid program errors may get through.

The results can be anywhere from obscure errors that show up at seemingly spurious times to complete failure to operate. Much has been done to ensure that value-integrity is maintained, but control structures are still fragile, and since they and their control conditions principally control the functionality of the machine or what happens on it, its stability is largely up to how astute the programmers were. With all the interactivity modern programs undergo, especially between routines written be different people, most of what we rely upon is only one character away from inoperability.

AI is very complex, and while its actual operation was understood with its earlier incarnations, modern AI is rather opaque as to how it comes up with any results it does. Normal programs can have their operations traced, which is essential for debugging, but modern AI is far too complex for that. We basically do not know how they get any particular result, making it almost impossible to debug them, let alone determine if their logic is correct enough to rely upon what they produce without independently checking it with measured reality or alternate, but reliable, method of computation.

With AI, we are really flying upon trust that we understood the basics and that they will continue to do what we expect. So far, their output has been inconsistent, but that is mainly due to the variability of the data they are trained upon, and what they come across since. We do not really know whether what they tell us or do now is what we can reasonably expect in future. They are becoming as unreliable as us in that regard. At least with some, we can ask them to explain why they produced the results they did, but what they tell us could be just as unreliable as some of their results.