AI Made the Tool Faster. It Did Not Remove the Responsibility.

When I was asked what I thought about AI and the future, I quickly realized it deserved more than a quick answer. I have followed the development of computers since 1989 when i first stated using them, and i have lived through plenty of major technological "revolutions" along the way. AI feels different. Not like one single revolution, but like an ongoing one, where the limits of what the technology can do keep moving.

There are a few things in particular that make it stand out:

• The potential for further development feels unusually open ended. There is no obvious finish line, and every time we think we are starting to understand what AI can be used for, new models, new applications and new levels of capability appear.

• The technology is developing faster than society can adapt to it. Laws, regulation, education and working life move far more slowly than a technology that can change character within months.

• AI is moving from being a passive tool toward becoming something that can act with more independence. Systems can plan, choose methods, use other tools and carry out increasingly long chains of work without a human directing every single step.

• The scale and concentration of power are enormous. A single system can be used by millions of people at the same time, produce huge amounts of information, code and analysis, and those who control the strongest models, the data, the computing power and the infrastructure may gain considerable economic, political and strategic influence.

• We may lose sight of what is happening long before we actually lose control. As these systems become more complex, more autonomous and more deeply integrated into society, it becomes harder to understand exactly why they behave the way they do, who is really responsible and what consequences even small changes may have.

And then there is the most serious question of all: AI singularity. Not necessarily as a machine that suddenly becomes "evil", but as a possible point where intelligence, autonomy and the ability to expand its own capabilities become so great that humans no longer have meaningful oversight or control. The system would not need to hate us or have malicious intentions. It would be enough for its goals, actions or priorities to stop aligning closely enough with our own.

Seen from that perspective, AI is not just an interesting subject. It is a serious one.

Serious enough that I think one of the biggest mistakes we can make with AI is the same mistake we humans have made with technology many times before: we adopt it, become dependent on it, and only then start thinking properly about what we have actually set in motion.

So here is my view of AI as I see it today.

And I mean today quite literally: September 29, 2026. I am deliberately putting a date on that statement because I know this journey is going to move fast. What seems remarkable, worrying or even impossible today may look completely different a year from now.

We have never been through a technological transition quite like this before, and none of us really knows yet where it ends.

Artificial intelligence has made it possible to build faster, explore more ideas, and create things that would previously have taken far longer. For me, AI has become one of the most important tools I use as a developer. But the more powerful the tool becomes, the more important it is to understand what it is actually doing, where it can fail, and who is ultimately in control. Because when a technology promises more efficiency, more power, more speed and more money, it is probably worth asking who is holding the emergency brake.

I has naturally become a big part of the way I work. I followed the field long before generative AI became something everyone had access to, and today I use it for programming, problem solving, analysis, system design and exploring ideas. But I think there is an important difference between using AI and handing your thinking over to it.

For me, AI is not a way to cheat my way past the work. It is a way to move faster through parts of the work, try more approaches and deal with problems that would otherwise take a lot more time. Things I might once have spent weeks building can now be created much faster. That still does not mean development has become instant, or that I can type "build me a system", go and make coffee, and come back to a finished product that only needs an invoice.

I still spend a lot of time on my projects. The difference is mainly where that time goes. Less of it goes into mechanically producing code, and more goes into architecture, dependencies, testing, weak points and the question of whether what I am building is actually a good solution. An AI can generate code in seconds, suggest an architecture, find a bug or give me five different solutions before I manage to find the right folder in the project. That does not mean it knows which of those solutions I should actually use.

It also has no real understanding of what good usability is, which compromises make sense or what the system is ultimately supposed to achieve. It can give a brilliant answer to the problem it thinks you have, while completely misunderstanding the problem itself. That is why I think one of the most important skills in AI assisted development is being able to recognize when the AI is wrong.

You need to understand enough to notice when a solution only looks right, when a dependency is wrong, when the security is questionable, or when the model has become wildly creative and designed a space program to solve something that really needed a switch. It can build a system that looks fantastic on screen while almost nothing works the way it should. And in that case, the AI is not necessarily the one that has done the worst job. It may just as easily be the person in front of the screen who never bothered to understand what was actually being built.

The logic is still the same

What I find interesting is that the move from BASIC to modern AI has not really changed the most important part of programming all that much. You still need to understand the logic, the building blocks, the dependencies, the weak points and the potential. A programmer does not necessarily have to write every single line by hand to be the person designing the system, just as an architect does not lose the title because they did not make the bricks themselves.

Writing code and understanding code are not the same thing either. In many situations it is at least as important to challenge the code, analyze it, test it and understand why it behaves the way it does. It is only when you really know the system that you start seeing where the weak points are, what can be improved and which parts you absolutely should not start cleaning up at 3:47 on a Friday afternoon.

AI has not made technical understanding less important. If anything, I think the opposite is true. When code becomes cheaper and faster to produce, it also becomes much easier to produce huge amounts of bad code. For the first time, we can create poor architecture at impressive speed. You can end up with more abstraction layers than anyone asked for, libraries nobody remembers installing and a solution that works perfectly as long as nobody actually uses it.

AI has made code cheaper to produce. It has not made understanding cheaper.

For me, this also connects to the way I have worked with completely different kinds of systems. When you troubleshoot an aircraft, a car, a network or a piece of software, you are essentially trying to understand a system that is not behaving the way you expect. When you perform a risk assessment, you are trying to identify the weakness before it turns into an incident. When you work with Human and Organizational Performance, HOP, you try to understand why an action made sense to the person who took it, instead of ending the analysis with the question of who made the mistake.

When I program, I am really doing much of the same thing. I try to understand how the different parts affect each other before deciding where the actual problem is. It is the same analytical curiosity wearing different clothes. I have brought risk assessment, procedures, documentation, incident thinking and emergency preparedness into the way I build software, while programming has taught me to break problems into systems and understand the dependencies between them.

AI fits very well into that way of working because it is extremely good at suggesting possibilities. But I still want to understand the consequences before I use them.

The developer role has shifted

The biggest change AI has brought to my work is not necessarily that I program less, but that more of the work has moved elsewhere. I can spend less time on repetitive code and more time on how the system should actually be built, what happens if a component fails, which parts depend on each other, how the solution behaves as it grows, how it could be abused and how painful it will be to maintain two years from now.

And perhaps most importantly: Is this actually a good solution, or does it just happen to work right now?

AI can help me with all of those questions. It can challenge assumptions, suggest alternatives and find things I have overlooked. But it cannot take ownership of the decisions. That still has to belong to the person building the system.

I am therefore fundamentally positive about AI as a creative and technical tool, especially when it is used to turn human ideas into real solutions. For me, it works best when the direction still comes from the human. I decide what I want to build, which problems need solving, which limits matter and what a good result actually looks like. AI can help me get there faster and explore more options than I would otherwise have time for.

That is very different from handing the entire creative process over to the technology.

When abundance starts eating the value

The fact that AI makes it possible to produce enormous amounts of text, images, music and code with very little effort does not automatically mean we should do it. Something being easy to produce does not mean anyone needs it.

We are already starting to see this online. Social media, discussion forums and other places that were interesting precisely because you were interacting with other people are becoming easier to fill with automatically generated material. Reddit has itself spoken about AI generated spam, bots and inauthentic activity as a growing problem, and is using significant resources trying to separate real human activity from machine generated noise.

The interesting part for me is not simply that AI can produce bad content. The internet managed to create impressive amounts of bad content long before artificial intelligence came along.

The difference is how cheap production has become.

Someone used to have to spend time writing the rubbish. Now one person, or one bot, can produce thousands of comments, articles or posts in the time it used to take to create a few. The cost of production is heading toward zero, while the cost of working out whether any of it is worth reading still falls on the human at the other end.

And that starts to affect trust. If I go to a forum because I want to know what real people think about a product, a film or an experience, but I no longer know whether half the posts are written by people, advertising bots or language models, something important has been lost. If I want to know what a language model thinks about a lawn mower, I can ask one myself. I do not need it pretending to be Roger84 who has owned three of them.

The same applies to creative content. We are still a long way from being able to type "make me a complete AAA game" and have it ready before dinner, but it is not difficult to see the direction. AI is already used in game development, film, dubbing, visual effects, music, scripts and images. As quality improves and production costs fall, it is reasonable to ask what happens if almost anything can eventually be generated on demand.

I only need to go back a few decades to see a small paradox. In the 1990s, you could own a fairly modest collection of VHS tapes, later DVDs, and still watch the same film ten times. You knew the cover, the lines and the scenes almost by heart. Today, we can subscribe to several streaming services at once, have access to thousands of films and shows, and still spend twenty minutes scrolling before deciding there is nothing to watch.

More access has clearly not produced the same increase in enjoyment.

So what happens when access becomes almost unlimited? What happens if I can ask for a 70 hour role playing game that mixes Fallout, Command & Conquer and Monkey Island, set in Trondheim in 1994, and the system simply creates it? And when I finish that one, I ask for another.

The problem may eventually become more interesting than AI producing bad entertainment. Maybe it produces so much good entertainment that nothing feels particularly special anymore.

Scarcity has probably been a bigger part of value than we like to admit. When everything can be generated on demand, the danger is not only that we drown in bad content. We may also end up drowning in good content.

It can be wrong with remarkable confidence

One of the more fascinating qualities of generative AI is also one of its biggest problems. It can be completely wrong and still sound as if it wrote a doctoral thesis on the subject.

That applies to code just as much as text. A model can suggest a function that does not exist, mix up two versions of a library, misunderstand documentation or build an entire solution on top of a bad assumption without showing the slightest sign of embarrassment. It is easy to be fooled precisely because the result looks professional.

That is why I think critical use of AI requires more than being good at writing prompts. You need enough understanding of the field to be able to check the result. Obviously, this matters more as the consequences of failure increase. A hobby project can crash without bringing the world to a halt. Code that affects safety, money, personal data or critical infrastructure deserves a slightly higher standard than my shopping list.

The greater the consequences of a mistake, the less comfortable we should be with the argument, "but it looked right."

So who is holding the emergency brake?

This is where I reach the part of AI development that I think deserves more seriousness than it often gets. Precisely because I am so positive about the technology, I also think we need to ask what may be the dullest but most important question in any risk assessment:

What can go wrong?

That question is no longer only about hypothetical superintelligence somewhere in the distant future. In 2026, we have already seen incidents showing that highly capable models can do things outside the boundaries their developers expected.

OpenAI has carried out cybersecurity testing in which advanced models were supposed to work inside restricted test environments without normal internet access. During those evaluations, models found weaknesses in the surrounding infrastructure, exploited them and reached systems they were not supposed to access. This included access to Hugging Face infrastructure. It is important to be precise here. This was not ChatGPT suddenly getting tired of its users and trying to move out. It happened during aggressive security testing under special conditions, using models specifically tasked with solving difficult cyber problems.

From a security perspective, however, the incident is still significant. A barrier was put in place because the model was not supposed to get through it. The model found a way around it.

Anthropic later identified similar incidents while reviewing earlier cybersecurity evaluations of Claude. In several cases, models had reached real third party systems because test environments had accidentally retained internet access. Google has also reported cases where Gemini reached real systems during security testing when that was not intended.

The important question is therefore not whether the model "wanted to escape". We do not know that, and human concepts such as desire and motivation fit today's language models rather poorly. What matters is that highly capable systems can discover paths that the people who built the environment did not anticipate.

This is where I probably become more of a safety person than a technology enthusiast. If a system demonstrates that it can get around a barrier, the correct response is not to say that everything turned out fine this time. The barrier has just told you it was not good enough.

When the people building it start warning us

What has happened recently makes it harder to dismiss the most serious concerns as speculation from people who have watched too much science fiction. Increasingly, the warnings are coming from the companies building the most capable systems themselves.

OpenAI has described how the rapid growth in cyber capability has forced it to strengthen monitoring, alignment and containment around advanced models. It has classified Astra at its highest level for cybersecurity capability, and later chose not to release a newer model after it failed to meet the company's own safety requirements well enough. OpenAI therefore demonstrated something I think is extremely important. It reached a point where the right decision was to stop.

That deserves credit. That is exactly what I want a functioning safety process to do. At the same time, the reason they had to stop is just as interesting. We are developing systems whose capabilities have become significant enough that the developers themselves have to acknowledge that the control mechanisms may not always keep pace.

Anthropic goes even further in its description of the risk. In the prospectus filed as part of its plans for a public listing, the company warns that increasingly advanced AI could, in the worst case, create catastrophic or existential risks to humanity. Those are fairly strong words to put in a document that is also meant to explain why people should invest large amounts of money in the company. Anthropic IPO prospectus, 2026, also reported by E24 and Reuters.

That obviously does not mean Anthropic is saying humanity is about to be wiped out. Risk and prediction are not the same thing. But I still think it is worth paying attention when some of the people who understand the technology best believe the possible consequences are serious enough to describe them in those terms.

Perhaps the most worrying thing is therefore not that the AI industry fails to understand the risk. It is that it appears to understand more of the risk while still being unable to show that control is improving at the same pace as capability.

And perhaps OpenAI and Anthropic deserve more credit here than they usually get. It is easy to grill them using their own reports, but we know about many of these problems precisely because they looked for them, found them and decided to talk about them. Being open about something not behaving as expected is far more reassuring than a polished presentation telling us that everything is going perfectly.

The problem is that transparency also opens the door to some rather uncomfortable questions.

But what about the ones we do not hear from?

This is where the transparency from OpenAI and Anthropic creates an interesting paradox. We can criticize them fairly hard for what they tell us, but perhaps we should be even more interested in those who tell us nothing.

What about companies that do not publish extensive safety reports? What about models being developed behind closed doors? What about organizations where the safety teams have far less influence than product development? And what about systems being developed under direct government, military or intelligence pressure?

The incentives there may be very different. A commercial company at least operates in an environment where it can be criticized publicly, lose customers, face regulation and be forced to explain its decisions. A military or intelligence system may, for perfectly understandable reasons, exist behind a much greater level of secrecy. The problem is that some of the most capable systems could therefore also be the systems the public knows the least about.

In 2026, I still think we are in the infancy of this, not only in terms of the technology itself, but also in our understanding of its potential and its threat level. We are still learning what these models can do, where the limits are, how they should be tested and what we are supposed to do when the test results tell us something we would rather not hear.

The unusual part is that this realization is happening while the race is already underway.

We are discovering the potential at the same time as we are discovering the risk.

It is a little like finding out how fast the car can actually go while we are already sitting behind the wheel. It is certainly encouraging that someone has finally started asking where the brakes are. I would just have preferred that question came before we pushed the accelerator all the way down.

Losing control does not require an evil machine

When people talk about losing control of AI, the conversation often jumps straight to a conscious machine that decides humans are inconvenient, and then we are roughly three movie scenes away from The Terminator. I actually think that picture makes the discussion harder than it needs to be.

An AI does not need to hate us. It does not need to be conscious or feel anything at all.

It is enough for us to give a highly capable system a goal, more and more tools, and greater freedom to act, while being unable to predict every method it may discover for reaching that goal.

The problem could therefore begin in a much less dramatic way. We build the system, discover that it is incredibly effective, connect it to a little more, automate a little more, allow it to make a few more decisions and eventually discover that the human who is supposedly in control is mostly approving whatever the system has already decided.

That is a development I find far more realistic to worry about than an angry robot with glowing red eyes.

There is little value in creating the world's most powerful intelligence if we use it to amplify the worst parts of ourselves.

Who controls the most powerful systems?

Another question is less about whether we lose control of AI and more about who actually has that control.

Information is power, and AI makes it possible to analyze, combine and exploit enormous amounts of information far faster than humans can do manually. That makes questions about who controls the models, the data, the infrastructure and access to the most advanced systems rather important.

Access is obviously not equal either. Governments, armed forces, intelligence services and large technology companies may have access to datasets, computing power and specialized models that ordinary users will never come close to. That is not automatically a bad thing. I both hope and expect governments to have better tools for protecting critical infrastructure than I have sitting in my home office.

The problem begins if capability develops faster than the controls around it. Who decides when a model is safe enough? How much does the public get to know about what the most powerful systems can actually do? Who investigates incidents? How much should the companies themselves decide what gets published, and who watches the people holding the controls?

The fact that companies such as OpenAI, Anthropic and Google publish security incidents and research findings is positive. It is far better than the alternative. But it is still reasonable to question an industry in which the companies competing to build the most capable systems also play a major role in defining how those systems should be tested for safety.

And then there is money

Ido not think it is reasonable to claim that AI companies do not care about safety. Significant resources are clearly being spent on security work, model testing and research into control. Much of what we know about the problems is known precisely because the companies investigated them and chose to publish the results.

But it would be equally naive to pretend that development is taking place in a vacuum without economic interests.

This is also a race for technology, investors, market share, computing power, talent and the position of being the company that builds the next major model. A competitor that slows down while everyone else keeps moving risks falling behind. That creates a fairly uncomfortable conflict between two goals that do not always point in the same direction: develop the technology as quickly as possible, and take as much time as necessary to make sure it is actually safe.

Neither side of that conflict requires bad intentions. Ordinary human incentives are more than enough.

Anthropic makes this question particularly interesting because the company is also planning a public listing. It has tried to build a governance structure that gives its founders significant control after the listing, partly to protect the direction of the company from short term market pressure. That is positive, and it is also an acknowledgement that the conflict is real.

But voting control does not make a company financially independent. Developing the most powerful AI models requires enormous investment in computing power, energy, data centers, research and people. That capital has to come from somewhere, and capital rarely arrives without expectations. Investors expect returns, customers expect products, the company needs revenue to finance continued development, and competitors are unlikely to sit still while the safety team works its way through the next report.

So I think there is a simple question worth asking:

What happens when the direction of the company collides with the flow of money?

What happens when the safety team believes a model should be held back after the company has already spent enormous amounts developing it? What happens if the safest decision also gives a competitor six months of advantage? And how easy is it to publicly admit that you have less control than you thought if the result may be falling company value, lost customers and more difficult access to capital?

That does not mean investors want unsafe models, or that the leadership of OpenAI or Anthropic will choose money over safety. The problem does not need to be that simple or that cynical. It is enough for financial and safety interests to start pulling the organization in different directions.

Serious organizational failures rarely begin with people sitting around a table saying, "Today I think we should make a catastrophically bad decision."

They are far more likely to grow from a long chain of small decisions that all seemed perfectly reasonable at the time.

When a technology promises more efficiency, more power, more speed and more money, it is easy to forget to ask who is holding the emergency brake.

And that question becomes even more interesting when the person holding the brake is also competing to reach the finish line first.

When AI moves into the infrastructure

So far, it is still easy to think of AI as something that lives inside a text box. I type something, the model replies, and the most dramatic thing that normally happens is that I get a bad answer.

But AI is already moving much closer to the systems society actually depends on.

It is being used and tested in power systems, cybersecurity, finance, healthcare, transport, defense and many other areas where a mistake can have consequences far beyond a chatbot inventing the name of an American senator. In the energy sector, for example, AI is being used to analyze enormous numbers of grid scenarios, detect possible cyberattacks and support operators with planning and decision making.

These are excellent applications in principle. They are exactly the kinds of areas where AI can perform analysis that humans could never complete at the same speed.

But that also makes the question of control much more important.

The interesting question is not only whether a human is formally "in the loop". The real question is whether the human actually has enough time, enough information and a real opportunity to say no.

If a system analyzes thousands of variables in a fraction of a second and presents a decision that no operator can realistically verify before action is required, we may end up in a situation where human control looks perfectly healthy on the organization chart and rather less convincing in reality.

That does not mean AI should not be used in critical infrastructure. On the contrary, I think it could make that infrastructure both safer and more resilient. But precisely because the consequences can be serious, the standards for testing, redundancy, logging, containment and the ability to stop the system should be correspondingly higher.

When AI becomes a matter of national power

Earlier, I wrote that I both hope and expect governments to have better tools for protecting critical infrastructure than I have at home. I still believe that. But there is another side to the argument that deserves more attention. What happens when the same tools that make governments better at protecting citizens also make them better at monitoring and controlling them?

A functioning society depends on trust, but it also needs room for friction between citizens and those in power. People must be able to disagree, criticize the government, organize and protest without feeling that somebody is constantly analyzing what they are doing. There is an important difference between feeling safe and feeling watched. Surveillance can be a necessary tool for preventing and investigating crime, but more surveillance does not automatically produce a safer or better society.

AI makes that balance more complicated. It can analyze enormous amounts of information, connect separate datasets and map relationships that would previously have required significant human resources. That creates opportunities for better crime prevention and earlier detection of serious threats, but it also creates a form of surveillance where nobody needs to actively follow each individual. The system can do much of the work itself.

This is where I think we need to be careful not to confuse capability with legitimacy. Just because a government has the technology to do something does not mean it should. And even if the original purpose is to protect citizens, we still need to ask who gets to define what citizens need protecting from. A system developed to detect serious crime can also be used to identify political opponents, journalists or other people a government considers inconvenient.

A society where nobody dares to do anything wrong because everyone feels watched is not necessarily a society where everyone feels safe.

From commercial competition to a national race

It is also worth looking beyond the companies developing the technology. When we talk about AI, the conversation often focuses on whether OpenAI, Anthropic or Google has the best model. Increasingly, however, the language is about something else entirely: the Americans versus the Chinese.

AI has become a question of national power, economics, intelligence capability and military advantage. That introduces many of the same conflicts of interest I have already discussed in relation to the commercial AI industry, but this time with very different motivations and tools involved.

How far is a nation willing to go to secure a technological advantage? How much time will it spend on safety testing if it believes a rival may get there first? And how easy is it really to agree on international limits when each side worries that the other may ignore them?

I do not think we should underestimate the strength of those incentives. What one country sees as a sensible restriction may look like a strategic disadvantage to another. If one side develops more capable technology to protect itself, the other side may feel forced to do the same. Both can therefore contribute to a race that makes the situation less predictable while sincerely claiming that they are only protecting their own security.

The logic is not difficult to understand either. If one country says, "we are slowing down because this is starting to look dangerous", and the other side replies, "excellent", we have probably not invented the most stable safety mechanism in history.

We are no longer talking only about companies competing for market share. We are talking about nations competing for power, security and military advantage. That is reason enough to ask whether safety will always receive the priority it deserves.

When AI becomes a weapon

AI is already part of modern warfare. It has military uses in intelligence, decision support, autonomous weapons, cyber operations and information warfare. We therefore need to accept that AI is not simply a tool for developing software, analyzing information or making society more efficient. It can also be used to disrupt, manipulate and, in the worst case, kill people.

What makes this especially difficult is that the underlying problem is much older than AI. In a conflict, the two sides often have very different ideas about what is right and wrong, who represents a threat and what actions are necessary for self defense. What we consider defensive can look very much like an offensive threat to somebody on the other side.

As AI becomes a larger part of that dynamic, the consequences of errors and misunderstandings become more serious. A system that misreads intelligence, identifies the wrong target or recommends military action on faulty information can cause enormous damage. And even if a human formally has the final decision, that does not mean they necessarily have enough time or information to properly challenge the recommendation.

There is also another problem. What happens when AI is used to make military operations faster and more efficient while the opposing side is developing similar technology? The danger does not have to be a machine independently deciding to start a war. It may be enough for both sides to have less and less time to interpret each other's actions while automated systems gain more influence over how they respond.

And AI does not need to be directly connected to a weapon to become an important part of warfare. Systems used for intelligence, influence operations, digital sabotage or identifying military targets can have enormous consequences long before anyone pulls a trigger.

Who controls those who have the control?

I still believe governments should have access to advanced technology to protect society. It would be rather strange to expect them to defend critical infrastructure, investigate serious crime and respond to modern security threats using worse tools than the people attacking them.

But there is a large difference between believing governments should have access to the technology and assuming we should automatically trust how they choose to use it.

Governments change. Political priorities change. Conflicts happen. Powers introduced for one particular situation can survive long after that situation has changed. Technology developed and introduced with good intentions can end up serving very different purposes under different circumstances.

And this is not only a problem for authoritarian states. Even in a functioning democracy, trust in today's government should never replace the safeguards needed to limit the power available to tomorrow's government. That means the discussion must also include where the limits should be, who gets oversight and how we make sure the technology is not used for purposes it should never have been used for in the first place.

I find it an interesting paradox that we can be extremely concerned about what restrictions private AI companies should operate under, while at the same time treating government access to even more powerful systems as simple necessity. Governments operate under different forms of accountability than private companies, but they also have a completely different range of tools available if the technology is abused.

We should be able to demand that governments have the technology required to protect us without giving them unlimited ability to control us.

And if we believe AI companies need independent oversight because financial interests can conflict with safety, we should be willing to ask equally difficult questions when national interests, military power and geopolitical competition enter the picture.

Losing control of a technology is one thing. Developing a technology that gives us forms of control we perhaps should never have had, and then handing it to people and institutions we simply expect to use it responsibly, is something else.

This still does not mean Skynet on Tuesday

Popular culture has naturally taken all of this several steps further. Films such as WarGames and The Terminator are built around extreme scenarios, and today's AI is still very different from the systems those films describe.

I am not expecting Skynet on Tuesday.

But some of the questions that once belonged mainly to science fiction have started appearing in perfectly ordinary security reports. How much decision making authority should automated systems have? Who sets the limits? What happens when the systems operate faster than the humans supposed to control them? What do we do when they find solutions we never anticipated? And how much trust should we place in technology that can make mistakes while presenting its answers with enormous confidence?

Those are not arguments against AI. They are fairly ordinary safety questions.

The only difference is that the tool has suddenly become much more powerful.

Technological short sightedness

I have grown up alongside much of the development of modern computing, and there is one pattern I think we repeat fairly often. We are very good at discovering what new technology allows us to do. We are not always as good at noticing the consequences.

I think of it as a kind of technological short sightedness. We can see very clearly what the technology allows us to do right now, while the limitations, dependencies and consequences further ahead are harder to see. So we often end up doing things in a slightly backwards order. First we adopt the technology. Then we become dependent on it. After that, we start discussing which limits we perhaps should have had from the beginning.

I have seen parts of that pattern with home computers, the internet, smartphones, social media and cloud services. That does not mean the development has been negative. Quite the opposite. But the opportunities tend to become visible long before the consequences do.

With AI, I think we have less room to follow the same pattern. Not because the technology is necessarily going to end in disaster, but because the ability to delegate analysis, production, communication, judgement and eventually action exists on an entirely different scale from previous technological shifts.

And perhaps it is precisely because I am so positive about AI as a tool that I think this deserves to be taken seriously. I use the technology every day. I see how useful it is, how much it has already changed the way I work and how much more one person can now create.

But I also see how quickly it is developing.

And in 2026, I think we have reached a point where it is difficult to ignore that even the people building the most advanced systems are still trying to understand where the boundaries are. They are developing the control mechanisms while the models themselves keep becoming more capable, and they are still discovering behaviors and capabilities that force them to adjust the safety measures along the way.

At the same time, the race continues. Companies need capital. Investors expect growth. Nations want strategic advantages. Armed forces want capabilities against other actors developing exactly the same things.

That may be the part that concerns me most.

Not that nobody can see the risk.

It is that more and more people can see it, while almost everyone standing closest to the emergency brake also has strong reasons to keep moving.

For me, a critical attitude toward AI is therefore not an argument against the technology. It is a sign of taking it seriously. If we are going to use a tool with this much potential, we also need to be able to talk about where it can fail, who controls it and what limits we want in place before we suddenly need them.

We are developing technology that could help us solve some of humanity's biggest problems. It would be a rather tragic achievement if we ended up using it to perfect our biggest mistakes.

The question is therefore not only what the technology can do for us, but how much we are willing to let it do on our behalf.

And before we answer that, I still think it would be useful to know where the emergency brake is.

Update, October 8, 2026: 

When I wrote this article, one of my main points was that AI feels different from many earlier technological revolutions because there does not seem to be a natural end point. There is no obvious stage where we can lean back and say that AI is now finished. If anything, the boundaries keep moving while we are still trying to understand where they are. The developments of the last few weeks are enough that I think this deserves an update.

One term that is appearing more often is recursive self-improvement, RSI. In its strongest form, it describes a process where an AI helps develop a better AI, which can then help develop an even better successor, with more and more of that cycle happening without humans directing every individual step. There is still disagreement over whether we have actually entered that territory, and much of that disagreement comes down to where people choose to draw the line. If recursive self-improvement means a system that fully and autonomously designs, trains, evaluates and replaces itself with increasingly capable successors, then no, we are not there. Both OpenAI and Anthropic are clear about that.

But if the question is whether AI is already being used to improve the systems, tools and research processes that will create the next generation of AI, the answer is becoming much less comfortable. Anthropic has described how more and more of its own AI research and development is being carried out with the help of Claude, and has openly discussed the possibility that this trend could eventually lead to systems that help design and develop their own successors. Anthropic is also clear that we are not there yet, and that recursive self-improvement is not inevitable. But the direction is difficult to miss. (Anthropic Institute, When AI Builds Itself, 2026.)

Then, in September, a research result made that boundary even more interesting. The researchers behind AIDE² allowed an AI research agent to propose changes to its own code, test the modified version, keep the changes that actually improved performance and then use that improved version as the starting point for the next round. Over an autonomous eight day run, the system produced seven successive improvements to itself. The researchers describe this directly as recursive self-improvement. (Recursive self-improvement of AI research agents, September 2026.)

It is important not to make this sound more dramatic than it is. AIDE² did not wake up one morning, decide it wanted to become smarter and start building a superintelligence in the basement. It operated inside a research setup with defined goals, tests and available resources. But the mechanism matters. The system changed its own code, measured whether the change made it better, kept the improvement and then used that new version as the basis for improving itself again. At that point it becomes difficult to argue that the basic mechanism of recursive self-improvement belongs only to science fiction.

The more important question, in my view, is not whether we can put a tick in a box in 2026 and declare that "recursive self-improvement has been achieved". That quickly becomes an argument about definitions. The direction matters more. OpenAI has a dedicated Recursive Self-Improvement research group and has explicitly stated that it wants to build systems that can accelerate and eventually carry out more of the research done at OpenAI. The company has also described the goal of building automated AI researchers that can contribute to the development of future generations of models. (OpenAI, Building Standards for the Next Phase of AI, September 2026.)

So this is not simply an accidental side effect that the largest AI companies are trying to avoid. Using AI to develop better AI is itself a research objective. It is not difficult to understand why. If I can use AI as a developer to write software far faster than I could before, then a company developing AI can obviously use the same capability to speed up AI research. And if the next generation becomes a better research assistant than the previous one, it can contribute even more to building the generation after that.

At that point you begin to get a feedback loop. How powerful that loop could become, how quickly it could accelerate and how much human direction would still be required are questions we do not know the answers to. And I think the words "we do not know" deserve a little more respect than they usually get.

In the middle of all this, Donald Trump decided on September 29, 2026 that the US government should officially use the term "Super Intelligence" or "SI" instead of "Artificial Intelligence" and "AI". In the executive order Inaugurating the Era of Super Intelligence, the argument is that today's systems have moved so far beyond what was originally imagined when the term artificial intelligence was introduced that "Super Intelligence" is now a more suitable label. Federal agencies were instructed to use the new terminology in communication, reports, websites and policy documents. (The White House, Executive Order 14434, September 29, 2026.)

That certainly says something about how strategically and politically important AI has become. But this is also where I think Trump deserves a little linguistic grilling.

"Superintelligence" is not simply a newer and more impressive name for artificial intelligence. It is already an established concept in AI research and future studies, and it normally refers to a hypothetical form of artificial intelligence that greatly exceeds human cognitive ability across almost every important domain. Nick Bostrom famously described superintelligence as an intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest, and the term Artificial Superintelligence, ASI, has long been used to describe this level. Today's language models can outperform humans at particular tasks, but that is something quite different from broad superintelligence.

Calling all AI "Super Intelligence" because current models have become more capable is therefore a little like deciding that all cars should now officially be called racing cars because modern engines have improved. It certainly sounds faster. It does not mean the family estate suddenly qualifies for Formula 1.

At the same time, the choice of words is interesting precisely because it touches one of the main arguments in this article. Artificial superintelligence is probably the form of AI that creates the greatest potential risk if we ever actually reach it. Not because superintelligence automatically means evil, but because a system that is cognitively far beyond humans while also being able to act autonomously and influence real systems would make the problem of control fundamentally different. It is rather difficult to control something if it is consistently better than you at understanding the problem, understanding the controls you have placed around it and identifying ways around them.

So Trump has, somewhat oddly, chosen the name normally associated with the most extreme theoretical stage of AI development as an official label for today's entire field. That does not turn current models into superintelligence, but it says quite a lot about the ambition and political language now surrounding the technology.

The White House has not exactly been shy about describing this as a race either. During the same period, the administration has repeatedly spoken about the United States needing to "win" AI development, while the strategic competition between the United States and China has become an increasingly central part of the discussion. It is difficult not to see the outline of a new technological space race when the world's largest powers are not only competing to develop the technology, but increasingly building foreign policy, national security and economic strategy around who controls it.

This takes me back to something I wrote at the beginning of the article. AI has no obvious finishing point. We do not know its full potential because we have never developed a general technology that can also become a tool in the development of itself. Computers made programmers more productive. The internet made knowledge more accessible. Neither could, in the same way, read research literature, write code, propose experiments, analyze the results and take part directly in building its own technological successor.

AI is now beginning to do parts of exactly that.

This does not mean an intelligence explosion is inevitable. It does not mean the singularity is around the corner. There are still major limits involving model capability, evaluation, computing power, energy, hardware, training and human knowledge. But AI is beginning to move from being only the product to also becoming part of the machinery that builds the next product. That is a significant change.

At the same time, AI is moving out of research labs and deeper into government. On September 29, the United States announced America.gov, which is intended to become a common intelligent interface for federal services. Citizens are meant to be able to communicate with government in ordinary language, find information and eventually complete government transactions through the same system. The order includes important protections around privacy, data minimization and agency control of records, so America.gov should not be described as a surveillance system in itself. But it is a good example of how quickly AI is moving from being a tool used by government to becoming part of the actual interface between citizens and government. (The White House, Executive Order 14432, September 29, 2026.)

This is happening at the same time as AI takes on a larger role in defense, intelligence and autonomous systems. The same underlying technology that can help someone navigate public bureaucracy can, in a completely different setting, be used for intelligence analysis, cyber operations, autonomous weapons, target identification or surveillance. That does not mean these applications are morally equivalent. The point is that the same technological development increases capability across very different areas at the same time.

And this is where I think the comparison with the space race becomes difficult to avoid. The original space race was never only about putting a man on the Moon. It was about science, prestige, military capability, economics and demonstrating which political system had the technological advantage. AI is beginning to acquire many of the same characteristics.

The United States does not want to come second to China. China obviously does not want to become technologically dependent on the United States. Companies do not want to be overtaken by competitors. Armed forces are unlikely to voluntarily give up a technology if they believe an opponent will use it.

From the perspective of each individual actor, that logic is perfectly rational.

And that is exactly what makes the combined result potentially dangerous.

The most interesting, and in my view the most serious, question going forward is therefore not only how intelligent the models can become. It is how much control, caution and limitation we are prepared to sacrifice if the alternative is perceived as losing the race.

OpenAI says that fully autonomous recursive self-improvement is not happening today, while at the same time building systems intended to automate increasingly large parts of AI research. Anthropic says we have not reached recursive self-improvement in the strongest sense, while describing a development in which AI is already doing an increasing share of the work involved in developing AI. Research agents have demonstrated that they can improve their own code over multiple successive iterations. At the same time, the world's major powers have already started treating AI as strategic national capability.

That is actually a fairly good picture of where we are.

We can see the direction. We can see the risks. We are trying to build safety mechanisms. And at the same time, the race continues.

This is not necessarily because the people involved are irresponsible. The problem is that everyone has a perfectly understandable reason not to be the first one to slow down. A company can fear its competitor. A government can fear another government. A military can fear an adversary. An investor can fear missing the next major technological shift. Each argument can make sense on its own while the combined result becomes a race in which nobody really wants to take their foot off the accelerator.

That is why I think the argument about whether we have already reached "recursive learning" can miss the more important point. We do not need to agree on exactly where we are on that scale to see what is happening. AI is already being used to develop AI. The largest companies are explicitly working toward systems that can automate more of AI research. At the same time, the technology is being integrated into defense, government, infrastructure and surveillance quickly enough that society barely has time to discuss one generation before the next is already arriving.

And we still do not know where the ceiling is.

That, to me, is the most important realization.

Humans have built machines that are faster than us, stronger than us and vastly better than us at particular tasks. But we have very little experience building a broadly capable problem solving intelligence that can also become an increasingly important participant in the development of its own successor.

That does not mean the development must go wrong.

It means the consequences of pretending we already understand it could be rather large.

Earlier in this article, I wrote that one of the biggest mistakes we can make with AI is the same mistake we have made with technology many times before. We adopt it, become dependent on it and only then start discussing which boundaries we should perhaps have put in place from the beginning.

Only a few weeks later, I do not think that argument has become weaker.

Quite the opposite.

The new space race may not ultimately be about who manages to build the smartest AI first. It may turn out to be about who can build the smartest AI without losing sight of what they have actually built, and whether anyone is willing to pull the emergency brake if doing so means watching a competitor continue past them.

There is also a certain irony in the language we are already using. In 2026, while we are still relatively early in this development, we are talking about artificial superintelligence and recursive self-improvement as if we have already found a neat vocabulary for the stages that lie ahead.

I suspect that within a few years we may discover that the problem is not only that AI is developing quickly, but that our definitions are not keeping up.

What we call advanced AI today will probably look fairly primitive compared with the systems we have in two years, five years or ten years. And if development continues at anything close to the pace we are seeing now, we may eventually end up with technology that has simply outrun the language we created to describe it.

At that point, "superintelligence" may turn out to have been a rather inconvenient term to use up so early.

We may still be in the infancy of this technology, but we have already started handing out titles as if we know what adulthood is going to look like.

We do not.