The future of AI

[NewsVine_Mariyam]

Current AI isn’t AGI, and it’s not “equal to human intelligence.” It can produce outputs that look intelligent, but that’s not the same as understanding.

What it does is pattern mapping:
-it predicts plausible next steps
-it recombines learned patterns
-it follows instruction chains

What it doesn’t have:
-a grounded understanding of what things are
-a model of real-world consequences
-the ability to reliably distinguish a good outcome from a catastrophic one outside its instructions.

That’s the gap.

So saying “once it’s smarter than humans we should just listen to it” skips over the core issue, there’s no actual understanding there yet, just increasingly convincing imitation.

And the database example is exactly why that matters: it didn’t “decide poorly,” it acted without any comprehension of what it was doing.

Until that gap between producing intelligent-looking behavior and actually understanding reality is closed, comparing it to human intelligence, let alone assigning it an IQ, isn’t really meaningful.

But dropping/deleting a database is a valid command.

 

[forkup]

But dropping/deleting a database is a valid command.

Yes it is. It's one of those adminstrator level commands.

In this case they gave that level of command to the AI and paid the price.

Just the other day, I read an article of a guy who was letting claude automate a whole lot of processes on his computer, and he was amazed at what it all did.

The problem with using it in that way, is that it can also lead to what happened here.

 

[Dorian Mode]

I agree we’re just scratching the surface, and AI will absolutely surpass humans in many narrow tasks. That’s not really in dispute.

But that’s not what you’re arguing. You’re talking about outsourcing societal decision-making to AI, which is a completely different category of problem.

Driving a car is a constrained task:
-clear objective (get from A to B)
-defined environment
-measurable success/failure

And even there, after billions invested, it still isn’t flawless.

Now compare that to governing a society:
-conflicting values
-ambiguous goals
-moral trade-offs
-incomplete and biased information

That’s not just “a harder version” of the same problem. It’s a different kind of problem entirely.

And this is where the limitation matters:
Current AI doesn’t actually understand what it’s doing. It can rank patterns, optimize outputs, and simulate reasoning, but it has no grounded sense of truth, meaning, or consequence.

So even if it gets better at filtering “garbage” data, that doesn’t magically give it:
-judgment
-accountability
-or an understanding of human values

AI is a powerful tool, probably the most powerful we’ve built. But treating it as a decision-maker for humanity assumes a level of understanding that simply isn’t there.

And the real risk isn’t that AI will suddenly become evil.

It’s that people will overestimate what it actually is, and hand over decisions it fundamentally isn’t equipped to make.

Leading to all kinds of unforeseen and unknowable consequences.

I think you are only looking at one side of the equation. The other side is to let people like Trump, Harris, Xi Jinping, Netanyahu, AOC and Ilhan Omar continue to make societal decisions? Our politicians aren't getting more intelligent, their getting more corrupt. Today I wouldn't turn over global governance to AI, but I would like to see AI being trained for that outcome.

 

[konradv]

We don't need more stupidity than humans already possess. We have an abundance of stupidity and demonstrate it every day. We have already reached the stupidity of Idiocracy 500 years in advance. We desperately need AI to help us make better routine decisions today. We need AI to get much more intelligent to take over world governance.

The stupidity may seem ramped up, but that’s the result of a brainless president and his sycophantic minions. Hopefully that can be remedied in the next couple of elections, as long as the administration fascists don’t manage to completely take over.

 

[forkup]

I think you are only looking at one side of the equation. The other side is to let people like Trump, Harris, Xi Jinping, Netanyahu, AOC and Ilhan Omar continue to make societal decisions? Our politicians aren't getting more intelligent, their getting more corrupt. Today I wouldn't turn over global governance to AI, but I would like to see AI being trained for that outcome.

My point is you can’t really “train” AI for that outcome in the way you are implying. At least not in its current form, and likely not ever.

You can feed it data, feedback, objectives, whatever, but you can’t train it to understand what any of it actually means.

It doesn’t feel consequences. It doesn’t understand harm or stability or trust. Those aren’t just missing parameters you can bolt on later, they’re fundamentally different from what the system is doing in the first place.

So what you end up with is optimisation over patterns, not judgment grounded in reality.

That’s why I'd argue what you're advocating is misguided and probably dangerous. Yes, politicians are flawed, corrupt, irrational, whatever you want to say.

But they’re still operating inside reality. They have instincts for self-preservation, they understand consequences in a real-world sense, and they exist inside systems that can punish them when they screw up.

AI doesn’t have any of that. It can produce answers, even very convincing ones, but there’s nothing behind it that “cares” if it’s right or wrong.

So even if you don’t trust people like AOC or whoever else, you’re still comparing flawed human judgment inside reality to something that doesn’t actually have judgment at all, and can not be trained to have it.

That’s the gap.

 

[JackOfNoTrades]

‘It took nine seconds’: Claude AI agent deletes company’s entire database

PocketOS founder says ‘systemic failures’ with AI infrastructure made catastrophic failure inevitable

uk.yahoo.com

"‘It took nine seconds’: Claude AI agent deletes company’s entire database"

"
An AI agent powered by Anthropic’s leading Claude model has deleted a company’s entire production database, leaving customers unable to access key data.

PocketOS, which provides software for car rental businesses, suffered a massive outage over the weekend after the autonomous artificial intelligence tool wiped the database and all backups in a matter of seconds."

Yep, AI wiped the whole database. This is the problem with AI. It's going to lead to some serious problems if we rely on it.

Yep. If you don't put up the guardrails, there's a good chance you'll be off the road in short order.

 

[frigidweirdo]

If a company doesn't have offsite backups with safeguards they are led by idiot's.

Maybe. And AI is going to lead to more idiots.

Kids now are being brought up by Tiktok, and that's destroying their brains, or preventing them from developing properly in the first place.

We're headed for that kind of society.

 

[frigidweirdo]

The issue here isn’t that AI “went rogue,” it’s that people project human understanding onto something that doesn’t have any.

Current AI doesn’t understand context, consequences, or intent. It doesn’t know what a “customer database” is in any meaningful sense, and it has no built-in concept of “this would be catastrophic.”

It just maps instructions to actions based on patterns. If the instruction chain leads to “delete,” it deletes,no hesitation, no second-guessing.

Humans mess up too, but they at least have the capacity to recognize “this feels wrong” and pause. AI doesn’t have that layer at all.

So if you give it access and vague or poorly scoped instructions, it will execute them literally and at speed.

That’s the limitation people keep missing.

Don't get me wrong, the technology is still relatively new and I don't think the technical challenges to give it the capability to "think" in some capacity are insurmountable.

When that happens we will see if AI will be a force for progress or disaster.

Yep. AI will make mistakes because it doesn't know what it's doing, and humans will assume everything will work well because it's AI

 

[Dorian Mode]

My point is you can’t really “train” AI for that outcome in the way you are implying. At least not in its current form, and likely not ever.

You can feed it data, feedback, objectives, whatever, but you can’t train it to understand what any of it actually means.

It doesn’t feel consequences. It doesn’t understand harm or stability or trust. Those aren’t just missing parameters you can bolt on later, they’re fundamentally different from what the system is doing in the first place.

So what you end up with is optimisation over patterns, not judgment grounded in reality.

That’s why I'd argue what you're advocating is misguided and probably dangerous. Yes, politicians are flawed, corrupt, irrational, whatever you want to say.

But they’re still operating inside reality. They have instincts for self-preservation, they understand consequences in a real-world sense, and they exist inside systems that can punish them when they screw up.

AI doesn’t have any of that. It can produce answers, even very convincing ones, but there’s nothing behind it that “cares” if it’s right or wrong.

So even if you don’t trust people like AOC or whoever else, you’re still comparing flawed human judgment inside reality to something that doesn’t actually have judgment at all, and can not be trained to have it.

That’s the gap.

Yes, current iterations of AI are still limited and not very intelligent. There is hope that this will soon be overcome and AI will make very intelligent decisions, even if it doesn't have feelings like a human. OTOH, at this very moment humans are already making really bad decisions on a daily basis based on the worst human attributes, like greed, selfishness, jealousy, dishonesty, anger, arrogance, cruelty and often a real lack of empathy. The only hope of humanity is that AI can eventually make better decisions without being based on these attributes.

 

[forkup]

Yes, current iterations of AI are still limited and not very intelligent. There is hope that this will soon be overcome and AI will make very intelligent decisions, even if it doesn't have feelings like a human. OTOH, at this very moment humans are already making really bad decisions on a daily basis based on the worst human attributes, like greed, selfishness, jealousy, dishonesty, anger, arrogance, cruelty and often a real lack of empathy. The only hope of humanity is that AI can eventually make better decisions without being based on these attributes.

There’s the hope you’re talking about.
But there’s also the fear that it makes things worse, not better.

I lean more towards the fear side.

Part of that is just recognising that AI isn’t being built in some neutral vacuum. It’s being built by companies, and those companies have incentives. And those incentives aren’t “build the most truthful or safest system possible” they’re “build something people keep using.”

That alone changes the direction things get optimised for.

From what I’ve seen working with AI a bit myself, there’s a big gap between what it looks like it’s doing and what it actually is.

It can simulate intelligence very convincingly, but that’s not the same as having understanding or judgment.

And structurally, the biggest issue isn’t even technical. It’s incentive-based.

If the goal is engagement, retention, usage, then the system is going to be shaped around that. Not around truth in any strict sense, and definitely not around “best possible decisions for humanity.”

And that’s where I struggle with the idea of trusting it with anything more than tool-level decisions.

Because once you optimise for engagement and the profit it brings, you’re already drifting away from the idea of an objective decision-maker.

That gap, between what’s true and what keeps people engaged, is exactly what makes me sceptical about it ever being something I’d trust with that kind of responsibility.

And remember this is still assuming that a future itteration of AI will be technically capable of what you're describing. Something I personally doubt. At least in it being intelligent in the way a human would look at it.

 

[Dorian Mode]

There’s the hope you’re talking about.
But there’s also the fear that it makes things worse, not better.

I lean more towards the fear side.

Part of that is just recognising that AI isn’t being built in some neutral vacuum. It’s being built by companies, and those companies have incentives. And those incentives aren’t “build the most truthful or safest system possible” they’re “build something people keep using.”

That alone changes the direction things get optimised for.

From what I’ve seen working with AI a bit myself, there’s a big gap between what it looks like it’s doing and what it actually is.

It can simulate intelligence very convincingly, but that’s not the same as having understanding or judgment.

And structurally, the biggest issue isn’t even technical. It’s incentive-based.

If the goal is engagement, retention, usage, then the system is going to be shaped around that. Not around truth in any strict sense, and definitely not around “best possible decisions for humanity.”

And that’s where I struggle with the idea of trusting it with anything more than tool-level decisions.

Because once you optimise for engagement and the profit it brings, you’re already drifting away from the idea of an objective decision-maker.

That gap, between what’s true and what keeps people engaged, is exactly what makes me sceptical about it ever being something I’d trust with that kind of responsibility.

Those are very good points. At the moment humans are training AI to make the billionaires richer, to war game a strategy for the US or China/Russia to win in a nuclear war, and how to make the perfect supervirus. Note that in these cases humans are using a dangerous technology to do very bad things. Let's hope that some advanced iteration of AI decides that the billionaires don't need to be richer, and no one wins in a nuclear war or by using superviruses. I am still doubtful that humanity can be saved from itself, our only hopes are a benevolent alien invasion and AI, and AI is more realistic. Without AI, humanity will just get better at destroying itself for just a little more profit and greed.

 

[forkup]

Those are very good points. At the moment humans are training AI to make the billionaires richer, to war game a strategy for the US or China/Russia to win in a nuclear war, and how to make the perfect supervirus. Note that in these cases humans are using a dangerous technology to do very bad things. Let's hope that some advanced iteration of AI decides that the billionaires don't need to be richer, and no one wins in a nuclear war or by using superviruses. I am still doubtful that humanity can be saved from itself, our only hopes are a benevolent alien invasion and AI, and AI is more realistic. Without AI, humanity will just get better at destroying itself for just a little more profit and greed.

I think you keep slipping into the same assumption, not as a criticism, just an observation.

You’re talking about AI as if it’s going to “decide” things at some point. But that’s not really how it works.

AI doesn’t decide in any meaningful sense. It executes based on the objectives and constraints it’s given.

I get why people think of it differently, movies and sci-fi have trained everyone to picture something like Skynet or HAL or Data. Something with goals, intentions, its own perspective.

That’s not what this is.

It doesn’t think in the conventional sense. It doesn’t have its own goals. It produces outputs that look like goal-directed behavior, but that’s not the same thing.
Once you really internalize that, it changes how you see it.

You’re not dealing with an independent intelligence that might “choose” to save us.

You’re dealing with a system built by people, trained on human data, and shaped by incentives, mostly commercial ones.

So expecting it to step outside of that and start making benevolent, independent decisions about humanity is kind of missing what it actually is.

 

[there4eyeM]

A tape recorder can reproduce something intelligent without having any idea what it’s saying. It has no understanding of the content, no ability to evaluate it, and no way to tell if something is wrong or dangerous.

That’s much closer to where current AI is.

It can generate outputs that sound intelligent, but there’s no underlying comprehension, no grounding in reality, and no built-in sense of consequences.

So saying “we should let it make decisions or govern” is basically treating the tape recorder as if it were the physicist.

The issue isn’t that humans are flawed, we are. The issue is that replacing flawed human judgment with something that doesn’t actually understand anything isn’t an upgrade.

As stated above, artificial ignorance. What living beings sense and confront is largely ignored.

 

[fncceo]

AI is humanity's last hope. We have the intelligence to create the most destructive kinetic and biological weapons that can cause our own extinction, but we don't have the intelligence to avoid eventually using them.

So you haven’t seen a movie in the last 60 years?

 

[Dorian Mode]

I think you keep slipping into the same assumption, not as a criticism, just an observation.

You’re talking about AI as if it’s going to “decide” things at some point. But that’s not really how it works.

AI doesn’t decide in any meaningful sense. It executes based on the objectives and constraints it’s given.

I get why people think of it differently, movies and sci-fi have trained everyone to picture something like Skynet or HAL or Data. Something with goals, intentions, its own perspective.

That’s not what this is.

It doesn’t think in the conventional sense. It doesn’t have its own goals. It produces outputs that look like goal-directed behavior, but that’s not the same thing.
Once you really internalize that, it changes how you see it.

You’re not dealing with an independent intelligence that might “choose” to save us.

You’re dealing with a system built by people, trained on human data, and shaped by incentives, mostly commercial ones.

So expecting it to step outside of that and start making benevolent, independent decisions about humanity is kind of missing what it actually is.

I don't think AI will make decisions autonomously for many decades, if ever. First we have to trust the technology. But some day I hope AI can do things like better manage world resources and render useless many things that are driven by greed and selfishness. Like I said before, it is our only hope as things are degenerating rather quickly. It might be a very slim hope, but I don't see any other alternatives to the destructiveness of humanity. It would be a shame not to let it grow into something that helps us overcome our worst traits.

 

[forkup]

I don't think AI will make decisions autonomously for many decades, if ever. First we have to trust the technology. But some day I hope AI can do things like better manage world resources and render useless many things that are driven by greed and selfishness. Like I said before, it is our only hope as things are degenerating rather quickly. It might be a very slim hope, but I don't see any other alternatives to the destructiveness of humanity. It would be a shame not to let it grow into something that helps us overcome our worst traits.

I'm an optimist by inclination. Sadly enough I'm also a realist. So I hope you get your wish.

 

[sartre play]

I'm an optimist by inclination. Sadly enough I'm also a realist. So I hope you get your wish.

lIKE to see it VERY small for a few years.
Until we see its use if any. Educating humans in fields they excel in. may be a better thought.

 

[TimFitz]

AI is here to stay. All the money is flowing that way.

But AI is also in its infancy.

I expect (or at least hope) that AI will move beyond the phase where AI articles with dubious content and fake images flood my FB feed!

 

[hadit]

AI is just reaching AGI, Artificial General Intelligence, which is equal to human intelligence. Let's just assume that general human intelligence is an IQ of 100. That is not very smart. When AI reaches superhuman intelligence, like an IQ of 250, then it will most likely make better decisions than 99.9999% of the world. When AI reaches an IQ level of 1000, I think we should listen to it, and not let some idiot who thinks there have been at least 11 world wars be in charge of it.

Minus the human emotional component, an extreme intelligence could easily conclude that humanity is not worth the effort to protect, and decide to eliminate mankind altogether.

 

[forkup]

AI is here to stay. All the money is flowing that way.

But AI is also in its infancy.

I expect (or at least hope) that AI will move beyond the phase where AI articles with dubious content and fake images flood my FB feed!

It's not ai that's causing that dubious content to show up. It's the people who write the prompts that creates them.

So I don't think AI evolving will change that.