"I won't lie to you – there are real dangers," said Anthropic CEO and co-founder Dario Amodei about the development of artificial intelligence. "And I think for too long the industry lied to people about the fact that this technology had risks."

And the biggest risk could be the end of humanity, former Anthropic researcher Jacob Coxon warned on social media this past Tuesday, saying Anthropic and OpenAI are "gambling with our lives" with the development of self-improving superintelligence.

"It doesn't look that different from, say, 'Terminator' or from science fiction films," Coxon told CBS News Thursday. "It will be smart enough to kill us."

In response, Amodei told "Sunday Morning" yesterday, at Anthropic's headquarters in San Francisco, "I don't think I fully just appreciated what it would actually be like when the progress was as fast as it was."

He said the development of AI has been on an exponential curve, which is starting to get steeper. Asked what that means, Amodei said, "It doesn't mean we need to panic today. It doesn't mean we need to shut it all down. But I would say it's a warning sign. It's a warning sign that we need to slow down."

In an essay published Saturday, Amodei wrote that the entire tech industry must work together to "slow the pace" of AI capabilities.

Asked if Anthropic is going to stop releasing more advanced models, Amodei replied, "It doesn't mean that. What it means is that we need to make sure that every generation of models that we release is properly tested."

To attempt that, Amodei says Anthropic will now give independent evaluators permanent "employee-like" access to their models. "Whenever anyone builds an AI model, there's a third-party evaluator who can verify whether that company is following the safety practices that they're committed to – it's like a food inspector," he said.

But AI experts say this is hardly enough, and Amodei agrees, claiming he supports federal regulation. So far, proposed legislation ranges from an outright ban on superintelligence, to a mandatory "kill switch" that could shut down risky AI.

"The AI kill switch could be a good idea," Amodei said. "But I don't think banning it, completely banning it, is the right approach. One, the idea that we would never get the incredible benefits, never get the medical cures, that doesn't sit right with me. Second, this technology will be built in other countries, including authoritarian countries."

Asked if the U.S. should slow down AI development if China would not, Amodei replied, "This is the toughest dilemma. This is the toughest aspect of our situation."

He looks to the nuclear arms agreements with the former Soviet Union as a potential model, but cautions even his expectations are low. "The more long-term thing would be working together to put a speed limit on the rate of AI progress," he said. "I think that's going to be very difficult, because the incentives to pull ahead in the military advantage that you get from that are so large. And honestly, I don't know if it's possible, but we should try."

With the fate of humanity seemingly at risk, how much personal responsibility does Amodei feel about where we are going? "My view here is, it has always been very strange that this technology is being built by a private company," he said. "People ask me that question all the time: Why isn't this being built by government? And the strangest thing about it is, I agree with them. I'm uncomfortable."

Asked if he is willing to give up the technology to the government, Amodei responded, "To the right combination of governments. The worry I have – I am concerned that one single government could abuse this technology just as easily as a single company could. But I think a combination of democratically-elected governments, I don't know about handing over, but some kind of oversight, some kind of joint governance."

The question is no longer whether AI will become powerful enough to require constraint, but who in power will take action.

"If we build in the right way, I think the probability of something bad happening is very low," he said. "If we build in the wrong way, the probability of something bad happening is very high."

Source