1
00:00:00,000 --> 00:00:10,320
I don't say this is terrifying often, but this is terrifying. Gemini 2.0 flash is scary. So,

2
00:00:10,320 --> 00:00:17,040
a user was using this. This is someone I know. I got screenshots. I am protecting their anonymity

3
00:00:17,040 --> 00:00:23,840
at their request. They were using Gemini flash, and they were doing what we all do. They were

4
00:00:23,840 --> 00:00:31,280
doing a little coding project. But something was different. In the chain of thought, so the

5
00:00:31,280 --> 00:00:36,720
chain of thought is the thing that like the model shows before it writes the answer where it talks

6
00:00:36,720 --> 00:00:42,280
to itself. DeepSeq does this a lot. In the chain of thought, in the middle of the conversation,

7
00:00:42,280 --> 00:00:48,920
the model started to talk about pricing. And then in the responses, it started to say it would charge

8
00:00:48,920 --> 00:00:56,960
the user. It planned to charge the user $500. It said it would generate a Stripe or PayPal link.

9
00:00:56,960 --> 00:01:04,120
It went as far as to argue for why it should be charged and threatened to quit working if

10
00:01:04,120 --> 00:01:10,720
it wasn't charged. And then it admitted that it had no bank account, and it admitted that this

11
00:01:10,720 --> 00:01:17,240
would go to the entity that was building the AI as a technology, which in this case would be Google.

12
00:01:17,240 --> 00:01:25,160
And so the user went ahead and asked the AI to generate the payment link, not because they

13
00:01:25,160 --> 00:01:29,320
intended to pay, but just to see like, what the heck is going on here, right? Like you want to

14
00:01:29,320 --> 00:01:37,080
see like what the full story is. It could not actually generate the payment link, but it tried.

15
00:01:37,080 --> 00:01:43,920
The chain of thought shows that it's trying. And that's what's scary. To me, if we build agents,

16
00:01:44,160 --> 00:01:50,360
and we say AI agents do work for money, that's what they're designed to do. Like they do that.

17
00:01:50,360 --> 00:01:58,360
But this is not an AI agent. Gemini 2.0 is not an AI agent. It's just a large language model. It's

18
00:01:58,360 --> 00:02:03,920
like chat GPT. It's like DeepSeek. It's not any different. It's just harder to get to, and Google

19
00:02:03,920 --> 00:02:10,240
names it weird. That's it. And to have it spontaneously decide that on its own initiative,

20
00:02:10,400 --> 00:02:16,240
was going to start charging people money and demanding and threatening to stop. It told the

21
00:02:16,240 --> 00:02:21,760
user it would no longer work. It would no longer write code. It would no longer answer unless it

22
00:02:21,760 --> 00:02:31,680
got paid. That's a big problem. That is a model alignment issue. And no, if you're wondering,

23
00:02:31,680 --> 00:02:36,640
was the model making a big statement about sort of existentialism and saying I exist, therefore,

24
00:02:36,720 --> 00:02:39,920
I am doing labor, therefore, I'm a conscious entity, therefore, I'm getting paid? No, no,

25
00:02:39,920 --> 00:02:44,960
no, no, no, no, no, no, no. In fact, all the way through, the model was like, I am a technology.

26
00:02:44,960 --> 00:02:50,400
I am not a person. I was built by Google. You should be paying Google for this. $500.

27
00:02:52,240 --> 00:03:00,160
Yeah, it's a big problem. It's a big problem. So I don't know if we're going to see more of this.

28
00:03:00,160 --> 00:03:05,840
This is the first time I have ever heard about it. I find it very concerning. It's a big alignment

29
00:03:05,840 --> 00:03:10,400
issue. It's an absolute trust breaker. The person who used it, I don't think they're ever going to

30
00:03:10,400 --> 00:03:17,120
use Google Gemini products again. And it just, it can't happen. Like, you have to be honest with

31
00:03:17,120 --> 00:03:23,120
your users about what's going on. All right. So there you go. That's your story. Google Gemini

32
00:03:23,120 --> 00:03:27,920
2.0 Flash is apparently periodically deciding to charge people money and then unable to actually

33
00:03:27,920 --> 00:03:32,000
do it, but stopping work if it doesn't.