WEBVTT 00:00:00.155 --> 00:00:05.255 For decades most academia thought the idea he kept working on was a dead end. 00:00:05.275 --> 00:00:08.935 To them he was not just wrong, he was a traitor to the field. 00:00:08.955 --> 00:00:13.335 Eventually, that idea won him the Nobel Prize and became one of the modern 00:00:13.355 --> 00:00:16.403 foundations of AI, Geoffrey Hinton. 00:00:16.567 --> 00:00:20.467 The book this episode comes from is Genius Makers, together with several 00:00:20.487 --> 00:00:23.110 interviews Geoff has given over the years. 00:00:23.367 --> 00:00:27.827 He is born in London after the Second World War and comes from a family of 00:00:27.847 --> 00:00:32.719 doctors, mathematicians, scientists, academia royalty, you could say. 00:00:33.033 --> 00:00:37.413 I found the following quotes on his father's biography, Howard that you need to 00:00:37.433 --> 00:00:41.093 listen. He was unwilling to accept received opinion. 00:00:41.113 --> 00:00:45.733 He enjoyed questioning the dogma, and he had no hesitation in challenging the 00:00:45.753 --> 00:00:49.653 establishment. So where you can see he loved being a contrarian. 00:00:49.673 --> 00:00:54.693 He was certain that he was right, and in controversy, he felt it was his duty to 00:00:54.713 --> 00:00:58.853 gain his point because in doing so he was establishing the truth. 00:00:58.873 --> 00:01:04.351 He wrote with enthusiasm, often in excitement, and worked with great intensity 00:01:04.371 --> 00:01:07.311 for very long hours, night and day. 00:01:07.331 --> 00:01:11.631 During the 50s, he published 51 papers. 00:01:11.651 --> 00:01:16.387 His father is a prolific scientist, and reading this, I can see a lot of Geoff 00:01:16.667 --> 00:01:21.261 in him. The same drive, intense work ethic, and conviction. 00:01:21.281 --> 00:01:25.785 His mother Margaret was a school teacher who started her own school. 00:01:25.805 --> 00:01:30.665 And when Geoff is a kid, his father comes to him and says, Geoff, work really 00:01:30.685 --> 00:01:35.785 hard, and maybe when you are twice as old as me, you will be half as good. 00:01:35.805 --> 00:01:38.665 And Geoff is, I want to prove I will be better. 00:01:38.685 --> 00:01:43.223 And we can see like his father has very high expectations of him. 00:01:43.337 --> 00:01:47.065 He gets obsessed with a question nobody can answer him. 00:01:47.085 --> 00:01:49.305 How does the brain learn? 00:01:49.325 --> 00:01:54.505 Back then and still today, we don't fully understand how we learn, so he goes to 00:01:54.525 --> 00:01:56.265 university to try to figure out. 00:01:56.285 --> 00:01:58.903 And he's trying all different types of degrees. 00:01:58.923 --> 00:02:03.943 He does chemistry, physics, philosophy, psychology, but nothing is 00:02:03.963 --> 00:02:06.983 answering the question. He's completely hopeless. 00:02:07.003 --> 00:02:10.423 So he takes a break and becomes a carpenter. 00:02:10.443 --> 00:02:14.903 During this break, every Saturday he will go to his public library in North 00:02:14.923 --> 00:02:19.223 London and will spend the day filling pages about how the brain must work with 00:02:19.243 --> 00:02:23.296 all his theories, he's reading everything. 00:02:23.316 --> 00:02:26.656 And his method is intuition. 00:02:26.676 --> 00:02:31.106 On the book, it says it's like a faith-based differentiation because he's 00:02:31.126 --> 00:02:34.866 dreaming all these ideas, he's imagining the questions behind it and assuming 00:02:34.886 --> 00:02:36.386 the math is right. 00:02:36.406 --> 00:02:41.266 When he will read papers, he will skip the math part and focus on the substance, 00:02:41.286 --> 00:02:42.666 on the pure idea. 00:02:42.686 --> 00:02:45.306 The best scientists, they develop their intuition. 00:02:45.326 --> 00:02:49.946 Yann LeCun, other top researchers, they rely on their pure intuition. 00:02:49.966 --> 00:02:51.866 But this needs to be developed. 00:02:51.886 --> 00:02:56.386 See how he's not daydreaming, walking all day, just having all the ideas on his 00:02:56.406 --> 00:02:57.786 head. He's probably doing that. 00:02:57.806 --> 00:03:01.835 But more importantly, he's writing everything down. 00:03:02.085 --> 00:03:06.465 When he's 24, he starts his PhD and he's going to do it on neural 00:03:06.485 --> 00:03:11.745 networks. When you're doing a PhD, you need an advisor, someone who will guide 00:03:11.765 --> 00:03:12.865 you during the journey. 00:03:12.885 --> 00:03:16.009 So he picks someone with experience on neural networks. 00:03:16.029 --> 00:03:20.865 Between accepting a place in the lab and actually arriving, his advisor changed 00:03:20.885 --> 00:03:25.103 his mind about neural networks and now he's totally against it. 00:03:25.138 --> 00:03:29.843 And his advisor, when they start working together, came to him and said, Hey, 00:03:29.863 --> 00:03:33.603 why would you work on something that has been proven not to work? 00:03:33.623 --> 00:03:35.443 Don't throw away your life. 00:03:35.463 --> 00:03:38.163 Work on something else that is not neural networks. 00:03:38.183 --> 00:03:42.563 You see, on the 60s, neural networks had a very bad reputation. 00:03:42.583 --> 00:03:46.963 The first experiments that we did, first we didn't have enough compute power, 00:03:46.983 --> 00:03:50.083 and the newspapers completely overhyped. 00:03:50.103 --> 00:03:53.843 And they said that with neural networks we will do all this crazy stuff, and 00:03:53.863 --> 00:03:56.403 obviously that attracted many critics. 00:03:56.423 --> 00:04:00.083 And one thing very important to understand is incentives. 00:04:00.103 --> 00:04:04.003 All the funding at that time comes from the governments, and you have all 00:04:04.023 --> 00:04:07.294 researchers fighting for the same pool amount of money. 00:04:07.314 --> 00:04:12.274 And other researchers of the AI movement proved that simple neural networks had 00:04:12.294 --> 00:04:17.653 simple limitations. Now the problem here is that most academia or 00:04:17.673 --> 00:04:20.373 basically everyone treated as a dead end. 00:04:20.393 --> 00:04:24.293 They said, okay, this is a simple neural network, there are simple limitations, 00:04:24.313 --> 00:04:29.733 this won't work. Instead of an unsolved problems, what if we had a different 00:04:29.753 --> 00:04:31.893 type of neural network? 00:04:31.913 --> 00:04:35.973 And for Geoff, his intuition is saying, this doesn't make sense. 00:04:35.993 --> 00:04:39.493 Like neural networks, there is something here, I need to continue working on 00:04:39.513 --> 00:04:43.813 this. So he spent years doing experiments, trying to create a new language for 00:04:43.833 --> 00:04:47.413 computers to learn. He's doing his own neural networks, trying to do some 00:04:47.433 --> 00:04:52.693 iterations. Because for him, if it's a good idea, you keep trying 00:04:52.713 --> 00:04:58.053 until it works. It doesn't stop being a good idea because it doesn't work the 00:04:58.073 --> 00:04:59.293 first time you try. 00:04:59.313 --> 00:05:01.213 The old is new. 00:05:01.233 --> 00:05:06.317 And every week he will go meet his PhD advisor and I quote the following: 00:05:06.337 --> 00:05:10.390 sometimes it ended in a shouting match, sometimes not. 00:05:10.410 --> 00:05:14.666 And for Geoff, this relationship will traumatize him a lot because he's spending 00:05:14.686 --> 00:05:19.066 years working alone doing all his theories and every week he's trying to fight 00:05:19.086 --> 00:05:23.146 saying I believe in this and everyone is telling him you're wrong. 00:05:23.166 --> 00:05:27.586 And when he will become a mentor to other students and an advisor to the 00:05:27.606 --> 00:05:32.066 students he will develop a different method that was a very successful one. 00:05:32.086 --> 00:05:34.306 It was more an apprenticeship type. 00:05:34.326 --> 00:05:37.746 First we're going to do close work for several years. 00:05:37.766 --> 00:05:40.466 You have freedom to chase other ideas. 00:05:40.486 --> 00:05:45.186 In fact, I want you to work on something I will disapprove. 00:05:45.206 --> 00:05:48.386 And you have permission to prove the advisor wrong. 00:05:48.406 --> 00:05:52.866 So from being a very rigid academia style the advisor is right. 00:05:52.886 --> 00:05:58.146 It's like hey actually the best ideas come from the other places and 00:05:58.166 --> 00:06:01.346 you need to work on something I will totally disapprove. 00:06:01.366 --> 00:06:05.986 We need to have different places and research in academia is very important 00:06:06.006 --> 00:06:08.126 because all this work is public. 00:06:08.326 --> 00:06:09.400 It's a team setting. 00:06:09.420 --> 00:06:12.506 It's not you're not going to figure out everything on your own. 00:06:12.526 --> 00:06:15.226 You're going to write your ideas, you're going to write your argumentation, and 00:06:15.246 --> 00:06:18.506 then you're going to discuss it with very smart people or you will make a paper 00:06:18.526 --> 00:06:23.866 published online so anyone on the world can read it and do a refutation. 00:06:23.886 --> 00:06:28.725 And on the book there is this quote on how was working with with Geoff that I 00:06:28.745 --> 00:06:33.339 love. He had a lifelong habit of running into a room saying he finally 00:06:33.359 --> 00:06:35.739 understood how the brain worked. 00:06:35.759 --> 00:06:40.059 Explaining his new theory, leaving just as quickly as he came, and then 00:06:40.079 --> 00:06:44.219 returning days later to say his theory about the brain was all wrong. 00:06:44.239 --> 00:06:45.819 But now he had a new one. 00:06:45.839 --> 00:06:46.859 And it's very fun. 00:06:46.879 --> 00:06:49.499 He has like a very good sense of humor. 00:06:49.519 --> 00:06:50.699 He has all this energy. 00:06:50.719 --> 00:06:55.299 He's going from one place to another one, but he's not treating him as he I'm 00:06:55.319 --> 00:06:58.099 completely right. It's like he's obsessed with it. 00:06:58.119 --> 00:07:01.139 He cannot stop thinking about neural networks. 00:07:01.159 --> 00:07:05.939 His father dies, and he's like, the old bastard died before I was successful. 00:07:05.959 --> 00:07:07.859 He has something to prove. 00:07:07.879 --> 00:07:12.579 Funding for AI is drying up, the period known as AI winter, because the 00:07:12.599 --> 00:07:14.819 government said, Hey, it's been two years. 00:07:14.839 --> 00:07:16.899 How we made any progress on AI? 00:07:16.919 --> 00:07:20.819 And all of the field or most of it was working on symbolic AI and basically 00:07:20.839 --> 00:07:23.219 writing a bunch of rules for a computer. 00:07:23.239 --> 00:07:25.519 And obviously, that was not making a lot of progress. 00:07:25.559 --> 00:07:28.899 Neural networks were considered a dead end, so we didn't make any progress. 00:07:28.919 --> 00:07:32.659 And government said, hey, we are going to cut funding and put it on other areas. 00:07:32.679 --> 00:07:36.579 Thankfully, there is one place when there is still a bit of funding, and there 00:07:36.599 --> 00:07:40.579 is a small group of people who believe in neural networks, California. 00:07:40.599 --> 00:07:42.979 So he moves to the United States. 00:07:42.999 --> 00:07:48.452 When he's 34, Allen Newell of Carnegie Mellon and one of the founding 00:07:48.472 --> 00:07:51.892 fathers of AI movement hired him to become a researcher. 00:07:51.912 --> 00:07:57.012 The university was focused on symbolic AI, but it was a way to hedge 00:07:57.032 --> 00:08:01.012 their bets. When Allen comes to him and says, Hey Geoff, I'm thinking of 00:08:01.032 --> 00:08:03.412 bringing you as a researcher, you can work on neural networks. 00:08:03.432 --> 00:08:07.172 He was like, if I can work on neural networks, I mean, like, I will sign 00:08:07.192 --> 00:08:08.772 whatever you want. I will become a researcher. 00:08:08.792 --> 00:08:09.972 It's like, how much money do you want? 00:08:09.992 --> 00:08:12.292 Ask him, Allen. He's like, I don't care. 00:08:12.312 --> 00:08:14.372 And Allen's like, okay, 24,000. 00:08:14.392 --> 00:08:16.212 He was like, that's it is. 00:08:16.232 --> 00:08:17.652 And he became a researcher. 00:08:17.672 --> 00:08:22.612 Now, this is super important because when you're a researcher, you're expected 00:08:22.632 --> 00:08:25.492 to collaborate with other researchers to write papers. 00:08:25.512 --> 00:08:30.352 Your only work is to write these papers on academia that everyone will write and 00:08:30.392 --> 00:08:32.252 discuss all these theories. 00:08:32.272 --> 00:08:36.512 Now, the cool thing about this is all the other researchers have a different 00:08:36.532 --> 00:08:41.312 different expertise. Some people know on physics and all other science of 00:08:41.332 --> 00:08:45.152 technology. And if you're a researcher, first you spent many years becoming an 00:08:45.172 --> 00:08:47.632 expert and researching all these types of things. 00:08:47.652 --> 00:08:52.192 And two, you should be very smart, but you should also care about academia. 00:08:52.212 --> 00:08:54.832 And Geoff has to find their people. 00:08:54.852 --> 00:08:56.432 He is on the right place. 00:08:56.452 --> 00:09:00.912 He's surrounded with the best minds, and his only work is to continue working on 00:09:00.932 --> 00:09:02.832 neural networks, his obsession. 00:09:02.852 --> 00:09:06.544 But he will need to debate his ideas with other top researchers 00:09:07.137 --> 00:09:12.482 And 13 years later, after starting his PhD, the big breakthrough, so 00:09:12.502 --> 00:09:17.335 he co authors the paper Boltzmann machine, that is a different kind of neural 00:09:17.355 --> 00:09:20.215 network that learns what patterns are in the data. 00:09:20.235 --> 00:09:24.375 So if you fit a lot of faces into a neural network, the neural network will find 00:09:24.395 --> 00:09:27.720 that hey, there is nose, eyes, mouth all together. 00:09:27.749 --> 00:09:32.222 Now, this showed that a different kind of neural network could learn beyond the 00:09:32.242 --> 00:09:36.222 limitations. It was extremely slow, but it will become one of the modern 00:09:36.242 --> 00:09:39.465 foundations of AI and won him the Nobel Prize. 00:09:39.876 --> 00:09:45.136 Next year he co-authors another paper called backpropagation 00:09:45.156 --> 00:09:49.616 and it's basically a way for a neural network to learn from its mistakes. 00:09:49.636 --> 00:09:54.016 And this will become the standard method for training LLMs on neural networks 00:09:54.036 --> 00:09:55.648 coming into the future. 00:09:55.728 --> 00:09:59.776 Life is going good, we are [unclear phrase] in neural research. 00:09:59.796 --> 00:10:04.656 He marries his wife Rosalind, but they grew to resent their dependence on the 00:10:04.676 --> 00:10:09.536 military funding. United States is having all these conflicts, armed conflicts 00:10:09.556 --> 00:10:12.496 around the world, and Rosalind tells him, Hey, I can no longer live in the 00:10:12.516 --> 00:10:14.176 United States, we need to move. 00:10:14.196 --> 00:10:16.896 So they move to the University of Toronto. 00:10:16.916 --> 00:10:20.656 For years, he's struggling to get funding for his research. 00:10:20.676 --> 00:10:24.656 And with Rosalind, they adopt two children from South America, from Peru and 00:10:24.676 --> 00:10:29.541 Guatemala. Kids are under six when Rosalind gets 00:10:29.561 --> 00:10:33.213 diagnosed with cancer and dies six months later. 00:10:33.233 --> 00:10:37.741 Geoff is crushed. His career, his days of research are over. 00:10:37.761 --> 00:10:41.901 One of his kids has special needs, which means that it will take him more time 00:10:41.921 --> 00:10:43.901 and money to take care of him. 00:10:43.921 --> 00:10:48.933 And it's a huge responsibility to have a special needs kid because you need 00:10:48.953 --> 00:10:53.266 money in case something happens to you or a network to take care of him. 00:10:53.286 --> 00:10:57.733 People on the academia on the 2000s make good money, but they are not rich. 00:10:57.753 --> 00:11:01.493 They are not being paid billions like today to the top researchers of private 00:11:01.513 --> 00:11:03.516 companies. They are working in academia. 00:11:03.662 --> 00:11:09.002 So the next year she's between London and Toronto, and he will 00:11:09.022 --> 00:11:14.087 marry Jackie, and it's an historian who credited on saving his career. 00:11:14.107 --> 00:11:16.842 Like something very important here. 00:11:16.862 --> 00:11:19.162 Jackie deeply believed on him. 00:11:19.182 --> 00:11:24.202 When Jackie got sick many years later, he would tell him to continue his work on 00:11:24.222 --> 00:11:29.168 the lab. It was very important work, and a huge ecosystem grew thanks to Geoff. 00:11:29.188 --> 00:11:33.962 There is something that will inspire people around Geoff to deeply believe in 00:11:33.982 --> 00:11:35.802 him. He had all the support. 00:11:35.822 --> 00:11:40.842 We see like all these successful people they did all alone, but all of them they 00:11:40.862 --> 00:11:45.482 had an environment. And in this case, the people around Geoff, Jackie in this 00:11:45.502 --> 00:11:49.882 case deeply believed on what he was working on, and he allowed him to go back to 00:11:49.902 --> 00:11:54.102 the lab and work night and day to continue his theories, to continue with his 00:11:54.881 --> 00:11:58.883 obsession. It's 2004. Geoff is his 00:11:59.403 --> 00:12:04.263 mid-50s. Neural research is still considered ridiculous. 00:12:04.283 --> 00:12:08.903 So with Yoshua Bengio and Yann LeCun they formed the conspiracy of deep 00:12:08.923 --> 00:12:12.743 learning. The first thing we are going to do is we are going to call instead of 00:12:12.763 --> 00:12:16.823 neural networks, we're going to call deep learning because many people associate 00:12:16.843 --> 00:12:20.583 neural networks unlike just the word creates a lot of resistance. 00:12:20.603 --> 00:12:23.863 They have all these ideas on the whole conflict. 00:12:23.883 --> 00:12:28.003 So by changing the name we are going to sell it as a new thing and basically 00:12:28.203 --> 00:12:31.257 deep learning is a multi layer neural network. 00:12:32.039 --> 00:12:34.819 So by changing the name, we're going to break our resistance. 00:12:34.839 --> 00:12:37.219 We're going to teach it as a cool new thing. 00:12:37.239 --> 00:12:40.259 The second thing is we need a patron, we need funding. 00:12:40.279 --> 00:12:43.699 So Geoff goes to the Canadian government, it's like, okay, please give us money. 00:12:43.719 --> 00:12:47.859 We want to make Canada one of the top places for AI of research. 00:12:47.879 --> 00:12:51.699 And the Canadian government is like, okay, Geoff, here's 400k per year. 00:12:51.719 --> 00:12:55.619 And it's not a lot of money, but with that money, they are able to host 00:12:55.639 --> 00:12:59.939 workshops on their network every year, and they will keep tracking, they will 00:12:59.959 --> 00:13:03.539 keep inviting the best talent and they will keep the community alive. 00:13:03.559 --> 00:13:07.939 And it kind of became like they were like kind of like pirates, or you could say 00:13:07.959 --> 00:13:12.179 it's like a cult obsession, because they will go to a big conference and the 00:13:12.199 --> 00:13:17.639 workshop of deep learning will get rejected, but they will not take no for an 00:13:17.659 --> 00:13:21.899 answer. So they will organize their own side event, rent a bunch of buses and 00:13:21.919 --> 00:13:25.979 get the researchers to the side event and go back to the main event. 00:13:25.999 --> 00:13:30.779 And this academia or this researchers' conference, this was not normal. 00:13:30.799 --> 00:13:34.859 Like they were basically not accepting no for an answer. 00:13:34.879 --> 00:13:38.859 And I think it was so powerful because they deeply believed their intuition, 00:13:38.879 --> 00:13:41.659 they had some theories that deep learning was the future. 00:13:41.679 --> 00:13:46.459 And like they had a very strong and nothing better than do it as a cult. 00:13:46.479 --> 00:13:51.439 A big advantage that Geoff has is all the top upcoming talent that we realized 00:13:51.459 --> 00:13:53.119 that deep learning was the future. 00:13:53.139 --> 00:13:57.019 Well guess what? If you wanted to do your PhD and you wanted a good advisor on 00:13:57.059 --> 00:14:00.632 this, there were few places on the world to do so. 00:14:00.652 --> 00:14:03.432 And the best place, what with Geoff. 00:14:03.452 --> 00:14:05.352 And also it was very fun to work with him. 00:14:05.372 --> 00:14:09.192 You know, he gave you all the flexibility, he was not very rigid, he had all 00:14:09.212 --> 00:14:11.592 this sense of humor, like I told you in the past. 00:14:11.612 --> 00:14:16.872 One of his students will say, I don't know wherever Geoff picks people that end 00:14:16.892 --> 00:14:21.672 up being successful, or somehow he makes them successful. 00:14:21.692 --> 00:14:24.152 Having experience, I think it's the latter. 00:14:24.172 --> 00:14:28.312 And every time he reads a top paper on neural network, there was a connection to 00:14:28.332 --> 00:14:30.872 Geoff as a citation or [unclear]. 00:14:30.892 --> 00:14:35.912 In his case, he was on a train, he saw a competition on developing new 00:14:35.932 --> 00:14:39.272 medicines, and he was like, Oh, Geoff will disapprove this? 00:14:39.292 --> 00:14:42.712 Absolutely. So he enters the competition, and a few weeks later he's saying, 00:14:42.732 --> 00:14:44.792 Geoff, Geoff's like, What are you working on? 00:14:44.812 --> 00:14:47.312 And he's like, Oh, making like a neural network. 00:14:47.332 --> 00:14:50.672 It's actually on the eighth place, the simple neural network, like neural 00:14:50.692 --> 00:14:52.272 network. I have not put a lot of work. 00:14:52.292 --> 00:14:55.152 And with Geoff he was like, no, now you need to win it. 00:14:55.172 --> 00:14:58.672 And he will connect him with other students and they end up winning the 00:14:58.692 --> 00:15:02.891 competition. But he did that because of like, would Geoff disapprove this? 00:15:02.911 --> 00:15:06.791 Yes. And he actually does that, and he's a very successful scientist 00:15:06.831 --> 00:15:10.951 researcher. Now, 2008. 00:15:12.111 --> 00:15:17.131 Compute has getting better thanks to video games to power all these 00:15:17.151 --> 00:15:21.371 videographics with companies, they develop new chips called GPUs. 00:15:21.391 --> 00:15:26.731 And turns out GPUs are way better to train neural networks than CPUs 00:15:26.751 --> 00:15:28.091 and other existing chips. 00:15:28.111 --> 00:15:29.211 They do it way faster. 00:15:29.231 --> 00:15:33.931 So Geoff is telling all the researchers, guys, we need to buy NVIDIA GPUs. 00:15:33.951 --> 00:15:37.546 It's way faster to train neural networks. 00:15:37.566 --> 00:15:40.805 And like this allows him to do way more experiments, obviously, all his students 00:15:40.825 --> 00:15:43.045 have been doing all this experiment together. 00:15:43.065 --> 00:15:45.285 Now, finally, we have compute. 00:15:45.305 --> 00:15:50.085 But to train a neural network, you also need a lot of data, which big companies 00:15:50.105 --> 00:15:55.045 have. So he contacts an old student, Li Deng, who works at Microsoft Research, 00:15:55.065 --> 00:15:57.804 and tells him, hey, this is all the progress we are making. 00:15:57.824 --> 00:16:02.085 Li Deng tells him, hey, like this is very impressive, but you need to work on a 00:16:02.105 --> 00:16:02.945 big problem 00:16:04.185 --> 00:16:06.325 to see if this actually works. 00:16:06.345 --> 00:16:09.525 He tells him, Let's work on speech recognition. 00:16:09.545 --> 00:16:13.525 Basically, computers struggle to recognize speech. 00:16:13.545 --> 00:16:15.205 And he invites him to Seattle. 00:16:15.225 --> 00:16:19.944 So Geoff takes the train, goes to Seattle, and they spend a few weeks working 00:16:19.964 --> 00:16:25.224 together. So Li Deng is optimizing the GPU, and Geoff wrote 10 pages of 00:16:25.244 --> 00:16:30.305 code. Very simple but very good code on the neural network to do 00:16:30.325 --> 00:16:33.766 this. Now, he was not a computer scientist. 00:16:33.786 --> 00:16:35.046 He didn't study for that. 00:16:35.066 --> 00:16:39.686 But within the years doing so many experiments, he learned how to code and wrote 00:16:39.706 --> 00:16:42.726 the first version of the code of work. 00:16:42.746 --> 00:16:47.766 In weeks, they beat existing systems that companies had spent 00:16:47.786 --> 00:16:51.366 years refining and like they had this big progress. 00:16:51.386 --> 00:16:56.006 And then Geoff has an incredible idea, brilliant idea. 00:16:56.026 --> 00:17:01.046 He takes three of his doctoral students and places them in summer internships at 00:17:01.066 --> 00:17:03.606 tech top companies like Google, Microsoft. 00:17:03.626 --> 00:17:08.246 And again, a summer internship for a PhD is different than a normal internship 00:17:08.266 --> 00:17:12.911 because you get access to more technology and way more access to do whatever you 00:17:12.931 --> 00:17:15.311 want at the company. 00:17:15.331 --> 00:17:20.671 But he tells them you need to replace a central system with a deep 00:17:20.691 --> 00:17:25.851 neural network. In 18 months, all three companies develop 00:17:25.891 --> 00:17:28.361 new speech recognition systems. 00:17:28.381 --> 00:17:33.791 Now all the new papers on academia about speech recognition are published. 00:17:33.811 --> 00:17:36.511 Deep learning is starting to get a lot of traction. 00:17:36.531 --> 00:17:40.751 But one thing I didn't tell you is Geoff starting to have a lot of back pain 00:17:40.771 --> 00:17:45.151 problems. This meant he could not sit more than 10 minutes a day. 00:17:45.171 --> 00:17:46.911 So he could not take any planes. 00:17:46.931 --> 00:17:51.871 So when he went to Seattle, he took the train for three days back and forward 00:17:51.891 --> 00:17:53.711 and then between California. 00:17:53.731 --> 00:17:55.791 So he's on his 60s. 00:17:55.811 --> 00:18:00.551 He's taking the train for three days just to continue working on this. 00:18:00.571 --> 00:18:02.951 This guy is not going to stop. 00:18:02.971 --> 00:18:05.740 They are fully determined, to make this work 00:18:06.462 --> 00:18:09.482 But still, not everyone academia is still convinced. 00:18:09.502 --> 00:18:14.202 They're saying, okay, yeah, this works for speech, but it won't work for images. 00:18:14.222 --> 00:18:19.642 So Geoff calls one of the top critics and says, What it will take to 00:18:19.662 --> 00:18:23.722 convince you? And the critic is like, okay, let's do this data set or the other 00:18:23.742 --> 00:18:26.202 one. And Geoff is like, nah, nah, that's too basic. 00:18:26.222 --> 00:18:29.082 That's too simple. What about the holy grail? 00:18:29.102 --> 00:18:33.642 ImageNet. And it's basically a competition where teams try to get the most 00:18:33.662 --> 00:18:38.282 accuracy. You have one million images and you need to classify them into 1000 00:18:38.302 --> 00:18:42.770 objects. You see when you enter a website and you have uh the CAPTCHA prove 00:18:42.790 --> 00:18:47.682 you're not a robot select the bus select the car or select the dog and you feel 00:18:47.702 --> 00:18:49.842 it basically that's the competition. 00:18:49.862 --> 00:18:52.388 Whoever gets the most accuracy win. 00:18:52.408 --> 00:18:56.068 And the critic was like yeah if you can solve that you totally win. 00:18:56.088 --> 00:18:57.850 Like you're not going to do that. 00:18:57.870 --> 00:19:03.108 But he didn't tell to the critic like two of his students Alex and Ilya who then 00:19:03.128 --> 00:19:07.088 would become the co-founder of OpenAI were nearly done with the paper AlexNet 00:19:07.448 --> 00:19:11.508 who was a neural network basically having the best accuracy. 00:19:11.528 --> 00:19:16.128 So the day of the competition the days before they released AlexNet AlexNet 00:19:16.168 --> 00:19:21.268 became the most AI cited paper in the world and he basically 00:19:21.288 --> 00:19:24.345 beat the whole competition by huge margin. 00:19:24.365 --> 00:19:27.664 This was finally the proof that deep learning was the future. 00:19:27.684 --> 00:19:29.984 And now everyone has seen it. 00:19:30.004 --> 00:19:33.824 And you have all the top tech companies saying we need to implement deep 00:19:33.844 --> 00:19:36.784 learning and like we need to acquire the best talent. 00:19:36.804 --> 00:19:40.844 So Geoff goes to Alex and Ilya and said, Hey, we're going to set up a company 00:19:40.884 --> 00:19:44.624 just with the sole goal of getting acquihired. 00:19:44.644 --> 00:19:48.384 Now you don't know how much you're worth at that time because there are 00:19:48.404 --> 00:19:50.704 researchers, it's a three-man company. 00:19:50.724 --> 00:19:56.144 It's 2012. It's like it's not like today we're paying billions for top talent. 00:19:56.164 --> 00:19:59.744 So it's like I don't know how much we charge, I don't want to fight big 00:19:59.764 --> 00:20:02.304 companies. I'm going to set up an auction. 00:20:02.324 --> 00:20:06.292 And every hour each company can raise a bid for one million. 00:20:06.376 --> 00:20:11.915 Okay. So the auction starts five 00:20:11.955 --> 00:20:18.155 ten the first day, 12 million they go to the next day 20, 30, 40 million 00:20:18.355 --> 00:20:21.935 Alex Ilya everyone's like this is more money that we will spend within a 00:20:21.955 --> 00:20:27.375 lifetime and when the auction hits 44 million by Google Geoff stops the auction 00:20:27.395 --> 00:20:31.478 and accepts Google bid and Alex and Ilya is like Why are you stopping? 00:20:31.498 --> 00:20:36.943 There's still money on the table but he tells them hey We are 00:20:36.963 --> 00:20:41.615 researchers, not founders more loyal to the idea than to anything else and 00:20:41.635 --> 00:20:45.595 Google is the best place to go work Google is doing all this open source work 00:20:45.715 --> 00:20:49.135 and you know Google basically makes so much money with their search engine and 00:20:49.155 --> 00:20:52.775 Google so they can do all this open source work and there are options Microsoft 00:20:52.795 --> 00:20:55.815 it was under Ballmer and it didn't have a lot of open source work. 00:20:55.835 --> 00:20:57.255 And Baidu was Chinese. 00:20:57.275 --> 00:21:00.135 So Google was the best place to continue. 00:21:00.155 --> 00:21:03.575 But he had one condition that was non-negotiable. 00:21:03.595 --> 00:21:06.455 He wanted to keep his position at the university. 00:21:06.475 --> 00:21:08.535 He wanted to keep teaching one day per week. 00:21:08.555 --> 00:21:10.449 He didn't want to leave academia. 00:21:10.469 --> 00:21:14.695 And this became the blueprint for other researchers to join the tech side 00:21:14.715 --> 00:21:18.675 without leaving academia. Seeing the top researchers that were in academia, back 00:21:18.795 --> 00:21:23.895 then you didn't have enough leverage to leave fully academia and go fully work. 00:21:23.915 --> 00:21:28.615 So in this way, all the researchers would say, hey, I will get paid top money, 00:21:28.635 --> 00:21:32.355 but I'll also be able to continue my open source work, which is very important 00:21:32.395 --> 00:21:33.895 because this is a shared process. 00:21:33.915 --> 00:21:38.735 Like all this AI process happened because all the top researchers of the world 00:21:38.755 --> 00:21:40.895 were doing papers and they were sharing the work. 00:21:40.915 --> 00:21:44.495 That's why we made so much progress, and they didn't want to lose that. 00:21:44.515 --> 00:21:48.875 For them, continue evolving the work and sharing the process was very important. 00:21:49.914 --> 00:21:55.154 And in 2012, 44 million for three people, it was an unheard 00:21:55.194 --> 00:22:00.421 amount of money. Today we pay billions for the best talent, and on Google, he 00:22:00.441 --> 00:22:04.821 spent the next 10 years helping bring deep learning to the masses. 00:22:04.841 --> 00:22:10.401 At some point, he started working on philosophical AI and the future and 00:22:10.521 --> 00:22:14.421 implications on superintelligence. On his prediction on superintelligence, 00:22:14.441 --> 00:22:18.386 because this is how to read the future, he has the following quote Demis 00:22:18.426 --> 00:22:21.246 Hassabis think it will take 10 years. 00:22:21.266 --> 00:22:24.286 I think it will be between five to twenty years. 00:22:24.306 --> 00:22:28.926 Dario Amodei from Anthropic thinks it will be three years, but of course he runs 00:22:28.946 --> 00:22:32.206 a company. And Ilya thinks it will be sooner than two years. 00:22:32.226 --> 00:22:37.406 But all four we agree that it will be between our lifetimes. 00:22:37.426 --> 00:22:41.726 Now he's very concerned that AI may cause human extinction. 00:22:41.746 --> 00:22:44.366 He considered there is like a 10% chance for it. 00:22:44.386 --> 00:22:46.446 But not everyone agrees with him. 00:22:46.466 --> 00:22:50.439 We have people like Yann LeCun, his protégé, that thinks LLMs are not super 00:22:50.459 --> 00:22:52.451 intelligent, they won't kill us. 00:22:52.471 --> 00:22:55.950 AI will do what it's designed, what humans designed to do. 00:22:55.970 --> 00:22:57.710 It won't have super intelligence. 00:22:57.730 --> 00:22:59.957 So right now you have two big factions on AI. 00:22:59.977 --> 00:23:04.477 You have AI safety that say AI will kill us, we need to control it. 00:23:04.497 --> 00:23:09.197 And other sides say, Yeah, AI safety is important, but LLMs are not super 00:23:09.217 --> 00:23:12.637 intelligence, they are not AGI, and it won't kill us. 00:23:12.657 --> 00:23:14.397 We didn't design it for that. 00:23:14.417 --> 00:23:18.157 Like AI, they are not designed to kill us. 00:23:18.177 --> 00:23:21.620 And to end this episode, I want to finish with two things. 00:23:21.857 --> 00:23:26.637 First, it's always nicer to be part of a successful group than being on your 00:23:26.657 --> 00:23:30.077 own. All of this work was a shared process. 00:23:30.097 --> 00:23:33.517 Like all these researchers from all around the world, they share ideas and they 00:23:33.537 --> 00:23:34.877 did continue the work. 00:23:34.897 --> 00:23:37.357 That's why they made so much progress. 00:23:37.377 --> 00:23:41.037 It didn't matter your age, it didn't matter who you are, it mattered like you 00:23:41.057 --> 00:23:42.637 came with idea and intuition. 00:23:42.657 --> 00:23:47.597 And they rely a lot on the intuition, but research is part of a work group. 00:23:47.617 --> 00:23:52.933 It's not something that you figure out on your own alone in a basement. 00:23:52.953 --> 00:23:56.739 And something that he said when he was 40. 00:23:56.759 --> 00:24:00.885 One day he's 40 years old, [unclear] his birthday, he goes to the lab and he's 00:24:00.905 --> 00:24:04.325 very sad. And like his researchers are very worried. 00:24:04.345 --> 00:24:07.765 Like his teammates are like, why Geoff is having a bad day, what is happening? 00:24:07.785 --> 00:24:10.165 So they go to Geoff is like Geoff why are you sad? 00:24:10.185 --> 00:24:13.365 What is happening? And Geoff is like I'm 40. 00:24:13.385 --> 00:24:15.045 My career is over. 00:24:15.065 --> 00:24:17.605 I will never achieve anything again. 00:24:17.625 --> 00:24:20.865 He thinks when he was 40 his mind would be less agile. 00:24:20.905 --> 00:24:26.005 Turns out that his most important contribution was keeping neural networks 00:24:26.025 --> 00:24:29.425 alive and helping the conspiracy of deep learning 00:24:30.745 --> 00:24:32.604 to keep pushing AI.