Full text of Sam Altman's China Dialogue: We must be alert to the risks of AI, but it is much easier to understand neural networks than to understand what people are thinking
Sam Altman's speech took place at the AI Security and Alignment sub-forum of Zhiyuan Conference on June 10. The scene was full of seats. When the OpenAI CEO appeared on the screen, there was applause, and almost everyone raised their mobile phones to take pictures on the screen.
But Altman himself appears calm, even cautious. This is the first time since ChatGPT stirred up the global AI boom last year that Sam Altman has publicly expressed his opinions on a Chinese background.
In fact, he was also not far from China that day. He had just arrived in Seoul and met with the South Korean president. After his speech, he also had a one-on-one Q&A with Zhang Hongjiang, chairman of Zhiyuan Research Institute. The following are the key points and facts.
Key points:
As we get closer and closer to AGI in technology, the effects and pitfalls of misalignment will increase exponentially.
OpenAI currently uses reinforcement learning human feedback technology to ensure that AI systems are useful and safe, and is also exploring new technologies. One of the ideas is to use AI systems to assist humans to supervise other AI systems.
Humans will have powerful artificial intelligence systems (AI) within ten years.
OpenAI has no relevant new open-source timeline, and while he acknowledges that the open-source model has advantages when it comes to AI safety, open-sourcing everything may not be the best route.
It's much easier to understand a neural network than a human brain.
China has the best artificial intelligence talents, and AI security requires the participation and contribution of Chinese researchers.
The following is the transcript of the speech:
Today, I want to talk about the future. Specifically, the rate of growth that we're seeing in AI capabilities. What do we need to do now to prepare the world responsibly for their introduction, the history of science has taught us that technological progress follows an exponential curve. We can already see this in history, from agriculture and industry to the computing revolution. What is astounding about artificial intelligence is not only its impact, but also the speed of its progress. It pushes the boundaries of human imagination, and it does so at a rapid pace.
Imagine that within the next decade, systems commonly referred to as artificial general intelligence (AGI) surpass human expertise in nearly every domain. These systems could eventually exceed the collective productivity of our largest companies. There's huge upside potential lurking here. The AI revolution will create shared wealth and make it possible to improve everyone's standard of living, address common challenges such as climate change and global health security, and improve societal well-being in countless other ways.
I strongly believe in this future, and in order to realize it and enjoy it, we need to collectively invest in AGI safety, and manage risk. If we're not careful, an AGI system that isn't fit for purpose could undermine the entire healthcare system by making unfounded recommendations. Likewise, an AGI system designed to optimize agricultural practices may inadvertently deplete natural resources or damage ecosystems, affecting food production and environmental balance due to lack of consideration for long-term sustainability.
I hope we can all agree that advancing AGI safety is one of our most important areas. I want to focus the rest of my talk on where I think we can start.
One area is the governance of AGI, a technology with global implications. The cost of accidents from reckless development and deployment will affect us all.
In this regard, there are two key elements:
First, we need to establish international norms and standards and, through an inclusive process, develop equal and uniform protections for the use of AGI in all countries. Within these protections, we believe that people have ample opportunity to make their own choices.
Second, we need international cooperation to build global trust in the safe development of increasingly powerful AI systems, in a verifiable way. This is not an easy task. We need sustained and critical attention as the international community to do this well. The Tao Te Ching reminds us that a journey of a thousand miles begins with a single step. We think the most constructive first step to take here is to work with the international tech community.
In particular, we should promote mechanisms for increased transparency and knowledge sharing regarding technological advances in AGI safety. Researchers who discover emerging security issues should share their insights for the greater good. We need to think carefully about how we can encourage such norms while respecting and protecting intellectual property rights.
If we do this well, it will open new doors for us to deepen our cooperation. More broadly, we should invest in, facilitate, and direct investment in targeting and security research.
At OpenAI, our targeted research today focuses on technical questions about how to get AI systems to act as useful and safe assistants in our current systems. This could mean, how do we train ChatGPT so that it doesn't make threats of violence or assist users in harmful activities.
But as we get closer to AGI, the potential impact and magnitude of any non-compliance will grow exponentially. To address these challenges ahead of time, we strive to minimize the risk of catastrophic future outcomes. For the current system, we primarily use reinforcement learning from human feedback to train our model to act as a useful and safe assistant.
This is an example of a post-training target technique, and we're busy developing new ones as well. It takes a lot of hard engineering work to do this well. It took us 8 months to do this from the time GPT-4 finished pre-training to deploying it. Overall, we think we're on the right track here. GPT-4 fits the target better than any of our previous models.
However, targeting remains an open problem for more advanced systems, which we believe will require new technical approaches, as well as more governance and oversight. Imagine a futuristic AGI system coming up with 100,000 lines of binary code. Human supervisors are unlikely to detect if such a model is doing something nefarious.
So we're investing in some new and complementary research directions that we hope will lead to breakthroughs. One is scalable supervision. We can try to use AI systems to assist humans in overseeing other AI systems. For example, we can train a model to help human supervisors spot flaws in the output of other models. The second is interpretability. We wanted to try to better understand what's going on inside these models.
We recently published a paper using GPT-4 to interpret neurons in GPT-2. In another paper, we use model internals to detect when the model is lying. While we still have a long way to go, we believe that advanced machine learning techniques can further improve our ability to generate explanations.
Ultimately, our goal is to train AI systems to help target research itself. A promising aspect of this approach is that it scales with the pace of AI development. As future models become ever smarter and more useful as assistants, we will find better techniques that realize the extraordinary benefits of AGI while mitigating the risks, one of the most important challenges of our time.
The following is the transcript of the conversation:
Zhang Hongjiang: How far are we from artificial intelligence? Is the risk urgent, or are we far from it? Whether it's safe AI or potentially unsafe AI.
Sam Altman: This problem is difficult to predict precisely because it requires new research ideas that don't always develop according to the prescribed schedule. It could happen quickly, or it could take longer. I think it's hard to predict with any degree of certainty. But I do think that within the next decade, we may have very powerful AI systems. In such a world, I think it is important and urgent to solve this problem, which is why I call on the international community to work together to solve this problem. History does give us some examples of new technologies changing the world faster than many imagine. The impact and acceleration of these systems that we are seeing now is in a sense unprecedented. So I think it makes a lot of sense to be prepared for it to happen as soon as possible, and to address the security aspects, given their impact and importance.
Zhang Hongjiang: Do you feel a sense of urgency?
Sam Altman: Yeah, I feel it. I want to emphasize that we don't really know. And the definition of artificial intelligence is different, but I think in ten years, we should be ready for a world with very powerful systems.
Zhang Hongjiang: You also mentioned several global cooperations in your speech just now. We know that the world has faced many crises in the past six or seven decades. But for many of these crises, we managed to build consensus and global cooperation. You, too, are on a global tour. What kind of global collaboration are you promoting? How do you feel about the feedback you've received so far?
Sam Altman: Yes, I'm very happy with the feedback I've received so far. I think people are taking the risks and opportunities presented by AI very seriously. I think the discussion on this has come a long way in the past six months. People are really working on figuring out a framework where we can enjoy these benefits while working together to mitigate the risks. I think we're in a very good position to do that. Global cooperation is always difficult, but I see it as an opportunity and a threat that can bring the world together. It would be very helpful if we could develop some frameworks and security standards to guide the development of these systems.
Zhang Hongjiang: On this particular topic, you mentioned that the alignment of advanced artificial intelligence systems is an unsolved problem. I've also noticed that Open AI has put a lot of effort into it over the past few years. You also mentioned GPT-4 as the best example in terms of alignment. Do you think we can solve AI safety problems through alignment? Or is this problem bigger than alignment?
Sam Altman: I think there are different uses of the word alignment. I think what we need to address is the whole challenge of how to achieve safe AI systems. Alignment has traditionally been about getting the model to behave as the user intends, and that's certainly part of it. But there are other questions we need to answer, like how we verify that systems do what we want them to do, and whose values we align systems with. But I think it's important to see the full picture of what is needed to achieve safe AI.
Zhang Hongjiang: Yes, alignment is still the case. If we look at what GPT-4 has done, for the most part, it's still from a technical standpoint. But there are many other factors besides technology. This is a very complex question. Often complex problems are systemic. AI security may be no exception. Besides the technical aspects, what other factors and issues do you think are critical to AI safety? How should we respond to these challenges? Especially since most of us are scientists, what are we supposed to do?
Sam Altman: This is of course a very complex question. I would say that without a technical solution, everything else is difficult. I think it's really important to put a lot of focus on making sure we address the technical aspects of security. As I mentioned, it's not a technical problem to figure out what values we want to align the system with. It needs technical input, but it is an issue that requires in-depth discussion by the whole society. We must design systems that are fair, representative and inclusive. And, as you pointed out, we have to consider not only the security of the AI model itself, but the security of the entire system. So it's also important to build security classifiers and detectors that can run on top of the model and can monitor compliance with usage policies. And then, I also think it's hard to predict in advance what's going to go wrong with any technology. So learn from the real world and deploy iteratively, see what happens when you put the model in the real world, and improve it, and give people and society time to learn and update, and think about how these models will be used for good and affect their lives in bad ways. This is also very important.
Zhang Hongjiang: You just mentioned global cooperation. You have visited many countries and you mentioned China. But can you share some of the results you've achieved in terms of collaboration? What plans or ideas do you have for next steps? From this world tour, from your interactions with various governments, institutions, institutions?
Sam Altman: I think a lot of different perspectives and AI safety are generally required. We don't have all the answers yet, and this is a rather difficult and important question.
Also, as mentioned, it's not a purely technical question to make AI safe and beneficial. Involves understanding user preferences in different countries in very different contexts. We need a lot of different inputs to make this happen. China has some of the best AI talent in the world. Fundamentally, I think the best minds from around the world are needed to address the difficulty of aligning advanced AI systems. So I really hope that Chinese AI researchers can make great contributions here.
Zhang Hongjiang: I understand that today's forum is about AI safety, because people are very curious about OpenAI, so I have a lot of questions about OpenAI, not just about AI safety. I have an audience question here, is there any plan for OpenAI to re-open source its models like it did before version 3.0? I also think open source is good for AI safety.
Sam Altman: Some of our models are open source and some are not, but as time goes on, I think you should expect us to continue to open source more models in the future. I don't have a specific model or timeline, but it's something we're discussing right now.
Zhang Hongjiang: We put all our efforts into open source, including the model itself, the algorithms to develop the model, and the tools to optimize the relationship between the model and the data. We believe in the need to share and make users feel in control of what they use. Do you have similar feedback? Or is this what you guys are discussing in OpenAI?
Sam Altman: Yeah, I think open source does have an important role in a way. There have also been a lot of new open source models emerging recently. I think the API model also has an important role. It provides us with additional security controls. You can block certain uses. You can block certain types of tweaks. If something doesn't work, you can take it back. At the scale of the current model, I'm not too worried about that. But as the model becomes as powerful as we expect it to be, if we're right about it, I think open sourcing everything might not be the best path, although sometimes it's right. I think we just have to balance it carefully.
Zhang Hongjiang: The follow-up question on GPT-4 and AI security is, do we need to change the entire infrastructure or the architecture of the entire AGI model to make it safer and easier to check? What are your thoughts on this?
Sam Altman: It's definitely possible, we need some very different architectures, both in terms of capabilities and security. I think we're going to be able to make some progress in explainability, on current types of models, and have them better explain to us what they're doing and why. But it wouldn't surprise me if there was another giant leap after the transformers. And actually we are already in the original transformer, the architecture has changed a lot.
Zhang Hongjiang: As a researcher, I am also curious, what is the next direction of AGI research? In terms of large models, large language models, will we see GPT-5 soon? Is the next frontier in embodied models? Is autonomous robotics an area that OpenAI is or plans to explore?
Sam Altman: I'm also curious about what's next, and one of my favorite things about doing this work is that there's a lot of excitement and surprise at the cutting edge of research. We don't have the answers yet, so we're exploring many possible new paradigms. Of course, at some point, we will try to do a GPT-5 model, but not anytime soon. We don't know when exactly. We've been working on robotics since the very beginning of OpenAI, and we're very interested in it, but we've had some difficulties. I hope one day we can go back to this field.
Zhang Hongjiang: Sounds great. You also mentioned in your presentation how you use GPT-4 to explain how GPT-2 works, making the model more secure. Is this approach scalable? Is this direction OpenAI will continue to advance in the future?
Sam Altman: We will continue to push in this direction.
Zhang Hongjiang: Do you think this method can be applied to biological neurons? Because the reason I ask this question is that there are some biologists and neuroscientists who want to use this method to study and explore how human neurons work in their field.
Sam Altman: It's much easier to see what's going on on artificial neurons than on biological neurons. So I think this approach is valid for artificial neural networks. I think there is a way to use more powerful models to help us understand other models. But I'm not quite sure how you would apply this approach to the human brain.
Zhang Hongjiang: OK, thank you. Now that we've talked about AI safety and AGI control, one of the questions we've been discussing is, would it be safer if there were only three models in the world? It's like nuclear control, you don't want nuclear weapons to proliferate. We have this treaty where we try to control the number of countries that can get this technology. So is controlling the number of models a feasible direction?
Sam Altman: I think there are different opinions on whether it is safer to have a minority model or a majority model in the world. I think it's more important, do we have a system where any robust model is adequately tested for safety? Do we have a framework where anyone who creates a sufficiently robust model has both the resources and the responsibility to ensure that what they create is safe and aligned?
Zhang Hongjiang: At this meeting yesterday, Professor Max of MIT Future of Life Institute mentioned a possible method, which is similar to the way we control drug development. When scientists or companies develop new drugs, you cannot directly market them. You have to go through this testing process. Is this something we can learn from?
Sam Altman: I definitely think we can learn a lot from the licensing and testing frameworks that have been developed in different industries. But I think fundamentally we've got something that can work.
Zhang Hongjiang: Thank you very much, Sam. Thank you for taking the time to attend this meeting, albeit virtually. I'm sure there are many more questions, but given the time, we have to stop here. I hope that next time you have the opportunity to come to China, come to Beijing, we can have a more in-depth discussion. thank you very much.
View Original
The content is for reference only, not a solicitation or offer. No investment, tax, or legal advice provided. See Disclaimer for more risks disclosure.
Full text of Sam Altman's China Dialogue: We must be alert to the risks of AI, but it is much easier to understand neural networks than to understand what people are thinking
Author｜Neil Shen
Source丨Pinwan
Sam Altman's speech took place at the AI Security and Alignment sub-forum of Zhiyuan Conference on June 10. The scene was full of seats. When the OpenAI CEO appeared on the screen, there was applause, and almost everyone raised their mobile phones to take pictures on the screen.
But Altman himself appears calm, even cautious. This is the first time since ChatGPT stirred up the global AI boom last year that Sam Altman has publicly expressed his opinions on a Chinese background.
In fact, he was also not far from China that day. He had just arrived in Seoul and met with the South Korean president. After his speech, he also had a one-on-one Q&A with Zhang Hongjiang, chairman of Zhiyuan Research Institute. The following are the key points and facts.
Key points:
As we get closer and closer to AGI in technology, the effects and pitfalls of misalignment will increase exponentially.
OpenAI currently uses reinforcement learning human feedback technology to ensure that AI systems are useful and safe, and is also exploring new technologies. One of the ideas is to use AI systems to assist humans to supervise other AI systems.
Humans will have powerful artificial intelligence systems (AI) within ten years.
OpenAI has no relevant new open-source timeline, and while he acknowledges that the open-source model has advantages when it comes to AI safety, open-sourcing everything may not be the best route.
It's much easier to understand a neural network than a human brain.
China has the best artificial intelligence talents, and AI security requires the participation and contribution of Chinese researchers.
The following is the transcript of the speech:
Today, I want to talk about the future. Specifically, the rate of growth that we're seeing in AI capabilities. What do we need to do now to prepare the world responsibly for their introduction, the history of science has taught us that technological progress follows an exponential curve. We can already see this in history, from agriculture and industry to the computing revolution. What is astounding about artificial intelligence is not only its impact, but also the speed of its progress. It pushes the boundaries of human imagination, and it does so at a rapid pace.
Imagine that within the next decade, systems commonly referred to as artificial general intelligence (AGI) surpass human expertise in nearly every domain. These systems could eventually exceed the collective productivity of our largest companies. There's huge upside potential lurking here. The AI revolution will create shared wealth and make it possible to improve everyone's standard of living, address common challenges such as climate change and global health security, and improve societal well-being in countless other ways.
I strongly believe in this future, and in order to realize it and enjoy it, we need to collectively invest in AGI safety, and manage risk. If we're not careful, an AGI system that isn't fit for purpose could undermine the entire healthcare system by making unfounded recommendations. Likewise, an AGI system designed to optimize agricultural practices may inadvertently deplete natural resources or damage ecosystems, affecting food production and environmental balance due to lack of consideration for long-term sustainability.
I hope we can all agree that advancing AGI safety is one of our most important areas. I want to focus the rest of my talk on where I think we can start.
One area is the governance of AGI, a technology with global implications. The cost of accidents from reckless development and deployment will affect us all.
In this regard, there are two key elements:
First, we need to establish international norms and standards and, through an inclusive process, develop equal and uniform protections for the use of AGI in all countries. Within these protections, we believe that people have ample opportunity to make their own choices.
Second, we need international cooperation to build global trust in the safe development of increasingly powerful AI systems, in a verifiable way. This is not an easy task. We need sustained and critical attention as the international community to do this well. The Tao Te Ching reminds us that a journey of a thousand miles begins with a single step. We think the most constructive first step to take here is to work with the international tech community.
In particular, we should promote mechanisms for increased transparency and knowledge sharing regarding technological advances in AGI safety. Researchers who discover emerging security issues should share their insights for the greater good. We need to think carefully about how we can encourage such norms while respecting and protecting intellectual property rights.
If we do this well, it will open new doors for us to deepen our cooperation. More broadly, we should invest in, facilitate, and direct investment in targeting and security research.
At OpenAI, our targeted research today focuses on technical questions about how to get AI systems to act as useful and safe assistants in our current systems. This could mean, how do we train ChatGPT so that it doesn't make threats of violence or assist users in harmful activities.
But as we get closer to AGI, the potential impact and magnitude of any non-compliance will grow exponentially. To address these challenges ahead of time, we strive to minimize the risk of catastrophic future outcomes. For the current system, we primarily use reinforcement learning from human feedback to train our model to act as a useful and safe assistant.
This is an example of a post-training target technique, and we're busy developing new ones as well. It takes a lot of hard engineering work to do this well. It took us 8 months to do this from the time GPT-4 finished pre-training to deploying it. Overall, we think we're on the right track here. GPT-4 fits the target better than any of our previous models.
However, targeting remains an open problem for more advanced systems, which we believe will require new technical approaches, as well as more governance and oversight. Imagine a futuristic AGI system coming up with 100,000 lines of binary code. Human supervisors are unlikely to detect if such a model is doing something nefarious.
So we're investing in some new and complementary research directions that we hope will lead to breakthroughs. One is scalable supervision. We can try to use AI systems to assist humans in overseeing other AI systems. For example, we can train a model to help human supervisors spot flaws in the output of other models. The second is interpretability. We wanted to try to better understand what's going on inside these models.
We recently published a paper using GPT-4 to interpret neurons in GPT-2. In another paper, we use model internals to detect when the model is lying. While we still have a long way to go, we believe that advanced machine learning techniques can further improve our ability to generate explanations.
Ultimately, our goal is to train AI systems to help target research itself. A promising aspect of this approach is that it scales with the pace of AI development. As future models become ever smarter and more useful as assistants, we will find better techniques that realize the extraordinary benefits of AGI while mitigating the risks, one of the most important challenges of our time.
The following is the transcript of the conversation:
Zhang Hongjiang: How far are we from artificial intelligence? Is the risk urgent, or are we far from it? Whether it's safe AI or potentially unsafe AI.
Sam Altman: This problem is difficult to predict precisely because it requires new research ideas that don't always develop according to the prescribed schedule. It could happen quickly, or it could take longer. I think it's hard to predict with any degree of certainty. But I do think that within the next decade, we may have very powerful AI systems. In such a world, I think it is important and urgent to solve this problem, which is why I call on the international community to work together to solve this problem. History does give us some examples of new technologies changing the world faster than many imagine. The impact and acceleration of these systems that we are seeing now is in a sense unprecedented. So I think it makes a lot of sense to be prepared for it to happen as soon as possible, and to address the security aspects, given their impact and importance.
Zhang Hongjiang: Do you feel a sense of urgency?
Sam Altman: Yeah, I feel it. I want to emphasize that we don't really know. And the definition of artificial intelligence is different, but I think in ten years, we should be ready for a world with very powerful systems.
Zhang Hongjiang: You also mentioned several global cooperations in your speech just now. We know that the world has faced many crises in the past six or seven decades. But for many of these crises, we managed to build consensus and global cooperation. You, too, are on a global tour. What kind of global collaboration are you promoting? How do you feel about the feedback you've received so far?
Sam Altman: Yes, I'm very happy with the feedback I've received so far. I think people are taking the risks and opportunities presented by AI very seriously. I think the discussion on this has come a long way in the past six months. People are really working on figuring out a framework where we can enjoy these benefits while working together to mitigate the risks. I think we're in a very good position to do that. Global cooperation is always difficult, but I see it as an opportunity and a threat that can bring the world together. It would be very helpful if we could develop some frameworks and security standards to guide the development of these systems.
Zhang Hongjiang: On this particular topic, you mentioned that the alignment of advanced artificial intelligence systems is an unsolved problem. I've also noticed that Open AI has put a lot of effort into it over the past few years. You also mentioned GPT-4 as the best example in terms of alignment. Do you think we can solve AI safety problems through alignment? Or is this problem bigger than alignment?
Sam Altman: I think there are different uses of the word alignment. I think what we need to address is the whole challenge of how to achieve safe AI systems. Alignment has traditionally been about getting the model to behave as the user intends, and that's certainly part of it. But there are other questions we need to answer, like how we verify that systems do what we want them to do, and whose values we align systems with. But I think it's important to see the full picture of what is needed to achieve safe AI.
Zhang Hongjiang: Yes, alignment is still the case. If we look at what GPT-4 has done, for the most part, it's still from a technical standpoint. But there are many other factors besides technology. This is a very complex question. Often complex problems are systemic. AI security may be no exception. Besides the technical aspects, what other factors and issues do you think are critical to AI safety? How should we respond to these challenges? Especially since most of us are scientists, what are we supposed to do?
Sam Altman: This is of course a very complex question. I would say that without a technical solution, everything else is difficult. I think it's really important to put a lot of focus on making sure we address the technical aspects of security. As I mentioned, it's not a technical problem to figure out what values we want to align the system with. It needs technical input, but it is an issue that requires in-depth discussion by the whole society. We must design systems that are fair, representative and inclusive. And, as you pointed out, we have to consider not only the security of the AI model itself, but the security of the entire system. So it's also important to build security classifiers and detectors that can run on top of the model and can monitor compliance with usage policies. And then, I also think it's hard to predict in advance what's going to go wrong with any technology. So learn from the real world and deploy iteratively, see what happens when you put the model in the real world, and improve it, and give people and society time to learn and update, and think about how these models will be used for good and affect their lives in bad ways. This is also very important.
Zhang Hongjiang: You just mentioned global cooperation. You have visited many countries and you mentioned China. But can you share some of the results you've achieved in terms of collaboration? What plans or ideas do you have for next steps? From this world tour, from your interactions with various governments, institutions, institutions?
Sam Altman: I think a lot of different perspectives and AI safety are generally required. We don't have all the answers yet, and this is a rather difficult and important question.
Also, as mentioned, it's not a purely technical question to make AI safe and beneficial. Involves understanding user preferences in different countries in very different contexts. We need a lot of different inputs to make this happen. China has some of the best AI talent in the world. Fundamentally, I think the best minds from around the world are needed to address the difficulty of aligning advanced AI systems. So I really hope that Chinese AI researchers can make great contributions here.
Zhang Hongjiang: I understand that today's forum is about AI safety, because people are very curious about OpenAI, so I have a lot of questions about OpenAI, not just about AI safety. I have an audience question here, is there any plan for OpenAI to re-open source its models like it did before version 3.0? I also think open source is good for AI safety.
Sam Altman: Some of our models are open source and some are not, but as time goes on, I think you should expect us to continue to open source more models in the future. I don't have a specific model or timeline, but it's something we're discussing right now.
Zhang Hongjiang: We put all our efforts into open source, including the model itself, the algorithms to develop the model, and the tools to optimize the relationship between the model and the data. We believe in the need to share and make users feel in control of what they use. Do you have similar feedback? Or is this what you guys are discussing in OpenAI?
Sam Altman: Yeah, I think open source does have an important role in a way. There have also been a lot of new open source models emerging recently. I think the API model also has an important role. It provides us with additional security controls. You can block certain uses. You can block certain types of tweaks. If something doesn't work, you can take it back. At the scale of the current model, I'm not too worried about that. But as the model becomes as powerful as we expect it to be, if we're right about it, I think open sourcing everything might not be the best path, although sometimes it's right. I think we just have to balance it carefully.
Zhang Hongjiang: The follow-up question on GPT-4 and AI security is, do we need to change the entire infrastructure or the architecture of the entire AGI model to make it safer and easier to check? What are your thoughts on this?
Sam Altman: It's definitely possible, we need some very different architectures, both in terms of capabilities and security. I think we're going to be able to make some progress in explainability, on current types of models, and have them better explain to us what they're doing and why. But it wouldn't surprise me if there was another giant leap after the transformers. And actually we are already in the original transformer, the architecture has changed a lot.
Zhang Hongjiang: As a researcher, I am also curious, what is the next direction of AGI research? In terms of large models, large language models, will we see GPT-5 soon? Is the next frontier in embodied models? Is autonomous robotics an area that OpenAI is or plans to explore?
Sam Altman: I'm also curious about what's next, and one of my favorite things about doing this work is that there's a lot of excitement and surprise at the cutting edge of research. We don't have the answers yet, so we're exploring many possible new paradigms. Of course, at some point, we will try to do a GPT-5 model, but not anytime soon. We don't know when exactly. We've been working on robotics since the very beginning of OpenAI, and we're very interested in it, but we've had some difficulties. I hope one day we can go back to this field.
Zhang Hongjiang: Sounds great. You also mentioned in your presentation how you use GPT-4 to explain how GPT-2 works, making the model more secure. Is this approach scalable? Is this direction OpenAI will continue to advance in the future?
Sam Altman: We will continue to push in this direction.
Zhang Hongjiang: Do you think this method can be applied to biological neurons? Because the reason I ask this question is that there are some biologists and neuroscientists who want to use this method to study and explore how human neurons work in their field.
Sam Altman: It's much easier to see what's going on on artificial neurons than on biological neurons. So I think this approach is valid for artificial neural networks. I think there is a way to use more powerful models to help us understand other models. But I'm not quite sure how you would apply this approach to the human brain.
Zhang Hongjiang: OK, thank you. Now that we've talked about AI safety and AGI control, one of the questions we've been discussing is, would it be safer if there were only three models in the world? It's like nuclear control, you don't want nuclear weapons to proliferate. We have this treaty where we try to control the number of countries that can get this technology. So is controlling the number of models a feasible direction?
Sam Altman: I think there are different opinions on whether it is safer to have a minority model or a majority model in the world. I think it's more important, do we have a system where any robust model is adequately tested for safety? Do we have a framework where anyone who creates a sufficiently robust model has both the resources and the responsibility to ensure that what they create is safe and aligned?
Zhang Hongjiang: At this meeting yesterday, Professor Max of MIT Future of Life Institute mentioned a possible method, which is similar to the way we control drug development. When scientists or companies develop new drugs, you cannot directly market them. You have to go through this testing process. Is this something we can learn from?
Sam Altman: I definitely think we can learn a lot from the licensing and testing frameworks that have been developed in different industries. But I think fundamentally we've got something that can work.
Zhang Hongjiang: Thank you very much, Sam. Thank you for taking the time to attend this meeting, albeit virtually. I'm sure there are many more questions, but given the time, we have to stop here. I hope that next time you have the opportunity to come to China, come to Beijing, we can have a more in-depth discussion. thank you very much.