Most of LLaMA's core original authors have resigned, and the inside story of Meta AI has been exposed! The competition for computing power was fierce, and the members of the large model team changed for three consecutive rounds

Original source: Xinzhiyuan

Image source: Generated by Unbounded AI‌

In the past six months, the popularity of ChatGPT has completely made Meta unable to sit still.

From the release of the open source large model series LLaMA in February, to the upgrade of Llama 2, and then to the coding model Code Llama, Meta is betting everything on ALL IN AI.

In the open source community, the free research and commercialization of the Llama series of models directly ignited the fission of the replacement model.

However, while Meta is doing well, the AI team is facing a wave of resignations.

The Information exclusively reported that due to internal disputes over computing power, most of the team members of the LLaMA and OPT projects have resigned.

Even, Meta directly abandoned the model they were developing to rival PaLM, and reorganized the two laboratory teams of Meta AI to focus on the development of Llama 2.

The departure of most of Meta's researchers just exposed that the shortage of computing power is the core problem of layout generative AI.

LLaMA papers have 14 authors, half of whom leave

To the outside world, tech giants have more computing resources than most companies. Some companies, including Meta, consistently emphasize that they are "GPU RICH" when recruiting AI researchers.

But in fact, I know very well that Meta's computing resource supply is also limited.

As a result, the computing power dispute started in the internal team, causing many people to abandon Meta.

Specifically, more than half of the 14 authors in the first-generation LLaMA model released in February this year have all resigned.

Some turned to AI startups, while others joined tech giants.

Paper address:

Research scientists and engineers leaving Meta include Timothée Lacroix, Guillaume Lample and Marie-Anne Lachaux, co-authors of the original LLaMA paper.

They now work at French startup Mistral AI, which Lacroix and Lample co-founded in June.

One of the authors, Armand Joulin, research director at Meta AI, left Meta in May and has now joined Apple.

Joelle Pineau, head of "Basic AI Research" at the Meta Artificial Intelligence Research Laboratory, said in an interview:

Retaining and attracting great people is probably where I spend the most time because I can't do anything without great researchers.

The wave of departures Meta is facing just highlights the challenges that large technology companies are facing in response to the surge in demand for artificial intelligence talents.

In the current wave of large models, major technology companies are eager to integrate AI into their products and services for fear of falling behind.

According to Kyle Langworthy, an executive at Riviera Partners, companies that are "hungry for talent" are lining up to poach Meta's artificial intelligence researchers.

In February, after the release of LLaMA ignited the enthusiasm of the community, Xiao Zha once announced the company's big move to the outside world overnight: the establishment of a top product team, focusing on AIGC.

In short, Xiao Zha puts the core focus of Meta's future on generative artificial intelligence in an attempt to catch up with competitors such as Google, Microsoft, and OpenAI.

The "Metaverse" that was once All In in 2021 has also been placed in a secondary position.

Immediately afterwards, Meta released and open sourced the latest research in the field of artificial intelligence from time to time, including the general visual model SAM "Split Everything", multi-modal AI models ImageBind, Llama2, Code Llama, etc.

Once these studies were released, they aroused great response in the open source community and received unanimous praise.

Now, with the loss of experienced researchers, Meta has added another obstacle to catch up with competitors.

So, what exactly happened to make these senior researchers leave in anger?

The computing power battle between LLaMA and OPT

In 2013, after Yann LeCun joined Meta, he established the artificial intelligence research laboratory Meta AI (formerly FAIR) when he was the director of AI research.

In addition to its main labs in Menlo Park, California, and New York (where LeCun is based), Meta AI also has offices in Montreal (where Pineau is based), Paris, London, and Seattle.

The establishment of the Meta AI Laboratory is mainly dedicated to the research of AI, applying artificial intelligence to various scenarios such as translation and MRI scanning.

However, after the emergence of ChatGPT, the company's executives became more eager to integrate AI into products at a deeper level.

In fact, Meta had been developing its own LLM before the release of ChatGPT.

In May 2022, a North American team of Meta AI announced and open sourced the large model OPT-175B.

Meta claims that this model consumes less energy than GPT-3, even though the two models learn quantitatively the same amount during training.

A few months later, the team began working on a second, larger model, designed to compete with Google's PaLM.

At the same time, another Meta AI team in Paris has begun to develop a separate large-scale language model, also known as LLaMA.

The model is smaller than OPT, the people said. The research team believes that smaller models will be more efficient at inference.

Then a problem arose. Teams from two laboratories in North America and Paris began to compete for access to computing resources, which in turn intensified the disputes among internal teams.

In particular, the LLaMA team feels that it has been ignored. Its model is small, but it receives much less computing resources than the North American OPT team.

Joelle Pineau, head of "Basic AI Research" said,

Decisions about how to allocate computing resources at Meta are made roughly once a month by a group of leaders from different parts of the business. How much is ultimately allocated, and to whom, is determined by organizational priorities and how close the project is to release.

If there is a dispute about the allocation of Meta AI employees, it will be submitted to the superior, which is the level of Pineau for processing.

In the interview, Pineau admitted that there is some tension between the LLaMA and OPT teams over the distribution of computing power, adding, "We try to find a way to listen to everyone's voice, even if we can't meet everyone's needs."

To some researchers, it feels like a competition.

Some Meta AI executives also questioned why both teams were working on similar projects but competing with each other for limited resources.

February Great Change

Tensions between the two teams reached a fever pitch in February this year.

Just this month, in an effort to catch up to competitors, Meta AI released LLaMA as an open-source model for the first time, licensed for research purposes.

In fact, one week before the release, Antoine Bordes, the co-director of Meta AI and head of the Paris branch, left Meta. The external reason for leaving Meta was that the working hours were as exhausting as those in California.

Now, Bordes has joined artificial intelligence company Helsing.

His departure further insulates the LLaMA team from executives in North America, the people said. Additionally, Pineau acknowledged that Bordes' departure "created some uncertainty" for the team.

It’s not just a matter of research morale on the LLaMA team; meanwhile, the OPT team has been facing attrition over the past few months.

It is said that the larger model being developed by the OPT team was directly abandoned by Meta. It is not difficult to understand that the loss of OPT personnel is directly related to this decision.

According to the Linkedin profile of the study author, half of the 19 authors of OPT papers have left Meta.

Paper address:

Pineau, the head of "Basic AI Research", said helplessly, "When all teams want to upgrade the model at this time, then you either choose one of them, or let them cooperate."

Team reorganization, unclear positioning

While all this was happening, Meta AI was also overwhelmed by the company's frequent changes.

Since November last year, Xiaozha has carried out several company-wide layoffs. Among them, Meta AI is not immune.

In February of this year, Meta AI leaders decided to bring together some members of the competing LLaMA and OPT teams to formally establish the "Generative AI" team, (former Apple executive Ahmad Al-Dahle is in charge), and also draw from Meta AI A lot of people.

In fact, we can see many LLaMA and OPT authors in the Llama 2 paper.

For example, in the OPT team, Moya Chen, Todor Mihaylov, and Punit Singh Koura joined the "Generative AI" team.

In the first generation of LLaMA team, Hugo Touvron, Thibaut Lavril, Xavier Martinet, Marie-Anne Lachaux, Naman Goyal, Aurelien Rodriguez joined the "Generative AI" team.

Interestingly, in the acknowledgment of Llama 2, three of the four teams that first initiated this research mentioned have left, and only Edouard Grave is still there.

Despite the success of the acclaimed open source model LLaMA, tensions between researchers continue to brew as Meta's changing attitude toward AI research continues.

Traditionally, Meta AI has had a bottom-up culture led by researchers, allowing the team to focus on the field of AI and make breakthrough progress.

But as Xiao Zha became more and more inclined to integrate AI into Meta's applications, Meta AI's focus also narrowed-disbanding research that was not product-oriented, such as protein folding.

At the same time, new projects are also jumping back and forth between the generative AI team and Meta AI.

For example, Llama 2, released by Meta in July, and Code Llama, which specializes in code generation, released in August, are each in charge of two teams.

In this regard, Pineau said, "This field is developing too fast. We are still exploring, which project should Meta AI be responsible for? Which project should the generative AI team be responsible for?"

In front of the stage, the open source model is bright and beautiful, but behind the scenes, the wave of resignations of AI researchers, disputes over computing power, and team reorganization have brought many uncertainties to Meta's efforts to catch up with its opponents in AI.

References: