Meta

Answering any natural language question on the world's fastest AI supercomputer

Fundamental AI Research (FAIR)

OptimalAI scientists support Meta AI's Fundamental AI Research (FAIR) team across 3D computer vision, knowledge intensive and multilingual natural language programming, reinforcement learning and emerging areas of AI. The mission of Meta's FAIR is to advance state-of-the-art AI through open, fundamental research and applications for the benefit of all.

Research SuperCluster

Some of OptimalAI scientists' research at Meta AI has utilized the Research SuperCluster (RSC). The RSC is one of the world's fastest AI supercomputers, capable of quintillions (a billion billion) of calculations per second. RSC helps AI researchers build new and better AI models that can learn from trillions of examples, work across hundreds of different languages and seamlessly analyze text, images, and video to develop new augmented reality tools. Researchers are able to train some of the largest AI models needed to develop advanced AI for computer vision, NLP, speech recognition and more.

Massively Multilingual Speech

Meta is focused on multilinguality in general and its 'Massively Multilingual Speech' (MMS) project increases the number of supported speech technology languages by 10-40x. The system utilizes a new dataset based on readings of publicly available religious texts and uses self-supervised learning. The team built pre-trained wav2vec 2.0 models covering 1,406 languages, a single multilingual automatic speech recognition model for 1,107 languages, speech synthesis models for the same number of languages, as well as a language identification model for 4,017 languages.

Some of the languages, such as Tatuyo, have only a few hundred speakers, and for most of these languages, no prior speech technology exists. However, the MMS models outperformed existing models and halved the word error rate of OpenAI's Whisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data.

Publication

GitHub

Meta AI: The first AI-powered speech translation system for a primarily oral language 2m 12s

Llama3

LLAMA3 (Large Language Model Meta AI) is Meta AI's third generation large language model, following the success of its predecessor, LLAMA2. This release enhances its foundation models, which now range from 10 billion to 85 billion parameters, expanding its capability and efficiency. Trained on over 3 trillion tokens, LLAMA3 incorporates 1.5 million new human annotations, emphasizing its commitment to training cutting-edge models using only publicly accessible datasets.

Like LLAMA2, LLAMA3 is pretrained on publicly sourced online data. An advanced version named LLAMA-3-chat is developed, initially through supervised fine-tuning. Subsequent refinements of LLAMA-3-chat leverage an updated methodology of Reinforcement Learning from Human Feedback (RLHF), employing advanced techniques like rejection sampling and enhanced proximal policy optimization (PPO).

The updated RLHF process integrates additional human feedback mechanisms, enhancing the model's learning efficacy and aligning closely with core requirements of safety and utility. These modifications aim to streamline the training process, improve the models' responsiveness, and ensure higher quality interaction patterns.

In a continued partnership with Microsoft, Meta released the LLAMA3 code as open source, reaffirming their commitment to transparency and the widespread availability of advanced AI tools for both research and commercial applications.

Publication

GitHub