Tag: AI Benchmarks

Highlights Meta's Llama 3 features 405 billion parameters, surpassing its predecessor. Supports eight languages with an expanded context window for larger requests. Significant improvements in math and knowledge benchmarks. Future multimodal versions to include image, video, and speech capabilities. Meta Platforms has released its most advanced artificial intelligence model to date, Llama 3, featuring enhanced multilingual capabilities and improved performance in coding and mathematics. 🔵 META UNVEILS BIGGEST LLAMA 3 AI MODEL, TOUTING LANGUAGE AND MATH GAINS Meta Platforms released the biggest version of its mostly free Llama 3 artificial intelligence models on Tuesday, boasting multilingual skills and general performance metrics that nip at the heels of paid… pic.twitter.com/5vVUBx5d1T — PiQ (@PiQSuite) July 23, 2024 As reported by Reuters, this latest iteration boasts 405 billion parameters, significantly surpassing its predecessor while still remaining smaller than some competitors' models. New Future for Llama 3 Meta Unveils Largest Llama 3 AI Model, Boasting Language and Math Improvements According to Reuters, Meta CEO Mark Zuckerberg expressed confidence in the future of Llama models, stating, "I expect future Llama models would overtake proprietary competitors by next year." Zuckerberg also predicted that the Meta AI chatbot powered by these models would become "the most popular AI assistant by the end of this year, with hundreds of millions of people using it already." The new Llama 3 model offers support for eight languages and features an expanded "context window," allowing for improved handling of larger user requests. Ahmad Al-Dahle, Meta's head of generative AI, told Reuters, "That was the number one feedback we got from the community," highlighting the benefits for computer code generation in particular. Updated AI Parameters It supports eight languages with an expanded context window for larger requests Reuters reports that Meta is releasing updated versions of its 8 billion and 70 billion parameter Llama 3 models alongside the flagship 405 billion parameter version. All three models are multilingual and feature the expanded context window. Category Benchmark Llama 3.1 8B Llama 3 8B - April Llama 3.1 70B Llama 3 70B - April Llama 3.1 405B General MMLU 73 65.3 86 80.9 88.6 MMLU (CoT) 48.3 45.5 66.4 63.4 73.3 MMLU PRO (5-shot, CoT) 80.4 76.8 87.5 82.9 88.6 Code IFEval 72.6 60.4 80.5 81.7 89 HumanEval (0-shot) 72.8 70.6 86 82.5 88.6 MBPP EvalPlus (base) (0-shot) 84.5 80.6 95.1 93 96.8 Math GSM8K (8-shot, CoT) 51.9 29.1 68 51 73.8 MATH (0-shot, CoT) 83.4 82.4 94.8 94.4 96.9 Reasoning ARC Challenge (0-shot) 32.8 34.6 46.7 39.5 51.1 GPQA (0-shot, CoT) 82.6 48.3 90 85.1 92.3 Tool use API-Bank (0-shot) 76.1 60.3 84.8 83 88.5 BFCL 8.2 1.7 29.7 14.7 35.3 Gorilla Benchmark API Bench 38.5 18.1 56.7 47.8 58.7 Nexus (0-shot) 68.9 - 86.9 - 91.6 Multilingual Multilingual MGSM - - - - - Meta's strategy of releasing Llama models largely free of charge aims to hone innovation and reduce dependence on potential competitors. As per Reuters, Zuckerberg believes this approach will lead to "innovative products, less dependence on would-be competitors and greater engagement on the company's core social networks." Future multimodal versions to include image, video, and speech capabilities The company claims significant improvements in key math and knowledge tests, with results suggesting that the largest Llama 3 model is approaching or surpassing the performance of leading paid models like Anthropic's Claude 3.5 Sonnet and OpenAI's GPT-4o in some areas. On the MATH benchmark of competition-level math word problems, Meta's model scored 73.8, compared to GPT-4o's 76.6 and Claude 3.5 Sonnet's 71.1. On the MMLU benchmark covering various academic subjects, Llama 3 achieved a score of 88.6, nearly matching GPT-4o's 88.7 and surpassing Claude 3.5 Sonnet's 88.3. Meta researchers also hinted at upcoming "multimodal" versions of the models, which will incorporate image, video, and speech capabilities. Early experiments suggest these models may compete with other multimodal offerings such as Google's Gemini 1.5 and Anthropic's Claude 3.5 Sonnet. FAQs What is the Llama 3 AI model by Meta? The Llama 3 AI model is Meta's most advanced artificial intelligence model, featuring 405 billion parameters, enhanced multilingual capabilities, and improved performance in coding and mathematics. What improvements does Llama 3 offer over its predecessor? Llama 3 supports eight languages, features an expanded context window for handling larger requests, and shows significant improvements in math and knowledge benchmarks. How does Llama 3 perform in benchmarks compared to competitors? Llama 3 scores highly on key benchmarks, nearly matching or surpassing leading models like Anthropic's Claude 3.5 Sonnet and OpenAI's GPT-4o in some areas. What is Meta's strategy for releasing Llama 3? Meta plans to release Llama 3 models largely free of charge to foster innovation and reduce dependence on potential competitors, aiming to boost engagement on its social networks. Are there plans for future versions of Llama 3? Yes, Meta researchers hinted at upcoming multimodal versions of the models that will incorporate image, video, and speech capabilities. Also Read: Apple Intelligence: AI-Powered Enhancements Boost Company Stock to Record High Also Read: Apple to Integrate OpenAI Technology in iOS 18: Enhancing AI Features

Tag: AI Benchmarks

Popular articles

Qualcomm Snapdragon 8s Gen 4 Leak Reveals Specs, AnTuTu Score, Launch Timeline and More

Google Unveils Gemini 2.5 – A Smarter AI Model That Thinks Before Answering

Paytm Payments Services Discontinues Ties With Third-Party Platforms, Transitions To Direct Processing

Instagram Teams Up with Schools to Tackle Online Bullying and Student Safet

Apple’s WWDC 2025 To Kick Off On June 9, Apple To Unveil iOS 19 and More

About us

Quick Links

Must Read

Qualcomm Snapdragon 8s Gen 4 Leak Reveals Specs, AnTuTu Score, Launch Timeline and More

Google Unveils Gemini 2.5 – A Smarter AI Model That Thinks Before Answering

Paytm Payments Services Discontinues Ties With Third-Party Platforms, Transitions To Direct Processing