Meta releases ‘Code Llama 70B’, an open-source behemoth to rival private AI development

Meta AI, the company that brought you Llama 2, the gargantuan language model that can generate anything from tweets to essays, has just released a new and improved version of its code generation model, Code Llama 70B. This updated model can write code in various programming languages, such as Python, C++, Java and PHP, from natural language prompts or existing code snippets. And it can do it faster, better and more accurately than ever before.

Code Llama 70B is one of the largest open-source AI models for code generation currently available and also happens to be the latest benchmark in the field of code generation, a branch of artificial intelligence that aims to automate the process of creating and modifying software. Code generation has been a long-standing goal of computer scientists, as it promises to make software development more efficient, accessible and creative. Imagine being able to tell your computer what you want it to do, and have it write the code for you. Or being able to modify and improve existing code with a few simple commands. Or being able to translate code from one language to another with ease. These are some of the possibilities that code generation models like Code Llama 70B can offer.

But code generation is not an easy task. Unlike natural language, which is often ambiguous and flexible, code is precise and rigid. It has to follow strict rules and syntax, and it has to produce the desired output and behavior. Moreover, code is often complex and lengthy, requiring a lot of context and logic to understand and generate. To tackle these challenges, code generation models need to have a lot of data, a lot of computing power, and a lot of intelligence.

That’s where Code Llama 70B comes in. Code Llama 70B is a state-of-the-art large language model (LLM) that has been trained on 500 billion tokens of code and code-related data, making it more capable and robust than its predecessors. It also has a larger context window of 100,000 tokens, which enables it to process and generate longer and more complex code.

Code Llama 70B is based on Llama 2, one of the largest LLMs in the world, with 175 billion parameters. Llama 2 is a general-purpose LLM that can generate text in any domain and style, from poetry to news articles. Code Llama 70B is a specialized version of Llama 2 that has been fine-tuned for code generation, using a technique called self-attention, which allows it to learn the relationships and dependencies between different parts of the code.

New standard for accuracy and adaptability

One of the highlights of Code Llama 70B is CodeLlama-70B-Instruct, a variant that has been fine-tuned for understanding natural language instructions and generating code accordingly. This variant scored 67.8 on HumanEval, a benchmark dataset of 164 programming problems that test the functional correctness and logic of code generation models. This score surpasses the previous best results of open models, such as CodeGen-16B-Mono (29.3) and StarCoder (40.1), and is comparable to closed models, such as GPT-4 (68.2) and Gemini Pro (69.4). CodeLlama-70B-Instruct can handle a variety of tasks, such as sorting, searching, filtering and manipulating data, as well as implementing algorithms, such as binary search, Fibonacci and factorial.

Code Llama 70B also includes CodeLlama-70B-Python, a variant that has been optimized for Python, one of the most popular and widely used programming languages in the world. This variant has been trained on an additional 100 billion tokens of Python code, making it more fluent and accurate in generating Python code. CodeLlama-70B-Python can also handle a range of tasks, such as web scraping, data analysis, machine learning (ML) and web development.

Accessible for both research and commercial use

Code Llama 70B is available for free download under the same license as Llama 2 and previous Code Llama models, which allows both researchers and commercial users to use and modify it. The model can be accessed and used through various platforms and frameworks, such as Hugging Face, PyTorch, TensorFlow and Jupyter Notebook. Meta AI also provides documentation and tutorials on how to use and fine-tune the model for different purposes and languages.

Mark Zuckerberg, the CEO of Meta AI, said in a statement posted to his Facebook account: “We’re open sourcing a new and improved Code Llama, including a larger 70B parameter model. Writing and editing code has emerged as one of the most important uses of AI models today. The ability to code has also proven to be important for AI models to process information in other domains more rigorously and logically. I’m proud of the progress here, and looking forward to including these advances in Llama 3 and future models as well.”

Code Llama 70B is expected to have a significant impact on the field of code generation and the software development industry, as it offers a powerful and accessible tool for creating and improving code. It can also lower the barrier to entry for people who want to learn coding, as it can provide guidance and feedback based on natural language instructions. Moreover, Code Llama 70B can potentially enable new applications and use cases, such as code translation, code summarization, code documentation, code analysis and code debugging.

Code Llama 70B and other Llama models are available for download directly from Meta AI. More information and documentation can be found on GitHub and Hugging Face.

VentureBeat's mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Discover our Briefings.