CodeFuse AI

Hello World! This is CodeFuse!

The mission of CodeFuse is to develop Code Large Language Models (Code LLMs) specifically designed to support the entire software development lifecycle, covering crucial stages such as design, requirements, coding, testing, deployment, operations, and maintenance. We are passionate about creating innovative solutions that empower developers throughout the software development process.

In this release, we are open sourcing

1. The MFT (Multi-Task Fine-Tuning) framework, known as MFTCoder;
2. Two datasets for enhancing the coding capabilities of LLMs, that is, Code Exercise and Evol-Instruction;
3. A faster and more reliable deployment framework based on FasterTransformer;

The resulting model ensemble, which includes CodeFuse-13B and CodeFuse-CodeLlama-34B, supports various code-related tasks such as code completion, text-to-code conversion, and unit test generation. In particular, CodeFuse-CodeLlama-34B, built upon CodeLlama as the base model and fine-tuned using the proposed MFT framework, achieves an impressive score of 74.4% (greedy decoding) in the HumanEval Python pass@1 evaluation, even surpassing the performance of GPT-4 (67%). We have plans to incorporate additional base LLMs into the ensemble in the near future.

We believe that our solution can significantly enhance the performance of pretrained LLMs across multiple related tasks simultaneously. We are committed to further exploring this direction and providing more open-source contributions. We also encourage engineers and researchers within this community to join us in co-constructing CodeFuse.

CodeFuse的使命是开发专门设计用于支持整个软件开发生命周期的大型代码语言模型（Code LLMs），涵盖设计、需求、编码、测试、部署、运维等关键阶段。我们致力于打造创新的解决方案，让软件开发者们在研发的过程中如丝般顺滑。

在本次发布中，我们开源了以下内容：

1. MFT（多任务微调）框架，也称为MFTCoder；
2. 两个用于增强LLMs编码能力的数据集，包括Code Exercise和Evol-Instruction；
3. 基于FasterTransformer的更快速、更可靠的部署框架;。

由此产生的模型集合包括CodeFuse-13B和CodeFuse-CodeLlama-34B，支持多种与代码相关的任务，如代码补全、文本转代码、单元测试生成等。值得一提的是，CodeFuse-CodeLlama-34B基于CodeLlama作为基础模型，并利用我们提出的MFT框架进行微调，在HumanEval Python pass@1评估中取得高达的74.4%（贪婪解码）的好成绩，甚至超过了GPT-4（67%）的表现。我们计划在不久的将来将更多的基础LLMs纳入到我们的模型集合中。

我们相信我们的解决方案可以显著提升预训练LLMs在多个相关任务上的性能。我们将进一步探索这个方向，并提供更多的开源贡献。与此同时，我们诚邀志同道合的工程师和研究人员加入这个社区，共同构建CodeFuse。