View on GitHub

MuCoCo Code Inconsistency Testing Framework

Home

MuCoCo: Code Consistency Testing Framework

A research framework for evaluating the consistency and reliability of Large Language Models (LLMs) in code-related tasks through code mutation.

Overview

Large Language Models (LLMs) often portray inconsistent behaviors which result in severe consequences for coding tasks. Developers typically employ benchmarks to assess Code LLMs, but most benchmarks are hand-crafted, static or do not target consistency property. In this work, we pose the following scientific question: How can we automatically discover inconsistent program behaviors in Code LLMs? To address this challenge, we propose an automated consistency testing method, called MUCOCO, which employs mutation analysis to expose inconsistent behaviors in code LLMs. Given a coding dataset (e.g., HumanEval), MUCOCO (1) transforms its programs into semantically equivalent programs (aka mutants), (2) validates mutant correctness using the test suite, and (3) detects inconsistencies when the original code query leads to correct LLM behavior, but its mutant induces an incorrect behavior (incorrect output or test failure). MUCOCO provides 11 semantic- preserving mutation operators spanning (a) lexical (b) syntactic and (c) logical mutations. We evaluate MUCOCO using four (4) state-of-the-art benchmarks, four (4) programming tasks, and seven (7) LLMs. Results show that consistency errors are prevalent in LLMs and MUCOCO is effective in exposing inconsistency: About one in nine inputs generated by MUCOCO exposed inconsistencies. In addition, we found that MUCOCO outperforms the closest baseline (TURBULENCE). Finally, we show that MUCOCO scales to multiple (two) mutations which outperforms atomic mutations. Overall, our work motivates the need to test Code LLMs for consistency properties.

Code Mutation Types

To test the code inconsistency in LLM models, this project utilizes code mutation. In this project, code mutation refers to the process of modifying a program in at least of the following ways:

  1. syntactically (change in flow of program),
  2. lexically (changing variable names) and/or,
  3. logically (changing logical statement conditions)

while preserving the semantics of the original program. The framework supports multiple types of code mutations to test LLM consistency that fall in 1 of the 3 aforementioned types:

Evaluated LLM Models

MuCoCo framework supports the following LLMs:

It should be noted that Codestral, GPT-4o, GPT5 and DeepSeek experiments were conducted through API keys, which you will need to obtain from their respective websites. Qwen, Gemma and Llama models have their model weights downloaded from HuggingFace from their respective repositories. Experiments using model weights downloaded from HuggingFace are conducted using Google Colab, while models using API keys are conducted through a local setup.

Getting Started

Downloading this repository

Your first step is to download this repository and save it locally on your computer.

High-level Overview of Project Directory

This section covers the high level overview of the overall project directory. Folders marked with a ‘❌’ do not need to be explored for running experiments and generally contain scripts necessary for running the project.

- Code Reasoning Model Research Project
  ├── baseline/ # Turbulence baseline testing
  ├── code_generation/ # MuCoCo code generation experiments
  ├── ❌ code_mutation/  
  ├── ❌ datasets/  
  ├── ❌ llm_models/  
  ├── mcq_inconsistency/ # MuCoCO mcq inconsistency experiments
  ├── MuCoCo_results/ # Scripts used for aggregating MuCoCo results
  ├── prediction_inconsistency/ # MuCoCo prediction inconsistency experiments
  ├── ❌ utility/
  ├── .env.example # .env.example for your .env file
  ├── ❌ .gitattributes
  ├── ❌ .gitignore
  ├── ❌ database.py  
  ├── README.md # this file
  ├── requirements-colab-big-code-bench.txt
  ├── requirements-colab.txt
  ├── requirements.txt
  └── setup.ipynb

Creating your .env file

Your next step is to create and populate your .env file. To create your .env file, simply copy the .env.example file and rename it to .env. To run all experiments on all models, you will need to ensure the necessary API tokens are filled in.

MongoDB

This project stores the dataset in MongoDB databases. Hence, you will need a MongoDB URI to store the datasets.

  1. Head to MongoDB and create a new account or sign in to an existing account
  2. Create a new project in MongoDB and give it a suitable name. A project houses multiple clusters, which are isolated database environments that you can use to manage, scale, and monitor your applications independently.
  3. Create a new cluster in your project. At the top of the page, you should have the option to choose between several configurations. Choose the “Free” option and choose a suitable name for your cluster. The “Free” tier will suffice for this project.

  4. Upon creating your cluster, you should be presented with a pop-up with 2 sections.

    The first section involves whitelisting IP address to access your cluster. You can go with “Allow Access From Anywhere” and add it to your “Network Access”. When you head to the Network Access tab afterwards, you should see the IP address “0.0.0.0/0” in the list.

    The second section involves creating users for your database. Create your first user for the database. Choose an appropriate username and password for this user. Should your team wish to share a single database, you can head to the “Database Access” tab and create more users from there.

  5. Now, you will need to connect to your cluster to add and pull data from the databases in the cluster. To connect to the cluster from VSCode, navigate to the “Clusters” page and click on “Connect”. Then, select the option “MongoDB for VSCode” and follow the steps on the pop-up page. From there, you will form your MongoDB Connection String (URI)

    For example, if your cluster name is “Question_Database”, your username is “AlexRider” and your password is “Ark_Angel”, then your MongoDB URI should look something like this: “mongodb+srv://AlexRider:Ark_Angel@Question_Database.ovenrr0.mongodb.net/”. Save your MongoDB URI in your .env file under the name “MONGODB_URI”

    This cluster will be used to store collections, which will house the datasets that you will be using.

  6. It is recommend for users to download MongoDB Compass for an intuitive UI to view any changes / entries in the MongoDB database conveniently. Alternatively, you can still use the MongoDB webpage to view the database.

Model Setup

This section will be covering the steps you need to take to set up the models for running MuCoCo experiments. It will be split into two sections - models using API keys and models using HuggingFace transformers that are run on Google Colab (or GPU)

Models Using API Keys

It should be noted that models that require API keys generally require credits to run and will require you to top up.

Codestral

You will need a Mistral API key to use Codestral.

  1. To obtain your own Mistral API Key, simply head to the official Mistral AI website and sign up or sign in to an existing account. Complete any sign up procedures.
  2. Next, navigate to the homepage of your account. On the left hand side of the homepage, you should see a tab called “API Keys”. Head to that page and create a new key. You may leave the expiration date empty.
  3. On the next pop-up, you should be presented with the API key. Copy down the API key onto your .env file under the name CODESTRAL_API_KEY.

GPT Models (GPT-4o and GPT-5)

You will need an OpenAI API key to use GPT-4o and GPT-5.

  1. Head to the official Open AI Platform Website for API keys.
  2. Login to your existing account or sign up.
  3. Click on “Create new secret key” button on the the top right of the page.
  4. Complete the rest of the steps and save your Open AI API key under OPENAI_API_KEY in .env

DeepSeek

You will need a DeepSeek API key to use DeepSeek-V3.2.

  1. Head to the official DeepSeek Platform and sign in.
  2. Login to your existing account or sign up.
  3. Once you are logged in into DeepSeek Platform, navigate to the “API Keys” tab on the left side.
  4. Click on “Create new API key” and save your DeepSeek API key under DEEPSEEK_API_KEY in .env

By this step, you should have your CODESTRAL_API_KEY, DEEPSEEK_API_KEY, OPENAI_API_KEY and MONGODB_URI fields filled in your .env file.

Models Using Google Colab

For models that use HuggingFace transformers (Qwen2.5-Coder-14B-Instruct, Gemma-3-12b-it, Llama-3.1-8b), we provide Google Colab notebooks that handle the setup and execution:

These notebooks automatically:

  1. Install the required dependencies (BigCodeBench uses requirements-colab-big-code-bench.txt, others use requirements-colab.txt)
  2. Load the model weights from HuggingFace
  3. Set up the experimental environment
  4. Provide ready-to-run experiment code

To use these notebooks:

  1. Upload the notebook to Google Colab
  2. Use Google Colab Pro+ with A100 GPU runtime for optimal performance
  3. Follow the instructions within each notebook
  4. The notebooks will guide you through the entire experimental setup

Running the notebooks

There are two types of notebooks: database builder notebooks and notebooks for running experiments. Database builder notebooks build your database on MongoDB using .csv dataset files downloaded from HuggingFace. On the other hand, experiment notebooks are used for running code inconsistency experiments.

Creating a virtual environment

Before running the notebooks, you will need to create a Python Virtual Environment (venv).

Do note that this set up process is specific to users using the MacOS. If you are using Windows or any other OS, you may still follow these steps, but some terminal commands will not work as intended and you will need to do some troubleshooting by yourself.

Once you have Python set up, ensure that you are in the correct project folder. Then, you can create a venv using the following command in your terminal:

python -m venv venv

If you are using MacOS, activate the venv using the following command (Windows will need another command):

source venv/bin/activate

Then, install the Python packages using pip:

pip install -r requirements.txt

Wait for the packages to finish installing. You may need to do some troubleshooting should you run into any dependency installation conflicts at this stage.

Building the database

  1. Ensure that you have the datasets downloaded in .csv format. From the project’s main directory, the csv datasets should be under datasets/open_ended_format. You should have datasets for BigCodeBench, CodeMMLU, CruxEval, and a modified HumanEval dataset.
  2. Navigate to code_generation/test_notebooks to start setting up for code generation experiments.
  3. Next, you may head to mcq_inconsistency or prediction_inconsistency for the respective MuCoCo experiments.

Notes for database building

  1. You will need to run code_generation/humaneval_database_builder.ipynb before you can run input_output_prediction/humaneval_database_builder.ipynb else it will fail. The input_output_prediction for humaneval database builder relies on the code generation counterpart.
  2. When building the database for BigCodeBench dataset, there may some “residual” files from the dataset that will appear in the directory. You can delete these files without any problems.

Identify Code Inconsistencies

Scripts for identifying code inconsistency scores are found under MuCoCo_results/notebooks from the project root directoy. The notebooks are named according to the RQ the results are reported in. Navigate to that project sub directory for more information.

You will need to download the MuCoCo experiment results from Figshare and store it in MuCoCo_results folder from the project root directory. The name of the folder storing the results should be named MuCoCo_experiment_results. These results are necessary for running the results aggregation scripts.