Back to Repositories

Testing AI-Driven Test Generation Pipeline in AlphaCodium

This test suite implements AI-driven test generation for coding problems, focusing on automated test case creation and validation within the AlphaCodium framework. The suite handles both AI-generated and public test cases while incorporating configurable validation steps.

Test Coverage Overview

The test suite provides comprehensive coverage for AI test generation functionality:

Handles multiple test case generation attempts with retry logic
Supports configurable number of AI-generated test cases
Integrates public test cases with AI-generated ones
Includes validation mechanisms for generated tests

Implementation Analysis

The implementation follows an async pattern for test generation and validation:

Uses async/await for test generation and validation flows
Implements error handling with retry mechanism
Utilizes YAML for test case structure and parsing
Employs functional programming with partial application

Technical Details

Key technical components include:

Logging framework for operation tracking
YAML processing for test case formatting
Configuration management via settings loader
AI inference integration for test generation
Async execution framework

Best Practices Demonstrated

The test suite exhibits several testing best practices:

Separation of concerns between generation and validation
Configurable test parameters
Robust error handling and retry logic
Structured logging for debugging
Clean integration of multiple test sources

codium-ai/alphacodium

alpha_codium/gen/stages/run_generate_ai_test.py

            
import functools
import logging

from alpha_codium.gen.stages.indirect.run_validate_ai_test import run_validate_ai_tests
from alpha_codium.gen.utils import load_yaml
from alpha_codium.settings.config_loader import get_settings
from alpha_codium.llm.ai_invoker import send_inference
from alpha_codium.log import get_logger

logger = get_logger(__name__)


async def run_generate_ai_tests(self, problem):
    counter_retry = 0
    while True:
        try:
            logger.info("--generate ai tests stage--")

            # get settings
            validate_ai_tests = get_settings().get('generate_ai_tests.validate_ai_tests', False)
            problem['number_of_ai_tests'] = get_settings().get("generate_ai_tests.number_of_ai_tests", 8)
            problem['use_test_explanations_possible_solutions'] = get_settings().get('generate_ai_tests.use_test_explanations')

            # get prompt
            f = functools.partial(self._run, problem=problem, prompt="code_contests_prompts_generate_ai_tests")

            # inference
            response_problem_tests, _ = await send_inference(f)
            problem['problem_ai_tests'] = load_yaml(response_problem_tests,
                                                    keys_fix_yaml=["input:", "output:", "explanation:"])['tests']
            problem['problem_ai_simple_test'] = problem['problem_ai_tests'][0]

            if validate_ai_tests:
                problem = await run_validate_ai_tests(self, problem)

            # adding public tests to the beginning and end of the list, for the ai-iterate stage
            if get_settings().get('generate_ai_tests.add_public_tests_to_ai_tests', False):
                for public_input, public_output in zip(problem['public_tests']['input'],
                                                       problem['public_tests']['output']):
                    # to the beginning of the list
                    problem['problem_ai_tests'].insert(0, {'input': public_input, 'output': public_output})
                    # to the end of the list
                    problem['problem_ai_tests'].append({'input': public_input, 'output': public_output})

            return problem
        except Exception as e:
            logging.error(f"'generate ai tests' stage, counter_retry {counter_retry}, Error: {e}")
            counter_retry += 1
            if counter_retry > 2:
                raise e