Back to Repositories

Testing JSON Parser Edge Cases in AutoGPT

This test suite validates JSON parsing functionality, focusing on handling malformed JSON inputs and edge cases. It tests both fixable and unfixable JSON formatting scenarios using pytest framework.

Test Coverage Overview

The test suite provides comprehensive coverage of JSON parsing scenarios.

Key areas tested include:

Malformed JSON with missing/extra commas
Trailing commas and newlines
Missing brackets and braces
Various numeral formats
Boolean and null value handling
Complex nested structures

Implementation Analysis

The testing approach uses pytest’s parameterized testing through fixtures to iterate over predefined test cases. The implementation leverages two main fixtures – fixable_json and unfixable_json – to validate the parsing behavior for different JSON malformations.

Key patterns include:

Fixture-based parameterization
Expected vs actual result comparison
Structured test data organization

Technical Details

Testing tools and configuration:

pytest framework for test execution
Custom json_loads parser implementation
Python’s native json module for validation
Parameterized test fixtures
Assertion-based validation

Best Practices Demonstrated

The test suite exemplifies several testing best practices including comprehensive edge case coverage, clear test data organization, and effective use of pytest features.

Notable practices:

Structured test data separation
Comprehensive edge case coverage
Clear test case organization
Effective fixture utilization
Maintainable test structure

significant-gravitas/autogpt

classic/forge/forge/json/test_parsing.py

            
import json

import pytest

from .parsing import json_loads

_JSON_FIXABLE: list[tuple[str, str]] = [
    # Missing comma
    ('{"name": "John Doe"   "age": 30,}', '{"name": "John Doe", "age": 30}'),
    ("[1, 2 3]", "[1, 2, 3]"),
    # Trailing comma
    ('{"name": "John Doe", "age": 30,}', '{"name": "John Doe", "age": 30}'),
    ("[1, 2, 3,]", "[1, 2, 3]"),
    # Extra comma in object
    ('{"name": "John Doe",, "age": 30}', '{"name": "John Doe", "age": 30}'),
    # Extra newlines
    ('{"name": "John Doe",
"age": 30}', '{"name": "John Doe", "age": 30}'),
    ("[1, 2,
3]", "[1, 2, 3]"),
    # Missing closing brace or bracket
    ('{"name": "John Doe", "age": 30', '{"name": "John Doe", "age": 30}'),
    ("[1, 2, 3", "[1, 2, 3]"),
    # Different numerals
    ("[+1, ---2, .5, +-4.5, 123.]", "[1, -2, 0.5, -4.5, 123]"),
    ('{"bin": 0b1001, "hex": 0x1A, "oct": 0o17}', '{"bin": 9, "hex": 26, "oct": 15}'),
    # Broken array
    (
        '[1, 2 3, "yes" true, false null, 25, {"obj": "var"}',
        '[1, 2, 3, "yes", true, false, null, 25, {"obj": "var"}]',
    ),
    # Codeblock
    (
        '```json
{"name": "John Doe", "age": 30}
```',
        '{"name": "John Doe", "age": 30}',
    ),
    # Multiple problems
    (
        '{"name":"John Doe" "age": 30
 "empty": "","address": '
        "// random comment
"
        '{"city": "New York", "state": "NY"},'
        '"skills": ["Python" "C++", "Java",""],',
        '{"name": "John Doe", "age": 30, "empty": "", "address": '
        '{"city": "New York", "state": "NY"}, '
        '"skills": ["Python", "C++", "Java", ""]}',
    ),
    # All good
    (
        '{"name": "John Doe", "age": 30, "address": '
        '{"city": "New York", "state": "NY"}, '
        '"skills": ["Python", "C++", "Java"]}',
        '{"name": "John Doe", "age": 30, "address": '
        '{"city": "New York", "state": "NY"}, '
        '"skills": ["Python", "C++", "Java"]}',
    ),
    ("true", "true"),
    ("false", "false"),
    ("null", "null"),
    ("123.5", "123.5"),
    ('"Hello, World!"', '"Hello, World!"'),
    ("{}", "{}"),
    ("[]", "[]"),
]

_JSON_UNFIXABLE: list[tuple[str, str]] = [
    # Broken booleans and null
    ("[TRUE, False, NULL]", "[true, false, null]"),
    # Missing values in array
    ("[1, , 3]", "[1, 3]"),
    # Leading zeros (are treated as octal)
    ("[0023, 015]", "[23, 15]"),
    # Missing quotes
    ('{"name": John Doe}', '{"name": "John Doe"}'),
    # Missing opening braces or bracket
    ('"name": "John Doe"}', '{"name": "John Doe"}'),
    ("1, 2, 3]", "[1, 2, 3]"),
]


@pytest.fixture(params=_JSON_FIXABLE)
def fixable_json(request: pytest.FixtureRequest) -> tuple[str, str]:
    return request.param


@pytest.fixture(params=_JSON_UNFIXABLE)
def unfixable_json(request: pytest.FixtureRequest) -> tuple[str, str]:
    return request.param


def test_json_loads_fixable(fixable_json: tuple[str, str]):
    assert json_loads(fixable_json[0]) == json.loads(fixable_json[1])


def test_json_loads_unfixable(unfixable_json: tuple[str, str]):
    assert json_loads(unfixable_json[0]) != json.loads(unfixable_json[1])