Back to Repositories

Validating JSON Schema and Site List Implementation in sherlock-project

This test suite validates the integrity and functionality of Sherlock’s data manifest against both local and remote JSON schemas, ensuring consistent data structure and client compatibility. The tests verify schema validation, error handling, and site list functionality.

Test Coverage Overview

The test suite provides comprehensive coverage of manifest validation and site list functionality.

Key areas tested include:

Local schema validation for development changes
Remote schema validation for client compatibility
Site list error type verification
Parametrized testing of different site configurations

Implementation Analysis

The testing approach utilizes pytest’s powerful features for structured test organization and parameterization. The implementation employs JSON schema validation with both local and remote schemas, using the jsonschema library for validation operations.

Notable patterns include:

Path resolution for test resources
Fixture utilization for shared resources
Parameterized test cases for multiple scenarios

Technical Details

Testing tools and configuration:

pytest framework for test execution
jsonschema library for schema validation
Custom markers for online tests
Relative path handling for resource files
JSON file parsing and validation

Best Practices Demonstrated

The test suite exemplifies several testing best practices, including separation of concerns between local and remote validation scenarios.

Quality aspects include:

Clear test function naming and documentation
Efficient resource handling with context managers
Isolated test cases with specific assertions
Structured parameterization for multiple test cases

sherlock-project/sherlock

tests/test_manifest.py

            
import os
import json
import pytest
from jsonschema import validate

def test_validate_manifest_against_local_schema():
    """Ensures that the manifest matches the local schema, for situations where the schema is being changed."""
    json_relative: str = '../sherlock_project/resources/data.json'
    schema_relative: str = '../sherlock_project/resources/data.schema.json'
    
    json_path: str = os.path.join(os.path.dirname(__file__), json_relative)
    schema_path: str = os.path.join(os.path.dirname(__file__), schema_relative)

    with open(json_path, 'r') as f:
        jsondat = json.load(f)
    with open(schema_path, 'r') as f:
        schemadat = json.load(f)

    validate(instance=jsondat, schema=schemadat)


@pytest.mark.online
def test_validate_manifest_against_remote_schema(remote_schema):
    """Ensures that the manifest matches the remote schema, so as to not unexpectedly break clients."""
    json_relative: str = '../sherlock_project/resources/data.json'
    json_path: str = os.path.join(os.path.dirname(__file__), json_relative)

    with open(json_path, 'r') as f:
        jsondat = json.load(f)

    validate(instance=jsondat, schema=remote_schema)

# Ensure that the expected values are beind returned by the site list
@pytest.mark.parametrize("target_name,target_expected_err_type", [
    ('GitHub', 'status_code'),
    ('GitLab', 'message'),
])
def test_site_list_iterability (sites_info, target_name, target_expected_err_type):
    assert sites_info[target_name]['errorType'] == target_expected_err_type