Back to Repositories

Testing Browser Runtime Integration in OpenHands

This test suite validates the browsing functionality of the EventStreamRuntime component in OpenHands, specifically focusing on the interaction between the runtime and ActionExecutor in a sandbox environment. The tests verify URL browsing capabilities and command execution while ensuring proper browser output observations.

Test Coverage Overview

The test suite provides comprehensive coverage of browser-related functionality in the EventStreamRuntime environment.

Key areas tested include:

HTTP server setup and validation
URL browsing actions and responses
Command execution and output handling
Browser output observation verification

Edge cases covered include server startup timing and cleanup operations. Integration points focus on the interaction between BrowseURLAction and CmdRunAction components.

Implementation Analysis

The testing approach implements a systematic verification of browsing capabilities using a local HTTP server as a controlled test environment. The implementation utilizes Python’s built-in HTTP server and combines command execution with browser actions to create a complete test scenario.

Technical patterns include:

Runtime initialization and cleanup patterns
Action-observation pattern for browser interactions
Asynchronous command execution handling
Browser state verification

Technical Details

Testing tools and components:

Python’s built-in http.server module
EventStreamRuntime for action execution
BrowseURLAction and CmdRunAction classes
BrowserOutputObservation for response validation
Custom logger implementation

Configuration includes sandbox environment setup and micromamba runtime configuration.

Best Practices Demonstrated

The test implementation showcases several testing best practices for browser-based functionality verification.

Notable practices include:

Proper test setup and teardown procedures
Explicit assertion checking for all operation outcomes
Comprehensive state verification
Clean resource management
Isolated test environment creation

The code organization follows a clear structure with setup, execution, verification, and cleanup phases.

all-hands-ai/openhands

tests/runtime/test_browsing.py

            
"""Browsing-related tests for the EventStreamRuntime, which connects to the ActionExecutor running in the sandbox."""

from conftest import _close_test_runtime, _load_runtime

from openhands.core.logger import openhands_logger as logger
from openhands.events.action import (
    BrowseURLAction,
    CmdRunAction,
)
from openhands.events.observation import (
    BrowserOutputObservation,
    CmdOutputObservation,
)

# ============================================================================================================================
# Browsing tests, without evaluation (poetry install --without evaluation)
# For eval environments, tests need to run with poetry install
# ============================================================================================================================

PY3_FOR_TESTING = '/openhands/micromamba/bin/micromamba run -n openhands python3'


def test_simple_browse(temp_dir, runtime_cls, run_as_openhands):
    runtime = _load_runtime(temp_dir, runtime_cls, run_as_openhands)

    # Test browse
    action_cmd = CmdRunAction(
        command=f'{PY3_FOR_TESTING} -m http.server 8000 > server.log 2>&1 &'
    )
    logger.info(action_cmd, extra={'msg_type': 'ACTION'})
    obs = runtime.run_action(action_cmd)
    logger.info(obs, extra={'msg_type': 'OBSERVATION'})

    assert isinstance(obs, CmdOutputObservation)
    assert obs.exit_code == 0
    assert '[1]' in obs.content

    action_cmd = CmdRunAction(command='sleep 3 && cat server.log')
    logger.info(action_cmd, extra={'msg_type': 'ACTION'})
    obs = runtime.run_action(action_cmd)
    logger.info(obs, extra={'msg_type': 'OBSERVATION'})
    assert obs.exit_code == 0

    action_browse = BrowseURLAction(url='http://localhost:8000')
    logger.info(action_browse, extra={'msg_type': 'ACTION'})
    obs = runtime.run_action(action_browse)
    logger.info(obs, extra={'msg_type': 'OBSERVATION'})

    assert isinstance(obs, BrowserOutputObservation)
    assert 'http://localhost:8000' in obs.url
    assert not obs.error
    assert obs.open_pages_urls == ['http://localhost:8000/']
    assert obs.active_page_index == 0
    assert obs.last_browser_action == 'goto("http://localhost:8000")'
    assert obs.last_browser_action_error == ''
    assert 'Directory listing for /' in obs.content
    assert 'server.log' in obs.content

    # clean up
    action = CmdRunAction(command='rm -rf server.log')
    logger.info(action, extra={'msg_type': 'ACTION'})
    obs = runtime.run_action(action)
    logger.info(obs, extra={'msg_type': 'OBSERVATION'})
    assert obs.exit_code == 0

    _close_test_runtime(runtime)