Back to Repositories

Testing ClickHouse Batch Synchronization Operations in Canal

This test suite validates the ClickHouse adapter synchronization functionality in the Canal project, focusing on basic database operations like insert, update, delete and truncate operations. The tests ensure proper data synchronization between source databases and ClickHouse.

Test Coverage Overview

The test suite provides comprehensive coverage of core database operations through the ClickHouse adapter.

Tests INSERT operations with various data types including long, string and date fields
Validates UPDATE functionality with before/after state comparison
Covers DELETE operations and data removal
Tests TRUNCATE operations with state verification
Handles destination routing and timestamps for all operations

Implementation Analysis

The testing approach utilizes JUnit framework with individual test methods for each database operation type.

The implementation uses a common initialization pattern through @Before setup, creating isolated test scenarios. The ClickHouseAdapter is tested using the Dml (Data Manipulation Language) class to simulate database events with proper data structures and timing controls.

Technical Details

JUnit 4 testing framework
Custom ClickHouseAdapter implementation
Dml class for database operation simulation
Thread.sleep() for async operation handling
LinkedHashMap for ordered data storage
Collections utility for list operations

Best Practices Demonstrated

The test suite demonstrates several testing best practices:

Proper test isolation through @Before setup
Comprehensive coverage of all CRUD operations
Structured test data preparation
Async operation handling
Clear test method naming conventions
Modular test case organization

alibaba/canal

client-adapter/clickhouse/src/test/java/com/alibaba/otter/canal/client/adapter/clickhouse/ClickHouseBatchSyncServiceTest.java

            
package com.alibaba.otter.canal.client.adapter.clickhouse;

import java.util.*;

import org.junit.Before;
import org.junit.Ignore;
import org.junit.Test;

import com.alibaba.otter.canal.client.adapter.clickhouse.sync.Common;
import com.alibaba.otter.canal.client.adapter.support.Dml;

@Ignore
public class ClickHouseBatchSyncServiceTest {

    private ClickHouseAdapter clickHouseAdapter;

    @Before
    public void init() {
        clickHouseAdapter = Common.init();
    }

    @Test
    public void insert() throws InterruptedException {
        Dml dml = new Dml();
        dml.setDestination("example");
        dml.setTs(new Date().getTime());
        dml.setType("INSERT");
        dml.setDatabase("mytest");
        dml.setTable("user");
        List<Map<String, Object>> dataList = new ArrayList<>();
        Map<String, Object> data = new LinkedHashMap<>();
        dataList.add(data);
        data.put("id", 1L);
        data.put("name", "Eric");
        data.put("role_id", 1L);
        data.put("c_time", new Date());
        data.put("test1", "sdfasdfawe中国asfwef");
        dml.setData(dataList);

        clickHouseAdapter.sync(Collections.singletonList(dml));
        Thread.sleep(10000L);
    }

    @Test
    public void update() {
        Dml dml = new Dml();
        dml.setDestination("example");
        dml.setTs(new Date().getTime());
        dml.setType("UPDATE");
        dml.setDatabase("mytest");
        dml.setTable("user");
        List<Map<String, Object>> dataList = new ArrayList<>();
        Map<String, Object> data = new LinkedHashMap<>();
        dataList.add(data);
        data.put("id", 1L);
        data.put("name", "Eric2");
        dml.setData(dataList);
        List<Map<String, Object>> oldList = new ArrayList<>();
        Map<String, Object> old = new LinkedHashMap<>();
        oldList.add(old);
        old.put("name", "Eric");
        dml.setOld(oldList);

        clickHouseAdapter.sync(Collections.singletonList(dml));
    }

    @Test
    public void delete() {
        Dml dml = new Dml();
        dml.setDestination("example");
        dml.setTs(new Date().getTime());
        dml.setType("DELETE");
        dml.setDatabase("mytest");
        dml.setTable("user");
        List<Map<String, Object>> dataList = new ArrayList<>();
        Map<String, Object> data = new LinkedHashMap<>();
        dataList.add(data);
        data.put("id", 1L);
        data.put("name", "Eric2");
        dml.setData(dataList);

        clickHouseAdapter.sync(Collections.singletonList(dml));
    }

    @Test
    public void truncate() throws InterruptedException {
        Dml dml = new Dml();
        dml.setDestination("example");
        dml.setTs(new Date().getTime());
        dml.setType("TRUNCATE");
        dml.setDatabase("mytest");
        dml.setTable("user");
        List<Map<String, Object>> dataList = new ArrayList<>();
        Map<String, Object> data = new LinkedHashMap<>();
        dataList.add(data);
        data.put("id", 1L);
        data.put("name", "Eric2");
        dml.setData(dataList);
        List<Map<String, Object>> oldList = new ArrayList<>();
        Map<String, Object> old = new LinkedHashMap<>();
        oldList.add(old);
        old.put("name", "Eric");
        dml.setOld(oldList);

        clickHouseAdapter.sync(Collections.singletonList(dml));
        Thread.sleep(1000L);

    }
}