Back to Repositories

Testing Storage Performance for Multi-Domain Message Processing in CAT

A comprehensive performance test suite for storage operations in the CAT monitoring system, focusing on message storage and retrieval capabilities across multiple domains and IP addresses.

Test Coverage Overview

This test suite evaluates the performance of message storage operations in a multi-domain, multi-IP environment.

Key areas covered include:

Message bucket management for large-scale storage
Performance testing across 300 domains and 10 IPs
Message tree storage and retrieval operations
Storage configuration and initialization testing

Implementation Analysis

The test implementation utilizes JUnit framework with ComponentTestCase for dependency injection and component testing. The approach focuses on measuring write performance for message storage across different domain-IP combinations.

Technical patterns include:

Message codec implementation using PlainTextMessageCodec
File-based storage management with configured base directories
Systematic performance measurement with QPS calculations

Technical Details

Testing tools and configuration:

JUnit test framework
Custom MessageCodec implementation
StorageConfiguration for base directory setup
MessageBucketManager for local storage operations
File system operations via Files.forDir()
Performance metrics tracking with System.currentTimeMillis()

Best Practices Demonstrated

The test suite exemplifies several testing best practices for performance evaluation.

Notable practices include:

Proper test initialization and cleanup in @Before methods
Systematic performance measurement and reporting
Comprehensive domain and IP coverage
Proper resource management and cleanup
Clear separation of setup and test execution

dianping/cat

cat-consumer/src/test/java/com/dianping/cat/consumer/dump/StoragePerformanceTest.java

            
/*
 * Copyright (c) 2011-2018, Meituan Dianping. All Rights Reserved.
 *
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.dianping.cat.consumer.dump;

import java.io.File;

import org.junit.Before;
import org.junit.Test;
import org.unidal.cat.message.storage.StorageConfiguration;
import org.unidal.helper.Files;
import org.unidal.lookup.ComponentTestCase;

import com.dianping.cat.Cat;
import com.dianping.cat.message.codec.PlainTextMessageCodec;
import com.dianping.cat.message.spi.DefaultMessageTree;
import com.dianping.cat.message.spi.MessageCodec;
import com.dianping.cat.message.spi.MessageTree;
import com.dianping.cat.message.storage.MessageBucketManager;
import com.dianping.cat.message.tree.MessageId;

public class StoragePerformanceTest extends ComponentTestCase {
	private MessageCodec m_codec = new PlainTextMessageCodec();

	@Before
	public void before() {
		File baseDir = new File(Cat.getCatHome(),"bucket/dump/20160415");

		Files.forDir().delete(new File(baseDir, "dump"), true);

		lookup(StorageConfiguration.class).setBaseDataDir(baseDir);
	}

	@Test
	public void testManyDomainIpWrite() throws Exception {
		TreeHelper.init(m_codec);
		long start = System.currentTimeMillis();
		MessageBucketManager manager = lookup(MessageBucketManager.class, "local");
		int hour = 405746;

		for (int i = 0; i < 10000; i++) {
			long interStart = System.currentTimeMillis();

			for (int domainIndex = 0; domainIndex < 300; domainIndex++) {
				String domain = "domain" + domainIndex;

				for (int ipIndex = 0; ipIndex < 10; ipIndex++) {
					String ip = "0a01020" + ipIndex;
					MessageId id = new MessageId(domain, ip, hour, i * 10 + ipIndex);
					MessageTree tree = TreeHelper.cacheTree(m_codec, id);

					((DefaultMessageTree) tree).setMessageId(id.toString());
					tree.setFormatMessageId(id);

					manager.storeMessage(tree, id);
				}
			}

			long duration = System.currentTimeMillis() - interStart;

			if (i % 100 == 0) {
				System.out.println("duration:" + duration + ":qps:" + 3000 * 1000 / duration);
			}
		}

		long duration = System.currentTimeMillis() - start;
		System.out.println("write cost" + duration);
	}
}