基于流式处理架构的日志采集系统的设计与实现

Design and Implementation of Log Collection System Based on Stream Processing Architecture

摘要:

对信息系统运行记录、操作日志、告警信息的采集问题进行了研究，提出了一种面向泛政府行业安全运行管理平台的统一日志采集系统。采用基于消息队列的流式处理架构，实现日志采集、日志处理、日志上报等各个环节的解耦；采用标准化接口和插件技术，实现各种异构日志信息的采集和数据上报；采用消息队列的流量削峰技术，保证日志传输的安全可靠；依据日志流量特征，提出一种支持动态调整消费组的设计模式，满足系统的高性能要求。整个系统由日志采集、数据上报、数据管理、系统管理、策略管理、Agent管理、日志源管理模块和日志采集代理(Agent)子系统组成，可满足对各类安全数据的集中分析、安全威胁感知和智能研判。

Abstract:

This paper studies the collection of operation record, operation log and alarm information of information system, and puts forward a unified log collection system for pan-government industry safety operation management platform. The flow processing architecture based on message queue is adopted to realize the decoupling of log collection, log processing and log reporting. Standardized interfaces and plug-ins are used to collect heterogeneous log information and data. The traffic peak clipping technology of message queue is adopted to ensure the safety and reliability of log transmission. According to the characteristics of log traffic, a design mode supporting dynamic adjustment of consumption group is proposed to meet the high performance requirements of the system. The whole system is composed of log collection, data reporting, data management, system management, policy management, Agent management, log source management module and log collection subsystem, which can meet the centralized analysis, security threat perception and intelligent analysis of various security data.