WO2024109569A1 - 数据处理方法及装置、电子设备及计算机可读存储介质 - Google Patents

数据处理方法及装置、电子设备及计算机可读存储介质 Download PDF

Info

Publication number
WO2024109569A1
WO2024109569A1 PCT/CN2023/131164 CN2023131164W WO2024109569A1 WO 2024109569 A1 WO2024109569 A1 WO 2024109569A1 CN 2023131164 W CN2023131164 W CN 2023131164W WO 2024109569 A1 WO2024109569 A1 WO 2024109569A1
Authority
WO
WIPO (PCT)
Prior art keywords
log
data
network element
log data
client
Prior art date
Application number
PCT/CN2023/131164
Other languages
English (en)
French (fr)
Inventor
周煜
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2024109569A1 publication Critical patent/WO2024109569A1/zh

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0803—Configuration setting
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/04—Processing captured monitoring data, e.g. for logfile generation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/2866—Architectures; Arrangements
    • H04L67/30—Profiles
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the present application relates to the field of communications, and in particular to a data processing method, a data processing device, an address device, and a computer-readable storage medium.
  • the business system in the device will generate a large amount of service log data when providing services, as well as the operating status log data of the corresponding business.
  • the operating system will also generate system log data.
  • the related ELK log data analysis system uses filebeat, logstash and other collection tools on the client to collect log data, and then simply filters all log data and enters them all into the elasticsearch (distributed search engine) on the server. Then, the Kibana data visualization system is used to perform statistical display of the data in elasticsearch.
  • the present application provides a data processing method, which is applied to a server, comprising: obtaining device registration information of a client, wherein the device registration information includes network element information of a network element deployed on the client; obtaining a log collection configuration file corresponding to the network element information, wherein the log collection configuration file includes a log reduction method, and the log reduction method is used to reduce the amount of log data of the network element; and pushing the log collection configuration file to the client.
  • a configuration file and a log collection tool are set, wherein the log collection tool is used to collect the log data of the network element, and process the log data according to the log reduction method to obtain reduced log data; and obtain the reduced log data reported by the client.
  • the present application provides a data processing method, which is applied to a client, comprising: receiving a log collection configuration file and a log collection tool from a server, the log collection configuration file comprising a log reduction method, and the log reduction method is used to reduce the amount of log data of a network element deployed on the client; the log collection tool is used to collect the log data and process the log data according to the log reduction method; the log data is collected by the log collection tool, and the collected log data of the network element is processed according to the log reduction method to obtain reduced log data; and the reduced log data is transmitted to the server.
  • the present application provides a data processing device, applied to a server, comprising: a first acquisition module, configured to acquire device registration information of a client, the device registration information including network element information of a network element deployed on the client; a second acquisition module, configured to acquire a log collection configuration file corresponding to the network element information, the log collection configuration file including a log reduction method, the log reduction method being used to reduce the amount of log data of the network element; a push module, configured to push the log collection configuration file and a log collection tool to the client, the log collection tool being used to collect the log data of the network element, and processing the log data according to the log reduction method to obtain reduced log data; and a third acquisition module, configured to acquire the reduced log data reported by the client.
  • the present application provides a data processing device, applied to a client, comprising: a receiving module, configured to receive a log collection configuration file and a log collection tool from a server, the log collection configuration file comprising a log reduction method, and the log reduction method is used to reduce the amount of log data of a network element deployed on the client; the log collection tool is configured to collect the log data and process the log data according to the log reduction method; a collection and processing module, configured to collect the log data through the log collection tool, and process the collected log data of the network element according to the log reduction method to obtain reduced log data; and a transmission module, configured to transmit the reduced log data to the server.
  • the present application provides an electronic device, comprising: a processor, a memory and a communication bus, the processor and the memory communicate with each other through the communication bus; the memory is configured to store a computer program; the processor is configured to execute the computer program stored in the memory to implement the data processing method described in the first aspect or the data processing method described in the second aspect.
  • the present application provides a computer-readable storage medium storing a computer program.
  • the processor implements the data processing method described in the first aspect or the data processing method described in the second aspect.
  • FIG1 is a schematic diagram of a flow chart of a data processing method in an embodiment of the present application.
  • FIG2 is a schematic diagram of client resource registration and server configuration delivery in an embodiment of the present application.
  • FIG3 is another flow chart of a data processing method in an embodiment of the present application.
  • FIG4 is a schematic diagram of a log processing flow in an embodiment of the present application.
  • FIG5 is a schematic diagram of the structure of a data processing system in an embodiment of the present application.
  • FIG6 is a schematic diagram of a structure of a data processing device in an embodiment of the present application.
  • FIG7 is another structural diagram of a data processing device in an embodiment of the present application.
  • FIG8 is a schematic diagram of the structure of an electronic device in an embodiment of the present application.
  • An embodiment of the present application provides a data processing method, which can be applied to a server. As shown in FIG1 , the data processing method may include the following steps 101 to 104 .
  • Step 101 Acquire device registration information of a client, where the device registration information includes network element information of a network element deployed at the client.
  • the device registration information includes auxiliary information such as the client IP address and network element information that can help quickly locate the device.
  • the network element information includes but is not limited to the identification of the network element.
  • Step 102 Obtain a log collection configuration file corresponding to the network element information.
  • the log collection configuration file includes a log reduction method, and the log reduction method is used to reduce the data volume of the log data of the network element.
  • the client can include multiple network element log types, such as operating system message logs, network element A service logs, network element B service logs, network element C system operation status logs, etc.
  • network element log types such as operating system message logs, network element A service logs, network element B service logs, network element C system operation status logs, etc.
  • a log collection configuration file corresponding to the network element information can be generated in real time; of course, in order to improve efficiency, a log collection configuration file corresponding to different network element information can also be generated in advance, and then a certain network element information can be obtained by querying a SQL statement.
  • the log configuration file corresponding to the information can be generated in real time; of course, in order to improve efficiency, a log collection configuration file corresponding to different network element information can also be generated in advance, and then a certain network element information can be obtained by querying a SQL statement.
  • obtaining a log configuration file corresponding to the network element information may include: obtaining the format type of the log data of the network element based on the network element information; obtaining a target log reduction method corresponding to the format type; and generating a log collection configuration file based on the target log reduction method.
  • the format types of log data include structured logs and unstructured logs.
  • Structured logs refer to logs in which fields are separated by specified delimiters or stored in the form of key-value pairs, and the meaning of each field is clear and fixed.
  • time series data is a data column recorded in chronological order for the same indicator, and there are multiple indicator types such as instantaneous value gague type and cumulative counter type.
  • the log format is "time
  • This type of log has the characteristics of strong extractability and high repeatability.
  • the limited fields can be aggregated as fixed tags.
  • Time series data B is similar to this and will not be described in detail.
  • log data in the structured log in which the field values of some fields in the log data do not change much within a period of time.
  • Some fields can be aggregated according to the preset time window, and their field values within the time window can be aggregated, such as averaging, summing, and finding the maximum value.
  • the log format is "time
  • time series data is generated, at least 4 different indicators are required to store the content of a log, which is not conducive to data acquisition. Since the number of IP, user name, and response status code is limited, these three tags can be used as the minimum dimension for aggregation. If a 5-minute time window is set, the data of the 5-minute period will be cached starting from the natural time and will be rounded up to 5 minutes. When a log of a unique dimension appears in the time window, the information of the dimension will be recorded.
  • All other data of the same dimension within 5 minutes can be calculated and processed according to the rules, such as summing, finding the maximum value, counting, etc. Finally, only one piece of data will be retained in the same time window and entered into the server, such as: ⁇ IP: A, username: A, responsecode: 200, avg_responsetime: 20ms, max_downloadrate: 3Mbps, avg_size: 200KB, sum_volume: 300MB ⁇ . If the minimum dimension generates 50 log records in the time window, this method will aggregate these 50 records into one record.
  • logs without a fixed format are called unstructured logs.
  • Such logs are collected by matching and extracting keywords. For example, there are many logs in the operating system log, and it is necessary to extract the logs of memory overflow.
  • the keyword is "oom (memory overflow)". Only the logs of key information need to be stored. Then, fuzzy matching can be performed based on the oom keyword, and the log lines that match the keyword are extracted.
  • Such logs can also be aggregated to reduce the amount of data.
  • a 5-minute time window is used to count the logs with the same type of keywords or print duplicate logs for merging to generate aggregated records. The aggregated records contain log information and the number of times the information appears in this 5-minute time window.
  • the server is pre-set with the format types of log data of different network elements and the correspondence between different format types and log reduction methods. Therefore, after obtaining the network element information from the client, the server can obtain the format type of the log data of the network element and the target log reduction method matching the format type through query.
  • different log configuration files can also be generated in advance for different network elements, and a mapping relationship between network element information and log configuration files can be established. Therefore, when obtaining a log configuration file corresponding to the network element information, the network element information can be used to directly query the mapping file. The mapping relationship is used to obtain the matching log configuration file.
  • the log configuration file includes not only the log reduction method but also a log path, which represents the storage path of the log data.
  • FIG2 a schematic diagram of client resource registration and server configuration delivery is provided in FIG2 .
  • Step 103 Push the log collection configuration file and the log collection tool to the client.
  • the log collection tool is used to collect the log data of the network element and process the log data according to the log reduction method to obtain the reduced log data.
  • Step 104 Obtain the reduced log data reported by the client.
  • the device registration information reported by the client may also include at least one of the node information to which the client belongs and the geographical location information of the client.
  • the server may summarize and aggregate the reduced log data reported by different clients using at least one of the node information and the geographic location information as an aggregation dimension.
  • the server uses log processing tools to reprocess the text data stored in the Kafka queue.
  • the log processing tool has similar functions to the client log collection tool. It can perform secondary aggregation on the data, and aggregate the logs collected from large-scale devices in higher dimensions, such as nodes and cities. It also generates different indexes for the data according to the network elements to which it belongs and enters them into elasticsearch (distributed search engine).
  • the time series data in Prometheus can also be processed by dimensionality reduction based on the original data to generate new dimensionality reduction indicators. For example, in a certain indicator of the log information collected by 1,000 devices in a unit time, there are 1,000 time series data. The 1,000 devices belong to 3 cities. Then, the server aggregates the 1,000 time series data according to the city dimension to form a new city-level monitoring indicator. The new indicator only contains 3 new time series data, which will have higher retrieval efficiency when troubleshooting and locating problems during operation and maintenance.
  • the present application embodiment can also use open source visualization
  • the data display component grafana displays the reduced log data.
  • the device registration information of the client is obtained, and the device registration information includes the network element information of the network element deployed on the client; the log collection configuration file corresponding to the network element information is obtained, and the log collection configuration file includes a log reduction method, and the log reduction method is used to reduce the data volume of the log data of the network element; the log collection configuration file and the log collection tool are pushed to the client, and the log collection tool is used to collect the log data of the network element, and the log data is processed according to the log reduction method to obtain the reduced log data; and the reduced log data reported by the client is obtained.
  • the log collection tool pushed to the client can process the log data according to the log reduction method configured in the log collection configuration file, the data volume of the reduced log data reported by the client to the server can be greatly reduced compared to the data volume of the simply filtered log data, thereby improving the efficiency of the data processing of the server.
  • An embodiment of the present application provides a data processing method, which can be applied to a client.
  • the data processing method may include the following steps 301 to 303 .
  • Step 301 Receive a log collection configuration file and a log collection tool from a server.
  • the log collection configuration file includes a log reduction method, which is used to reduce the amount of log data of a network element deployed on a client.
  • the log collection tool is used to collect log data and process the log data according to the log reduction method.
  • Step 302 collect log data through a log collection tool, and process the collected log data of the network element according to a log reduction method to obtain reduced log data.
  • Step 303 Transmit the reduced log data to the server.
  • the log reduction method is used to collect the logs of the network elements.
  • the log data is processed to obtain reduced log data (i.e., step 302), including at least one of the following: converting the log data of the network element into time series data, and using the time series data as the reduced log data; the same indicator in the time series data is a data column recorded in chronological order, and there are at least two times in the data column with different indicator values; the log data of the network element is aggregated according to a preset time window to obtain aggregated log data, and the aggregated log data is used as the reduced log data; the log data of the network element is filtered using preset keywords to obtain filtered log data, and the filtered log data is used as the reduced log data.
  • converting the log data of the network element into time series data includes: obtaining a first indicator in the log data whose indicator value does not change with time and a second indicator whose indicator value changes with time; collecting a first indicator value of the first indicator and a second indicator value of the second indicator at different times; generating time series data based on the first indicator, the first indicator value, the second indicator and the second indicator value.
  • the aggregating and processing the log data of the network element according to a preset time window to obtain aggregated log data includes: obtaining a dimensional aggregation indicator corresponding to the network element; and obtaining target log data corresponding to the preset time window from the log data of the network element; performing aggregation processing on the indicator values belonging to the dimensional aggregation indicators in the target log data to obtain aggregated processing indicator values; generating aggregated log data based on the dimensional aggregation indicators, the aggregation processing indicator values, the non-dimensional aggregation indicators and the non-dimensional aggregation indicator values; the dimensional aggregation indicators correspond to the aggregation processing indicator values, the non-dimensional aggregation indicators correspond to the non-dimensional aggregation indicator values, and the non-dimensional aggregation indicators are indicators in the log data other than the dimensional aggregation indicators.
  • the transmission of the reduced log data to the server includes: if the reduced log data belongs to a text log, sending the reduced log data to the Kafka queue of the server; if the reduced log data belongs to time series data, in response to a request for obtaining the time series database of the client, sending the reduced log data to the time series database.
  • the reduced log data corresponding to the text log of the client is sent to the Kafka queue (kafka) of the server. If time series data is generated, the client exposes the interface, and the server prometheus (time series database) makes regular requests for acquisition.
  • FIG4 For ease of understanding, a schematic diagram of the log processing flow in FIG4 is provided.
  • the present application provides a data processing system.
  • the data processing system mainly includes: a server 501 and a client 502.
  • the server 501 is configured to obtain device registration information of the client 502, the device registration information includes network element information of the network element deployed on the client 502; obtain a log collection configuration file corresponding to the network element information, the log collection configuration file includes a log reduction method, and the log reduction method is used to reduce the amount of log data of the network element; push the log collection configuration file and the log collection tool to the client 502, the log collection tool is used to collect the log data of the network element, and process the log data according to the log reduction method to obtain reduced log data; and obtain the reduced log data reported by the client 502.
  • the client 502 is configured to receive a log collection configuration file and a log collection tool from the server 501, the log collection configuration file including a log reduction method, the log reduction method being used to reduce the amount of log data of a network element deployed on the client; the log collection tool being used to collect log data and process the log data in accordance with the log reduction method; collecting log data through the log collection tool, and processing the collected log data of the network element in accordance with the log reduction method to obtain reduced log data; and transmitting the reduced log data to the server 501.
  • the log collection configuration file including a log reduction method, the log reduction method being used to reduce the amount of log data of a network element deployed on the client; the log collection tool being used to collect log data and process the log data in accordance with the log reduction method; collecting log data through the log collection tool, and processing the collected log data of the network element in accordance with the log reduction method to obtain reduced log data; and transmitting the reduced log data to the server 501.
  • a data processing device is provided in an embodiment of the present application.
  • the data processing device mainly includes: a first acquisition module 601, a second acquisition module 602, a push module 603, and a third acquisition module 604.
  • the first acquisition module 601 is configured to acquire device registration information of the client, where the device registration information includes network element information of a network element deployed at the client.
  • the second acquisition module 602 is configured to acquire a log collection configuration file corresponding to the network element information, where the log collection configuration file includes a log reduction method, and the log reduction method is used to reduce the data volume of the log data of the network element.
  • the push module 603 is configured to push the log collection configuration file and the log collection tool to the client.
  • the log collection tool is used to collect the log data of the network element and process the log data according to the log reduction method to obtain the reduced log data.
  • the third acquisition module 604 is configured to acquire the reduced log data reported by the client.
  • the second acquisition module 602 is further configured to, based on the network element information, Acquire the format type of the log data of the network element; acquire the target log reduction method corresponding to the format type; and generate a log collection configuration file based on the target log reduction method.
  • the second acquisition module 602 is further configured to query the mapping relationship between the preset network element information and the log collection configuration file to obtain the target log collection configuration file corresponding to the network element information; and use the target log collection configuration file as the log collection configuration file corresponding to the network element information.
  • the device registration information further includes: at least one of: information about the node to which the client belongs and information about the geographical location of the client.
  • the data processing device is further configured to, after obtaining the reduced log data reported by the client, summarize and aggregate the reduced log data reported by different clients using at least one of the node information and the geographic location information as an aggregation dimension.
  • a data processing device is provided in an embodiment of the present application.
  • the device mainly includes a receiving module 701, a collection and processing module 702, and a transmission module 703.
  • the receiving module 701 is configured to receive a log collection configuration file and a log collection tool from the server.
  • the log collection configuration file includes a log reduction method, which is used to reduce the amount of log data of the network element deployed on the client; the log collection tool is used to collect log data and process the log data according to the log reduction method.
  • the collection and processing module 702 is configured to collect log data through a log collection tool, and process the collected log data of the network element according to a log reduction method to obtain reduced log data.
  • the transmission module 703 is configured to transmit the reduced log data to the server.
  • the collection and processing module 702 can also be configured to perform at least one of the following operations: converting the log data of the network element into time series data, and using the time series data as the reduced log data; the same indicator in the time series data is a data column recorded in chronological order, and there are at least two times in the data column with different indicator values; aggregating the log data of the network element according to a preset time window to obtain aggregated log data, and using the aggregated log data as the reduced log data; using preset keywords to filter the log data of the network element to obtain filtered log data, and using the filtered log data as the reduced log data.
  • Log data converting the log data of the network element into time series data, and using the time series data as the reduced log data; the same indicator in the time series data is a data column recorded in chronological order, and there are at least two times in the data column with different indicator values; aggregating the log data of the network element according to a preset time window to obtain aggregated log data, and using the aggregated log data as the reduced log data;
  • the electronic device mainly includes: a processor 801 , a memory 802 and a communication bus 808 .
  • the processor 801 and the memory 802 communicate with each other via the communication bus 808 .
  • the memory 802 stores a computer program that can be executed by the processor 801.
  • the processor 801 executes the computer program stored in the memory 802 to implement the following steps: obtaining device registration information of the client, the device registration information includes network element information of the network element deployed on the client; obtaining a log collection configuration file corresponding to the network element information, the log collection configuration file includes a log reduction method, and the log reduction method is used to reduce the data volume of the log data of the network element; pushing the log collection configuration file and the log collection tool to the client, the log collection tool is used to collect the log data of the network element, and process the log data according to the log reduction method to obtain reduced log data; obtaining the reduced log data reported by the client; or, receiving the log collection configuration file and the log collection tool from the server, the log collection configuration file includes a log reduction method, and the log reduction method is used to reduce the data volume of the log data of the network element deployed on the client; the log collection tool is used to collect log data and process the log data according to the log reduction method; collecting log data through the log collection tool, and processing the collected log data of the network element according to the log
  • the communication bus 808 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus.
  • PCI Peripheral Component Interconnect
  • EISA Extended Industry Standard Architecture
  • the communication bus 808 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in FIG8 , but it does not mean that there is only one bus or one type of bus.
  • the memory 802 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk storage.
  • RAM random access memory
  • non-volatile memory such as at least one disk storage.
  • the memory may also be at least one storage device located away from the processor 801.
  • the processor 801 may be a general-purpose processor, including a central processing unit (CPU), a network processor ( It can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • CPU central processing unit
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • a computer-readable storage medium in which a computer program is stored.
  • the computer program When the computer program is executed on a computer, the computer executes the above-mentioned data processing method.
  • the computer-readable storage medium it can be implemented in whole or in part by software, hardware, firmware or any combination thereof.
  • software it can be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part.
  • the computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device.
  • the computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction is transmitted from a website site, a computer, a server or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, microwave, etc.) mode to another website site, computer, server or data center.
  • the computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integration.
  • the available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (such as a DVD) or a semiconductor medium (such as a solid-state hard disk), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • Debugging And Monitoring (AREA)

Abstract

本申请涉及一种数据处理方法、一种数据处理装置、一种电子设备及一种计算机可读存储介质，该数据处理方法获取客户端的设备注册信息；获取与网元信息对应的日志采集配置文件，日志采集配置文件包括日志减少方式；向客户端推送日志采集配置文件和日志采集工具，日志采集工具用于采集网元的日志数据，并按照日志减少方式对日志数据进行处理得到减少后日志数据；获取客户端上报的减少后日志数据。

Description

数据处理方法及装置、电子设备及计算机可读存储介质
相关申请的交叉引用
本申请要求于2022年11月21日提交的中国专利申请NO.202211457759.3的优先权，该中国专利申请的内容通过引用的方式整体合并于此。
技术领域
本申请涉及通信领域，尤其涉及数据处理方法、数据处理装置、地址设备及计算机可读存储介质。
背景技术
设备中的业务系统在进行服务时会产生大量的服务日志数据，同时也会产生对应业务的运行状态日志数据，操作系统也会产生系统日志数据，相关的ELK日志数据分析系统是在客户端使用filebeat、logstash等采集工具采集到日志数据后，将所有日志数据仅进行简单过滤后全部录入服务端的elasticsearch(分布式搜索引擎)中，然后再使用Kibana数据可视化系统对elasticsearch中的数据进行统计展示。
虽然上述过程对日志数据进行了简单过滤，但是过滤后的日志数据的数据量仍然很大，可能会导致服务端处理数据的效率变低。
公开内容
第一方面，本申请提供一种数据处理方法，应用于服务端，包括：获取客户端的设备注册信息，所述设备注册信息包括部署于所述客户端的网元的网元信息；获取与所述网元信息对应的日志采集配置文件，所述日志采集配置文件包括日志减少方式，所述日志减少方式用于减少所述网元的日志数据的数据量；向所述客户端推送所日志采 集配置文件和日志采集工具，所述日志采集工具用于采集所述网元的日志数据，并按照所述日志减少方式对所述日志数据进行处理得到减少后日志数据；获取所述客户端上报的所述减少后日志数据。
第二方面，本申请提供一种数据处理方法，应用于客户端，包括：接收来自于服务端的日志采集配置文件和日志采集工具，所述日志采集配置文件包括日志减少方式，所述日志减少方式用于减少部署于所述客户端的网元的日志数据的数据量；所述日志采集工具用于采集所述日志数据并按照所述日志减少方式对所述日志数据进行处理；通过所述日志采集工具采集所述日志数据，并按照所述日志减少方式对采集的所述网元的日志数据进行处理，得到减少后日志数据；以及向所述服务端传输所述减少后日志数据。
第三方面，本申请提供一种数据处理装置，应用于服务端，包括：第一获取模块，配置为获取客户端的设备注册信息，所述设备注册信息包括部署于所述客户端的网元的网元信息；第二获取模块，配置为获取与所述网元信息对应的日志采集配置文件，所述日志采集配置文件包括日志减少方式，所述日志减少方式用于减少所述网元的日志数据的数据量；推送模块，配置为向所述客户端推送所日志采集配置文件和日志采集工具，所述日志采集工具用于采集所述网元的日志数据，并按照所述日志减少方式对所述日志数据进行处理得到减少后日志数据；以及第三获取模块，配置为获取所述客户端上报的所述减少后日志数据。
第四方面，本申请提供一种数据处理装置，应用于客户端，包括：接收模块，配置为接收来自于服务端的日志采集配置文件和日志采集工具，所述日志采集配置文件包括日志减少方式，所述日志减少方式用于减少部署于所述客户端的网元的日志数据的数据量；所述日志采集工具配置为采集所述日志数据并按照所述日志减少方式对所述日志数据进行处理；采集处理模块，配置为通过所述日志采集工具采集所述日志数据，并按照所述日志减少方式对采集的所述网元的日志数据进行处理，得到减少后日志数据；以及传输模块，配置为向所述服务端传输所述减少后日志数据。
第五方面，本申请提供一种电子设备，包括：处理器、存储器和通信总线，处理器和存储器通过通信总线完成相互间的通信；所述存储器，配置为存储计算机程序；所述处理器，配置为执行所述存储器中所存储的计算机程序，实现第一方面所述的数据处理方法或第二方面所述的数据处理方法。
第六方面，本申请提供一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时，使得所述处理器实现第一方面所述的数据处理方法或第二方面所述的数据处理方法。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本申请的实施例，并与说明书一起用于解释本申请的原理。
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本申请实施例中数据处理方法的一种流程示意图；
图2本申请实施例中客户端资源注册及服务端配置下发的示意图；
图3为本申请实施例中数据处理方法的又一种流程示意图；
图4为本申请实施例中日志处理流程的示意图；
图5为本申请实施例中数据处理系统的结构示意图；
图6为本申请实施例中数据处理装置的一种结构示意图；
图7为本申请实施例中数据处理装置的又一种结构示意图；以及
图8为本申请实施例中电子设备的结构示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、 完整地描述，显然，所描述的实施例是本申请的示例性实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动的前提下所获得的所有其他实施例，都属于本申请保护的范围。
需要说明的是，本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
本申请实施例提供一种数据处理方法，可应用于服务端，如图1所示，该数据处理方法可以包括以下步骤101至104。
步骤101、获取客户端的设备注册信息，设备注册信息包括部署于客户端的网元的网元信息。
在实际应用中，将需要采集日志的所有设备在服务端的资源管理中心中进行注册，设备注册信息包括客户端IP地址、网元信息等能帮助快速定位的辅助信息。
本申请实施例中，网元信息包括但不限于网元的标识。
步骤102、获取与网元信息对应的日志采集配置文件，日志采集配置文件包括日志减少方式，日志减少方式用于减少网元的日志数据的数据量。
在实际应用中，根据客户端部署的网元的不同，客户端可以包括多种网元日志类型，比如操作系统message日志、网元A服务日志、网元B服务日志、网元C系统运行状态日志等。
本申请实施例中，可以实时生成与网元信息对应的日志采集配置文件；当然为了提高效率，也可以预先生成与不同网元信息对应的日志采集配置文件，然后通过sql语句查询的方式获取某一个网元信 息对应的日志配置文件。
在一些实施方式中，所述获取与网元信息对应的日志配置文件(即，步骤102)可以包括：基于网元信息，获取网元的日志数据的格式类型；获取与格式类型对应的目标日志减少方式；以及基于目标日志减少方式生成日志采集配置文件。
本申请实施例中，日志数据的格式类型包括结构化日志和非结构化日志。结构化日志是指日志由指定的分隔符分割字段或者以key-value(键值)对的形式存储，每个字段代表的含义清晰固定的日志。
本申请实施例中，结构化日志中存在一类日志数据，这类日志数据的某些字段的字段值通常不会发生变化，只有一些字段的字段值可能会随着时间的改变而发生改变，对于这类日志数据，前者描述的字段(即，字段值通常不发生变化的字段)的字段值只需记录一次，无需反复记录并上报，而后者描述的字段(即，字段值可能会随着时间的改变而发生改变的字段)的字段值则需要正常向服务端上报，因此可以将这类日志数据转换为时序数据。时序数据是同一指标按时间顺序记录的数据列，且有瞬时值gague类型、累计counter类型等多种指标类型。
例如，日志的形式为“时间|IP|用户名|响应状态码(responsecode)|响应时长(responsetime)”或“time＝XXX，IP＝XXX，username＝XXX，responsecode＝XXX，responsetime＝XXX”，这类日志具有可提取性强、重复性高的特点，对于此类结构化日志中变化的字段较少的日志可以将有限的字段作为固定标签进行聚合，一般如IP、username等标签的数量是有限的，则可以将其生成{IP＝A，username＝A}、{IP＝B，username＝B}的时序数据A和数据B，即在时序数据A中，只记录一次IP＝A、username＝A，其它的字段(即时间time、响应状态码responsecode和响应时长responsetime)是随着时间变化而变化的。时序数据B与此类似，不再展开描述。
本申请实施例中，结构化日志中还存在一类日志数据，这类日志数据中存在某些字段的字段值在一段时间内变化幅度不大，对于这 些字段可以按照预设的时间窗，对其在时间窗内的字段值进行聚合处理，比如求均值、求和、求最大值等。
例如，日志的形式为“时间|IP|用户名|响应状态码|响应时长|下载速率|请求大小|服务流量|”，此时如果生成时序数据则需要最少4个不同指标来存储一条日志里的内容，不利于获取数据。由于IP、用户名、响应状态码的数量是有限的，则可以将这三个标签作为最小维度进行聚合。若设定5min的时间窗，则从自然时间开始按5min规整，将该5min的数据进行缓存，当该时间窗内出现唯一维度的日志时，记录该维度的信息，5min内所有相同维度的其他数据可以按照规则进行计算处理，如求和、求最大值、计数等，最终同一时间窗内只保留一条数据录入服务端，如：{IP:A，username:A，responsecode:200，avg_responsetime:20ms，max_downloadrate:3Mbps，avg_size:200KB，sum_volume:300MB}，若最小维度在时间窗内产生了50条日志记录，此方法则会将这50条记录聚合成一条记录。
本申请实施例中，对于无固定格式的日志称为非结构化日志，此类日志使用匹配提取关键字的方式采集，例如操作系统日志中的日志众多，需要提取内存溢出的日志，关键字为“oom(内存溢出)”，仅需要关键信息的日志进行存储，则可以根据oom关键词模糊匹配，匹配到该关键字的日志行才进行提取。此类日志也可以采用聚合的方式降低数据量，使用5min的时间窗，统计同一类关键字的日志或打印的重复日志进行归并，生成聚合的记录，聚合的记录包含日志信息和该信息在此5min时间窗内出现的次数。
应理解，服务端预先设置有不同网元的日志数据的格式类型以及不同的格式类型与日志减少方式的对应关系，因此服务端在获取来自客户端的网元信息后，可以通过查询的方式获取网元的日志数据的格式类型、以及与该格式类型匹配的目标日志减少方式。
本申请实施例中也可以预先为不同的网元生成不同的日志配置文件，并建立网元信息与日志配置文件之间的映射关系，因此在获取与网元信息对应的日志配置文件时，可以采用该网元信息直接查询映 射关系得到匹配的日志配置文件。
本申请实施例中，为了提高数据处理效率，日志配置文件中除了包括日志减少方式，还包括日志路径，该日志路径表征日志数据的存储路径。
为了方便理解，给出了图2的客户端资源注册及服务端配置下发的示意图。
步骤103、向客户端推送所日志采集配置文件和日志采集工具，日志采集工具用于采集网元的日志数据，并按照日志减少方式对日志数据进行处理得到减少后日志数据。
步骤104、获取客户端上报的减少后日志数据。
在实际应用中，为了方便服务端管理大规模设备，客户端上报的设备注册信息还可以包括客户端所归属的节点信息和客户端所处的地理位置信息中的至少一者。
相应地，服务端在获取客户端上报的减少后日志数据后，可以以节点信息和地理位置信息中的至少一者为聚合维度，对不同客户端上报的减少后日志数据进行汇总聚合。
服务端使用日志处理工具对存储在卡夫卡队列(kafka)中的文本数据进行再次处理，日志处理工具具有与客户端日志采集工具类似的功能，可以对数据进行二次聚合，将从大规模设备上采集上来的日志以更高的维度，例如节点、地市的维度将数据进行汇总聚合，并将数据根据所属网元生成不同的索引录入elasticsearch中(分布式搜索引擎)。
Prometheus(时序数据库)中的时序数据也能在原数据基础上进行降维处理，生成新的降维指标，例如在单位时间内由1000台设备采集上来的日志信息的某个指标中有1000条时序数据，1000台设备归属在3个地市下，则在服务端根据地市维度对该1000条时序数据进行聚合后形成新的地市级监控指标，新指标仅包含3条新时序数据，在运维排查定位问题时，会有更高的检索效率。
为了支持多种数据源接入，支持曲线图、表格、柱状图、饼图等多种数据可视化方式展示，本申请实施例还可以采用开源的可视化 数据展示组件grafana展示减少后日志数据。
通过在grafana配置好对应的elasticsearch和prometheus数据源地址，在同一个监控页面中可以同时添加基于prometheus的时序数据信息或绘图、elasticsearch的数据绘图、以及原始的日志详细信息查询。对于长周期的日志监控查询可以使用聚合后的数据或指标进行展示，由于聚合数据的数据量比原始日志数据小得多，查询性能会大幅提高，并且可以多维度、多网元地监控，从最高层级地市监控下钻到节点监控最终下钻到设备和网元级别，帮助运维人员快速定位故障。
本申请实施例提供的技术方案中，获取客户端的设备注册信息，设备注册信息包括部署于客户端的网元的网元信息；获取与网元信息对应的日志采集配置文件，日志采集配置文件包括日志减少方式，日志减少方式用于减少网元的日志数据的数据量；向客户端推送所日志采集配置文件和日志采集工具，日志采集工具用于采集网元的日志数据，并按照日志减少方式对日志数据进行处理得到减少后日志数据；以及获取客户端上报的减少后日志数据。由于向客户端推送的日志采集工具能够按照日志采集配置文件中配置的日志减少方式对日志数据进行处理，所以，可以使得客户端向服务端上报的减少后日志数据的数据量相比于简单过滤的日志数据的数据量大大减少，提高服务端数据处理的效率。
本申请实施例提供一种数据处理方法，可应用于客户端，如图3所示，该数据处理方法可以包括以下步骤301至303。
步骤301、接收来自于服务端的日志采集配置文件和日志采集工具，日志采集配置文件包括日志减少方式，日志减少方式用于减少部署于客户端的网元的日志数据的数据量；日志采集工具用于采集日志数据并按照日志减少方式对日志数据进行处理。
步骤302、通过日志采集工具采集日志数据，并按照日志减少方式对采集的网元的日志数据进行处理，得到减少后日志数据。
步骤303、向服务端传输减少后日志数据。
在一些实施方式中，所述按照日志减少方式对采集的网元的日 志数据进行处理，得到减少后日志数据(即，步骤302)包括以下至少一者：将网元的日志数据转化为时序数据，并将时序数据作为减少后日志数据；时序数据中的同一指标为按时间顺序记录的数据列，数据列中存在至少两个时间对应的指标值不同；按照预设的时间窗对网元的日志数据进行聚合处理，得到聚合日志数据，并将聚合日志数据作为减少后日志数据；采用预设的关键字对网元的日志数据进行筛选处理，得到筛选日志数据，并将筛选日志数据作为减少后日志数据。
在一些实施方式中，所述将网元的日志数据转化为时序数据包括：获取日志数据中指标值不随时间发生变化的第一指标和指标值随时间发生变化的第二指标；采集获得第一指标的第一指标值、以及不同时刻下第二指标的第二指标值；基于第一指标、第一指标值、第二指标和第二指标值，生成时序数据。
在一些实施方式中，所述按照预设的时间窗对网元的日志数据进行聚合处理，得到聚合日志数据包括：获取与网元对应的维度聚合指标；以及，从网元的日志数据中，获取与预设的时间窗对应的目标日志数据；对于目标日志数据中属于维度聚合指标的指标值进行聚合处理，得到聚合处理指标值；基于维度聚合指标、聚合处理指标值、非维度聚合指标以及非维度聚合指标值，生成聚合日志数据；维度聚合指标与聚合处理指标值对应，非维度聚合指标与非维度聚合指标值对应，非维度聚合指标为日志数据中除维度聚合指标之外的指标。
在一些实施方式中，所述向服务端传输减少后日志数据(即，步骤303)包括：若减少后日志数据属于文本日志，向服务端的卡夫卡队列发送减少后日志数据；若减少后日志数据属于时序数据，响应于客户端的时序数据库的获取请求，向时序数据库发送减少后日志数据。
本申请实施例中，客户端的文本日志对应的减少后日志数据发送至服务端的卡夫卡队列(kafka)，如果是产生时序数据，客户端暴露接口，由服务端prometheus(时序数据库)定时请求获取。
为了方便理解，给出了图4的日志处理流程的示意图。
本申请实施例中提供了一种数据处理系统，具体实施可参见上 述数据处理方法部分的描述，重复之处不再赘述，如图5所示，该数据处理系统主要包括：服务端501和客户端502。
服务端501配置为获取客户端502的设备注册信息，设备注册信息包括部署于客户端502的网元的网元信息；获取与网元信息对应的日志采集配置文件，日志采集配置文件包括日志减少方式，日志减少方式用于减少网元的日志数据的数据量；向客户端502推送所日志采集配置文件和日志采集工具，日志采集工具用于采集网元的日志数据，并按照日志减少方式对日志数据进行处理得到减少后日志数据；以及获取客户端502上报的减少后日志数据。
客户端502配置为接收来自于服务端501的日志采集配置文件和日志采集工具，日志采集配置文件包括日志减少方式，日志减少方式用于减少部署于客户端的网元的日志数据的数据量；日志采集工具用于采集日志数据并按照日志减少方式对日志数据进行处理；通过日志采集工具采集日志数据，并按照日志减少方式对采集的网元的日志数据进行处理，得到减少后日志数据；以及向服务端501传输减少后日志数据。
本申请实施例中提供了一种数据处理装置，具体实施可参见上述应用于服务端的数据处理方法部分的描述，重复之处不再赘述，如图6所示，该数据处理装置主要包括：第一获取模块601、第二获取模块602、推送模块603、第三获取模块604。
第一获取模块601配置为获取客户端的设备注册信息，设备注册信息包括部署于客户端的网元的网元信息。
第二获取模块602配置为获取与网元信息对应的日志采集配置文件，日志采集配置文件包括日志减少方式，日志减少方式用于减少网元的日志数据的数据量。
推送模块603配置为向客户端推送所日志采集配置文件和日志采集工具，日志采集工具用于采集网元的日志数据，并按照日志减少方式对日志数据进行处理得到减少后日志数据。
第三获取模块604配置为获取客户端上报的减少后日志数据。
在一些实施方式中，第二获取模块602还配置为基于网元信息， 获取网元的日志数据的格式类型；获取与格式类型对应的目标日志减少方式；以及基于目标日志减少方式生成日志采集配置文件。
在一些实施方式中，第二获取模块602还配置为查询预先设置的网元信息与日志采集配置文件的映射关系，得到与网元信息对应的目标日志采集配置文件；将目标日志采集配置文件，作为与网元信息对应的日志采集配置文件。
在一些实施方式中，设备注册信息还包括：客户端所属的节点信息和客户端所处的地理位置信息中的至少一者。
在一些实施方式中，该数据处理装置还配置为获取客户端上报的减少后日志数据之后，以节点信息和地理位置信息中的至少一者为聚合维度，对不同客户端上报的减少后日志数据进行汇总聚合。
本申请实施例中提供了一种数据处理装置，具体实施可参见上述应用于客户端的数据处理方法部分的描述，重复之处不再赘述，如图7所示，该装置主要包括接收模块701、采集处理模块702、传输模块703。
接收模块701配置为接收来自于服务端的日志采集配置文件和日志采集工具，日志采集配置文件包括日志减少方式，日志减少方式用于减少部署于客户端的网元的日志数据的数据量；日志采集工具用于采集日志数据并按照日志减少方式对日志数据进行处理。
采集处理模块702配置为通过日志采集工具采集日志数据，并按照日志减少方式对采集的网元的日志数据进行处理，得到减少后日志数据。
传输模块703配置为向服务端传输减少后日志数据。
在一些实施方式中，采集处理模块702还可以配置为执行以下操作中的至少一者：将网元的日志数据转化为时序数据，并将时序数据作为减少后日志数据；时序数据中的同一指标为按时间顺序记录的数据列，数据列中存在至少两个时间对应的指标值不同；按照预设的时间窗对网元的日志数据进行聚合处理，得到聚合日志数据，并将聚合日志数据作为减少后日志数据；采用预设的关键字对网元的日志数据进行筛选处理，得到筛选日志数据，并将筛选日志数据作为减少后 日志数据。
本申请实施例中还提供了一种电子设备，如图8所示，该电子设备主要包括：处理器801、存储器802和通信总线808，处理器801和存储器802通过通信总线808完成相互间的通信。存储器802中存储有可被处理器801执行的计算机程序，处理器801执行存储器802中存储的计算机程序，实现如下步骤：获取客户端的设备注册信息，设备注册信息包括部署于客户端的网元的网元信息；获取与网元信息对应的日志采集配置文件，日志采集配置文件包括日志减少方式，日志减少方式用于减少网元的日志数据的数据量；向客户端推送所日志采集配置文件和日志采集工具，日志采集工具用于采集网元的日志数据，并按照日志减少方式对日志数据进行处理得到减少后日志数据；获取客户端上报的减少后日志数据；或，接收来自于服务端的日志采集配置文件和日志采集工具，日志采集配置文件包括日志减少方式，日志减少方式用于减少部署于客户端的网元的日志数据的数据量；日志采集工具用于采集日志数据并按照日志减少方式对日志数据进行处理；通过日志采集工具采集日志数据，并按照日志减少方式对采集的网元的日志数据进行处理，得到减少后日志数据；向服务端传输减少后日志数据。
上述电子设备中提到的通信总线808可以是外设部件互连标准(Peripheral Component Interconnect，简称PCI)总线或扩展工业标准结构(Extended Industry Standard Architecture，简称EISA)总线等。该通信总线808可以分为地址总线、数据总线、控制总线等。为便于表示，图8中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
存储器802可以包括随机存取存储器(Random Access Memory，简称RAM)，也可以包括非易失性存储器(non-volatile memory)，例如至少一个磁盘存储器。存储器还可以是至少一个位于远离前述处理器801的存储装置。
上述的处理器801可以是通用处理器，包括中央处理器(Central Processing Unit，简称CPU)、网络处理器(Network Processor， 简称NP)等，还可以是数字信号处理器(Digital Signal Processing，简称DSP)、专用集成电路(Application Specific Integrated Circuit，简称ASIC)、现场可编程门阵列(Field-Programmable Gate Array，简称FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。
在本申请的实施例中，还提供了一种计算机可读存储介质，该计算机可读存储介质中存储有计算机程序，当该计算机程序在计算机上运行时，使得计算机执行上述的数据处理方法。
在上述计算机可读存储介质中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。该计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行该计算机指令时，全部或部分地产生按照本申请实施例所述的流程或功能。该计算机可以是通用计算机、专用计算机、计算机网络或者其他可编程装置。该计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，计算机指令从一个网站站点、计算机、服务器或者数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、微波等)方式向另外一个网站站点、计算机、服务器或数据中心进行传输。该计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。该可用介质可以是磁性介质(例如软盘、硬盘、磁带等)、光介质(例如DVD)或者半导体介质(例如固态硬盘)等。
需要说明的是，在本文中，诸如“第一”和“第二”等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。 在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上所述仅是本申请的具体实施方式，使本领域技术人员能够理解或实现本申请。对这些实施例的多种修改对本领域的技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本申请的精神或范围的情况下，在其它实施例中实现。因此，本申请将不会被限制于本文所示的这些实施例，而是要符合与本文所申请的原理和新颖特点相一致的最宽的范围。

Claims (10)

  1. 一种数据处理方法，应用于服务端，包括：
    获取客户端的设备注册信息，所述设备注册信息包括部署于所述客户端的网元的网元信息；
    获取与所述网元信息对应的日志采集配置文件，所述日志采集配置文件包括日志减少方式，所述日志减少方式用于减少所述网元的日志数据的数据量；
    向所述客户端推送所日志采集配置文件和日志采集工具，所述日志采集工具用于采集所述网元的日志数据，并按照所述日志减少方式对所述日志数据进行处理得到减少后日志数据；以及
    获取所述客户端上报的所述减少后日志数据。
  2. 根据权利要求1所述的方法，其中，所述获取与所述网元信息对应的日志采集配置文件包括：
    基于所述网元信息，获取所述网元的日志数据的格式类型；
    获取与所述格式类型对应的目标日志减少方式；以及
    基于所述目标日志减少方式生成所述日志采集配置文件。
  3. 根据权利要求1所述的方法，其中，所述获取与所述网元信息对应的日志采集配置文件包括：
    查询预先设置的网元信息与日志采集配置文件的映射关系，得到与所述网元信息对应的目标日志采集配置文件；以及
    将所述目标日志采集配置文件，作为与所述网元信息对应的日志采集配置文件。
  4. 根据权利要求1所述的方法，其中，所述设备注册信息还包括：所述客户端所属的节点信息和所述客户端所处的地理位置信息中的至少一者；
    所述获取所述客户端上报的所述减少后日志数据之后，所述方 法还包括：
    以所述节点信息和所述地理位置信息中的至少一者为聚合维度，对不同客户端上报的减少后日志数据进行汇总聚合。
  5. 一种数据处理方法，应用于客户端，包括：
    接收来自于服务端的日志采集配置文件和日志采集工具，所述日志采集配置文件包括日志减少方式，所述日志减少方式用于减少部署于所述客户端的网元的日志数据的数据量；所述日志采集工具用于采集所述日志数据并按照所述日志减少方式对所述日志数据进行处理；
    通过所述日志采集工具采集所述日志数据，并按照所述日志减少方式对采集的所述网元的日志数据进行处理，得到减少后日志数据；以及
    向所述服务端传输所述减少后日志数据。
  6. 根据权利要求5所述的方法，其中，所述按照所述日志减少方式对采集的所述网元的日志数据进行处理，得到减少后日志数据包括以下至少一者：
    将所述网元的日志数据转化为时序数据，并将所述时序数据作为所述减少后日志数据；所述时序数据中的同一指标为按时间顺序记录的数据列，所述数据列中存在至少两个时间对应的指标值不同；
    按照预设的时间窗对所述网元的日志数据进行聚合处理，得到聚合日志数据，并将所述聚合日志数据作为所述减少后日志数据；
    采用预设的关键字对所述网元的日志数据进行筛选处理，得到筛选日志数据，并将所述筛选日志数据作为所述减少后日志数据。
  7. 一种数据处理装置，应用于服务端，包括：
    第一获取模块，配置为获取客户端的设备注册信息，所述设备注册信息包括部署于所述客户端的网元的网元信息；
    第二获取模块，配置为获取与所述网元信息对应的日志采集配 置文件，所述日志采集配置文件包括日志减少方式，所述日志减少方式用于减少所述网元的日志数据的数据量；
    推送模块，配置为向所述客户端推送所日志采集配置文件和日志采集工具，所述日志采集工具用于采集所述网元的日志数据，并按照所述日志减少方式对所述日志数据进行处理得到减少后日志数据；
    第三获取模块，配置为获取所述客户端上报的所述减少后日志数据。
  8. 一种数据处理装置，应用于客户端，包括：
    接收模块，配置为接收来自于服务端的日志采集配置文件和日志采集工具，所述日志采集配置文件包括日志减少方式，所述日志减少方式用于减少部署于所述客户端的网元的日志数据的数据量；所述日志采集工具用于采集所述日志数据并按照所述日志减少方式对所述日志数据进行处理；
    采集处理模块，配置为通过所述日志采集工具采集所述日志数据，并按照所述日志减少方式对采集的所述网元的日志数据进行处理，得到减少后日志数据；
    传输模块，配置为向所述服务端传输所述减少后日志数据。
  9. 一种电子设备，包括：处理器、存储器和通信总线，其中，处理器和存储器通过通信总线完成相互间的通信；
    所述存储器，配置为存储计算机程序；
    所述处理器，配置为执行所述存储器中所存储的计算机程序，实现权利要求1至6任一项所述的数据处理方法。
  10. 一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时，使得所述处理器实现权利要求1至6任一项所述的数据处理方法。
PCT/CN2023/131164 2022-11-21 2023-11-13 数据处理方法及装置、电子设备及计算机可读存储介质 WO2024109569A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211457759.3 2022-11-21
CN202211457759.3A CN118101450A (zh) 2022-11-21 2022-11-21 数据处理方法、装置、设备及存储介质

Publications (1)

Publication Number Publication Date
WO2024109569A1 true WO2024109569A1 (zh) 2024-05-30

Family

ID=91160377

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/131164 WO2024109569A1 (zh) 2022-11-21 2023-11-13 数据处理方法及装置、电子设备及计算机可读存储介质

Country Status (2)

Country Link
CN (1) CN118101450A (zh)
WO (1) WO2024109569A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106250406A (zh) * 2016-07-21 2016-12-21 柳州龙辉科技有限公司 一种日志处理方法
CN111740884A (zh) * 2020-08-25 2020-10-02 云盾智慧安全科技有限公司 一种日志处理方法及电子设备、服务器、存储介质
WO2021189953A1 (zh) * 2020-10-12 2021-09-30 平安科技（深圳）有限公司 日志数据处理方法、装置、计算机设备和存储介质
CN114237927A (zh) * 2021-11-10 2022-03-25 浪潮通用软件有限公司 运行日志管理方法和系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106250406A (zh) * 2016-07-21 2016-12-21 柳州龙辉科技有限公司 一种日志处理方法
CN111740884A (zh) * 2020-08-25 2020-10-02 云盾智慧安全科技有限公司 一种日志处理方法及电子设备、服务器、存储介质
WO2021189953A1 (zh) * 2020-10-12 2021-09-30 平安科技（深圳）有限公司 日志数据处理方法、装置、计算机设备和存储介质
CN114237927A (zh) * 2021-11-10 2022-03-25 浪潮通用软件有限公司 运行日志管理方法和系统

Also Published As

Publication number Publication date
CN118101450A (zh) 2024-05-28

Similar Documents

Publication Publication Date Title
US11379475B2 (en) Analyzing tags associated with high-latency and error spans for instrumented software
US10956362B1 (en) Searching archived data
US11120344B2 (en) Suggesting follow-up queries based on a follow-up recommendation machine learning model
US11277312B2 (en) Behavioral based device clustering
US11775501B2 (en) Trace and span sampling and analysis for instrumented software
US20210011932A1 (en) Generating events in excess of licensed event count
US11288283B2 (en) Identifying metrics related to data ingestion associated with a defined time period
CN108509326B (zh) 一种基于nginx日志的服务状态统计方法及系统
JP4132441B2 (ja) 管理対象オブジェクトのデータ管理装置
US20190034429A1 (en) Translating a natural language request to a domain-specific language request using templates
CN107634848A (zh) 一种采集分析网络设备信息的系统和方法
US11170016B2 (en) Navigating hierarchical components based on an expansion recommendation machine learning model
US10133779B2 (en) Query hint management for a database management system
US20190034500A1 (en) Creating dashboards for viewing data in a data storage system based on natural language requests
CN108920607B (zh) 字段发现方法、装置及电子设备
CN112559567A (zh) 适用于olap查询引擎的查询方法及装置
WO2017000592A1 (zh) 数据处理方法、装置及系统
CN102541884A (zh) 数据库优化方法和装置
CN111338888B (zh) 一种数据统计方法、装置、电子设备及存储介质
CN113051460A (zh) 基于Elasticsearch的数据检索方法、系统、电子设备及存储介质
CN111949705A (zh) 慢查询统计分析方法、装置和系统
US20190034247A1 (en) Creating alerts associated with a data storage system based on natural language requests
CN113259151B (zh) 融合内容分发网络cdn中域名带宽明细监控方法及系统
WO2024109569A1 (zh) 数据处理方法及装置、电子设备及计算机可读存储介质
CN114116872A (zh) 数据处理方法、装置、电子设备及计算机可读存储介质