WO2023279912A1 - 一种数据处理方法、装置、系统、计算机设备、可读存储介质及计算机程序产品 - Google Patents

一种数据处理方法、装置、系统、计算机设备、可读存储介质及计算机程序产品 Download PDF

Info

Publication number
WO2023279912A1
WO2023279912A1 PCT/CN2022/097848 CN2022097848W WO2023279912A1 WO 2023279912 A1 WO2023279912 A1 WO 2023279912A1 CN 2022097848 W CN2022097848 W CN 2022097848W WO 2023279912 A1 WO2023279912 A1 WO 2023279912A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
index
component
query
indicator
Prior art date
Application number
PCT/CN2022/097848
Other languages
English (en)
French (fr)
Inventor
包增辉
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Publication of WO2023279912A1 publication Critical patent/WO2023279912A1/zh
Priority to US18/128,144 priority Critical patent/US20230237064A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/258—Data format conversion from or to a database
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/248—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F7/06—Arrangements for sorting, selecting, merging, or comparing data on individual record carriers
    • G06F7/14—Merging, i.e. combining at least two sets of record carriers each arranged in the same ordered sequence to produce a single set having the same ordered sequence

Definitions

  • the embodiment of the present application is based on the Chinese patent application with the application number 202110772159.5 and the filing date of July 08, 2021, and claims the priority of the Chinese patent application.
  • the entire content of the Chinese patent application is hereby incorporated into the embodiment of the present application as refer to.
  • the present application relates to the field of computer technology, and in particular to a data processing method, device, system, computer equipment, readable storage medium and computer program product.
  • edge nodes In some application scenarios (such as live video on demand or cloud games, etc.), in order to reduce delays, services will be deployed closer to users (ie, edge nodes). These edge nodes form edge clusters, which need to be monitored The performance and stability of the deployed service, etc. Therefore, the indicator data generated in the application will be collected to detect the performance and stability of the service.
  • the service endpoints in each edge cluster regularly pull and store indicator data. When user equipment needs to obtain indicator data, the central cluster pulls the required indicator data from the edge cluster service endpoints, and then feeds back to the user device.
  • the indicator data Since a lot of data will be generated in the service endpoint, the indicator data will generally be saved for a period of time, and the data will be deleted after a period of time to ensure the normal operation of the service endpoint, which makes the performance of data storage poor and cannot be processed. Efficient data transfer is not conducive to storing long-lived data.
  • Embodiments of the present application provide a data processing method, device, system, computer equipment, readable storage medium, and computer program product, which can realize efficient data transmission by transmitting pre-aggregated index data.
  • An embodiment of the present application provides a data processing method, which is applied to a central cluster, and the central cluster includes a plurality of computer devices, and each of the computer devices is used to run at least one of the sending and receiving components, the coordinating writing components, and the database components kind;
  • the methods include:
  • the coordinating writing component converts the pre-aggregated index data into converted index data in a target storage format, and merges the converted index data to obtain stored index data;
  • the database component writes the storage index data to a storage disk.
  • An embodiment of the present application provides a data processing system.
  • the data processing system includes a transceiver node, a coordinating write node, and a database node. implemented by computer equipment;
  • the transceiver node is used to receive the collected index data sent by the edge cluster
  • the transceiver node is further configured to perform pre-aggregation processing on the collected index data to obtain pre-aggregated index data, and send the pre-aggregated index data to the coordinating write node;
  • the coordinating write node is used to convert the pre-aggregated index data into converted index data in a target storage format, and merge the converted index data to obtain stored index data;
  • the coordinating write node is also used to write the stored index data into the database node;
  • the database node is configured to write the storage indicator data into a storage disk.
  • the embodiment of the present application provides a data processing device, which is applied to computer equipment.
  • the computer equipment runs a transceiver component, a coordinated writing component, and a database component.
  • the device includes:
  • the indicator collection module is configured to receive the collected indicator data sent by the edge cluster through the transceiver component;
  • the data push module is configured to perform pre-aggregation processing on the collected index data based on the sending and receiving component, obtain pre-aggregated index data, and send the pre-aggregated index data to the coordination writing component;
  • a data conversion module configured to convert the pre-aggregated indicator data into converted indicator data in a target storage format based on the coordinated writing component
  • a data compression module configured to merge and process the conversion index data to obtain storage index data
  • a data storage module configured to write the storage index data into the database component, and the database component writes the storage index data into a storage disk.
  • An embodiment of the present application provides a computer device, including a processor, a memory, and an input and output interface;
  • the processor is connected to the memory and the input/output interface respectively, wherein the input/output interface is used to receive data and output data, the memory is used to store the computer program, and the processor is used to call the computer program, so that the computer device including the processor executes the program.
  • the data processing method in the application example is used to receive data and output data
  • the memory is used to store the computer program
  • the processor is used to call the computer program, so that the computer device including the processor executes the program.
  • An embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the embodiment of the present application in the data processing method.
  • An embodiment of the present application provides a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium.
  • the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various modes in the embodiments of the present application.
  • transceiver components there are transceiver components, coordinated writing components, and database components running in the central cluster, and the acquired data is pre-aggregated through the transmitting and receiving components, and the pre-aggregated indicator data is transmitted to the coordinated writing component , to reduce the amount of data, so as to achieve efficient data transmission, which is conducive to optimizing the performance of data storage and improving the efficiency of data processing;
  • the embodiment of the present application first writes the obtained storage index data into the database component, and then And the storage index data is written to the storage disk by the database component, so that the storage index data is transferred through the database component, and there is no need to delete the storage index data after saving for a period of time, so as to improve the life cycle of data storage.
  • FIG. 1 is a network interaction architecture diagram for data processing provided by an embodiment of the present application
  • FIG. 2 is a data processing architecture diagram provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of a data processing scenario provided by an embodiment of the present application.
  • Fig. 4 is a flow chart of a data storage method provided by an embodiment of the present application.
  • FIG. 5 is a flow chart of a data query method provided by an embodiment of the present application.
  • FIG. 6 is a schematic diagram of an index query scenario provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of the architecture of a data processing system provided by an embodiment of the present application.
  • FIG. 8 is an architecture diagram of a monitoring collection node provided by an embodiment of the present application.
  • FIG. 9 is a schematic diagram of another data processing system architecture provided by an embodiment of the present application.
  • Fig. 10 is a schematic diagram of a cloud game deployment scenario provided by the embodiment of the present application.
  • Fig. 11 is a schematic diagram of a data processing device provided by an embodiment of the present application.
  • FIG. 12 is a schematic structural diagram of a computer device provided by an embodiment of the present application.
  • the embodiments of the present application may use cloud technology and blockchain technology to realize data transmission and storage.
  • Cloud Storage is a new concept extended and developed on the concept of cloud computing.
  • Distributed cloud storage system (hereinafter referred to as storage system) refers to the storage system through cluster application, grid technology and distributed storage file system, etc. It is a storage system that integrates a large number of different types of storage devices (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to jointly provide data storage and service access functions.
  • the storage method of the storage system is: create logical volumes, when creating logical volumes, allocate physical storage space for each logical volume, the physical storage space may be composed of disks of a storage device or several storage devices.
  • the client stores data on a logical volume, that is, the data is stored on the file system.
  • the file system divides the data into many parts, and each part is an object.
  • the object not only contains data but also contains data identification (ID, ID Entity) and other additional information
  • the file system writes each object to the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access data, the file system can be based on each
  • the object's storage location information allows the client to access the data.
  • the process of the storage system to allocate physical storage space for the logical volume is: according to the capacity estimation of the object stored in the logical volume (this estimation often has a large margin relative to the capacity of the actual object to be stored) and the independent redundant disk array ( RAID (Redundant Array of Independent Disk) group, which divides the physical storage space into strips in advance, and a logical volume can be understood as a strip, thus allocating physical storage space for the logical volume.
  • RAID Redundant Array of Independent Disk
  • the data processing system of the embodiment of the present application includes a central cluster and an edge cluster.
  • the edge cluster may include at least one service endpoint, and each service endpoint may generate more index data. Therefore, the embodiment of the present application may also be based on Big data realizes the processing of indicator data.
  • Big Data refers to the collection of data that cannot be captured, managed and processed by conventional software tools within a certain period of time. It requires a new processing model to have stronger decision-making power, insight and discovery and process optimization capabilities. Massive, high-growth and diverse information assets.
  • blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.
  • Blockchain (Block Chain), essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information for verification The validity of its information (anti-counterfeiting) and the generation of the next block.
  • the blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.
  • the embodiment of the present application can realize the processing of indicator data based on cloud technology and block chain technology.
  • the embodiment of the present application can be applied to the field of cloud games to monitor the indicator data generated by cloud games.
  • Online gaming technology Cloud gaming technology enables thin clients with relatively limited graphics processing and data computing capabilities to run high-quality games.
  • the game is not run on the player's game terminal, but in the cloud server, and the cloud server renders the game scene into a video and audio stream, which is transmitted to the player's game terminal through the network.
  • the player's game terminal does not need to have powerful graphics computing and data processing capabilities, but only needs to have basic streaming media playback capabilities and the ability to obtain player input instructions and send them to the cloud server.
  • the embodiments of the present application may also be applied in the field of application program monitoring to monitor indicator data generated in application programs, etc., which is not limited here.
  • the application program may be any application program, such as a video application program, a file editing application program, and a music application program.
  • FIG. 1 is a network interaction architecture diagram of data processing provided by the embodiment of the present application.
  • the embodiment of the present application can be implemented by the central cluster 101 .
  • the central cluster 101 is a cluster composed of multiple computer devices where the associated application program is located, and can deploy services related to the application program to manage one or more edge clusters, and can manage service endpoints in each edge cluster, Services can be provided to edge clusters.
  • the central cluster 101 can perform data interaction with the edge cluster 102.
  • the edge cluster 102 can include one or at least two service endpoints, such as service endpoint 102a, service endpoint 102b, and service endpoint 102c.
  • Each The service endpoint can be considered as an edge node, wherein the edge cluster 102 includes a computer room or node that is closer to the user than the central cluster, and the computer room or node where the service is deployed is called a service endpoint or an edge node.
  • the service endpoint in the edge cluster is timed Pull storage index data, compared with the central cluster, the transmission distance between the edge cluster and the user device is closer.
  • the central cluster 101 can acquire the collection index data from any service endpoint, and the collection index data is the index data generated in the service endpoint, or the index data monitored by the service endpoint.
  • the central cluster 101 can compress and convert the acquired collection index data to obtain storage index data (index data for storage), and write the storage index data into the database components included in the central cluster.
  • the central cluster 201 includes a plurality of computer devices, wherein each computer device runs at least one component with a specific functional module (such as a sending and receiving component (having the function of sending and receiving data), a coordinating writing component (having the function of writing data function), database component (with the function of writing data to disk), indicator query component (with the function of querying indicators) and indicator visualization component (with the function of visual indicators)) to complete the data processing provided by the embodiment of the present application method.
  • a specific functional module such as a sending and receiving component (having the function of sending and receiving data), a coordinating writing component (having the function of writing data function), database component (with the function of writing data to disk), indicator query component (with the function of querying indicators) and indicator visualization component (with the function of visual indicators)
  • At least one computer device running the same component constitutes a node in the central cluster
  • at least one computer device running the transceiver component constitutes a transceiver node in the central cluster
  • at least one computer device running the coordinated writing component constitutes a central cluster
  • at least one computer device running with database components constitutes a database node in the central cluster
  • at least one computer device running with index query components constitutes an index query node in the central cluster
  • runs with index visualization components At least one computer device constitutes an indicator visualization node in the central cluster.
  • the central cluster 101 may receive the indicator query information sent by the user equipment (such as the user equipment 103a, the user equipment 103b, or the user equipment 103c, etc.), obtain the indicator query result from the database component based on the indicator query information, and use the The index query result is sent to the user equipment.
  • the user equipment such as the user equipment 103a, the user equipment 103b, or the user equipment 103c, etc.
  • FIG. 2 is a data processing architecture diagram provided by an embodiment of the present application.
  • the data processing system includes a central cluster 201, an edge cluster 202, and user equipment 203, wherein the central cluster 201 includes a plurality of computer devices, and each computer device is used to run the transceiver component 2011 and coordinate the writing component 2012. At least one of the database component 2013, the indicator query component 2014, and the indicator visualization component 2015, wherein the edge cluster 202 may include at least one service endpoint.
  • the collection indicator data sent by the edge cluster 202 is received by the transceiver component 2011, the transceiver component 2011 performs pre-aggregation processing on the collection indicator data, obtains the pre-aggregate indicator data, and sends the pre-aggregate indicator data to the coordination writing component 2012, coordinates writing
  • the input component 2012 converts the pre-aggregated index data into the converted index data with the target storage format, and merges the converted index data to obtain the stored index data, writes the stored index data into the database component 2013, and the database component 2013 stores the index data Data is written to the storage disk.
  • the central cluster 201 may also receive index query information sent by the user equipment 203 , obtain an index query result from the database component 2013 based on the index query information, and send the index query result to the user equipment 203 .
  • FIG. 3 is a schematic diagram of a data processing scenario provided by an embodiment of the present application.
  • the central cluster 301 can receive the collected index data sent by the edge cluster 302 through the transceiver component. into the component. Based on the coordinated writing component, the pre-aggregated indicator data is format-converted, specifically, the pre-aggregated indicator data is converted into converted indicator data in the target storage format, and the converted indicator data is merged to obtain stored indicator data.
  • the central cluster 301 can write the storage index data into the database component, and based on the database component, the storage index data can be written into the storage disk.
  • the sending and receiving components, coordinated writing components and database components are integrated in the central cluster, and the receiving and receiving components can compress the acquired data to reduce the amount of data, and send the compressed data to the coordinated writing into the component, which can realize data transmission more efficiently, which is conducive to optimizing data storage performance.
  • the obtained storage index data is written into the database component. Processing efficiency.
  • the central cluster 301 may also receive the index query information sent by the user equipment 303, and send an index query request to the index query component based on the index query information, where the index query request includes the index query information.
  • the central cluster 301 can split the index query request into M fragmented query requests based on the index query component, where M is a positive integer. Based on M shard query requests, obtain the query index shard data corresponding to the M shard query requests from N database components, integrate the M query index shard data, obtain the index query results, and compile the index query results sent to the user equipment 303.
  • the central cluster, edge cluster, and user equipment mentioned in the embodiments of the present application can all be a kind of computer equipment, wherein the central cluster can be composed of one computer equipment or at least two computer equipment, and the edge cluster can be composed of It may be composed of one computer device or at least two computer devices, and the computer device in the embodiment of the present application includes but not limited to a terminal device or a server.
  • a computer device can be a server or a terminal device, or a system composed of a server and a terminal device.
  • the terminal device mentioned above can be an electronic device, including but not limited to mobile phones, tablet computers, desktop computers, notebook computers, handheld computers, vehicle-mounted devices, augmented reality/virtual reality (AR/VR, Augmented Reality/ Virtual Reality) equipment, head-mounted displays, smart TVs, wearable devices, smart speakers, digital cameras, cameras and other mobile Internet devices (MID, Mobile Internet Device) with network access capabilities, or in scenarios such as trains, ships, and flights terminal equipment, etc.
  • AR/VR Augmented Reality/ Virtual Reality
  • MID Mobile Internet Device
  • the server mentioned above can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, Cloud servers for basic cloud computing services such as network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery network (CDN, Content Delivery Network), and big data and artificial intelligence platforms.
  • cloud services cloud databases, cloud computing, cloud functions, cloud storage, Cloud servers for basic cloud computing services such as network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery network (CDN, Content Delivery Network), and big data and artificial intelligence platforms.
  • CDN Content Delivery Network
  • the data involved in the embodiments of the present application may be stored in a computer device, or the data may be stored based on cloud storage technology, which is not limited here.
  • FIG. 4 is a flow chart of a data storage method provided in an embodiment of the present application.
  • the method is implemented by a central cluster, the central cluster includes a plurality of computer devices, and each computer device is used to run at least one of the sending and receiving components, the coordination writing component, the database component, the indicator query component and the indicator visualization component
  • the data storage process includes the following steps:
  • Step S401 receiving the collection index data sent by the edge cluster through the transceiver component.
  • the edge cluster may include at least one service endpoint, and the central cluster may receive collection index data sent by any service endpoint or any number (that is, at least two) service endpoints in the edge cluster through the transceiver component.
  • the collected index data refers to the specific data corresponding to the collected index, which includes but not limited to the round-trip time delay (RTT, Round-Trip Time) per second, packet loss rate and data update frequency, etc., which can reflect The performance of the application, the collection index data is a service endpoint or any number of service endpoints in the edge cluster, the data corresponding to the indicators collected from the associated user equipment (that is, the user equipment using the application), for example, for For the indicator "RTT", the collection indicator data corresponding to the indicator "RTT" collected by the edge cluster may be "0.5 seconds", etc.
  • the sending and receiving component refers to a component that can obtain data from the edge cluster and send the obtained data to other components, that is, the sending and receiving component refers to a component that can receive data and push data.
  • the sending and receiving component may include an indicator proxy component, through which the central cluster can establish a network persistent connection with the edge cluster, and receive the collected indicator data sent by the edge cluster through the network persistent connection.
  • the sending and receiving component includes an indicator proxy component and a monitoring collection component. The central cluster can establish a long network connection with the edge cluster through the indicator proxy component included in the sending and receiving component, and receive the collection indicator data sent by the edge cluster through the long network connection, and collect Indicator data is sent to the monitoring collection component.
  • the indicator proxy component is equivalent to an access gateway, such as the MetricProxy component, which can receive the transmission control protocol (TCP, Transmission Control Protocol) connection of the edge cluster and maintain the connection and heartbeat with the edge cluster, namely Realize the establishment of a network long connection with the edge cluster, receive the collection index data sent by the edge cluster, and push the processed collection index data to other components in the central cluster (such as monitoring and collection components, etc.).
  • TCP Transmission Control Protocol
  • Transmission Control Protocol Transmission Control Protocol
  • the monitoring collection component can be considered as a component integrated with pre-aggregation functions, such as the Prometheus component, which can receive the indicator push request from the indicator proxy component, obtain the collection indicator data pushed by the indicator proxy component, and perform pre-aggregation processing on the collection indicator data , for example, the acquired data may be cached in the monitoring collection component.
  • pre-aggregation functions such as the Prometheus component
  • the network long connection refers to a long-term (set time threshold) network connection between the indicator agent component and the edge cluster, which can send multiple data packets, such as Transmission Control Protocol (TCP, Transmission Control Protocol) long connection or Hypertext Transfer Protocol (HTTP, Hypertext Transfer Protocol) long connection, etc.
  • TCP Transmission Control Protocol
  • HTTP Hypertext Transfer Protocol
  • the network long connection can be maintained through heartbeat, for example, take a service endpoint in the edge cluster as an example, the service endpoint and the indicator
  • the proxy components send heartbeat packets to maintain a long network connection between the service endpoint and the indicator proxy component. For example, due to the HTTP persistent connection, there is an overhead of link establishment and disconnection each time.
  • long connections other than HTTP long connections such as TCP long connections, etc.
  • HTTP long connections such as TCP long connections, etc.
  • long-term connections of different protocol types have different characteristics.
  • the target connection characteristics that meet the actual needs can be obtained, and the protocol type corresponding to the target connection characteristics can be obtained. Based on this The protocol type establishes a long network connection between the indicator agent component and the edge cluster, where the target connection characteristics refer to the characteristics determined according to the requirements of the connection effect to be achieved (such as small link establishment and disconnection overhead, and small connection maintenance overhead) .
  • TCP can be determined as the protocol type corresponding to the target connection characteristic. Based on the protocol The type establishes a TCP long connection between the indicator proxy component and the edge cluster, so that the connection between the indicator proxy component and the edge cluster will not be frequently established and disconnected, which can reduce the overhead for establishing and disconnecting the link;
  • HTTP can be determined as the protocol type corresponding to the target connection characteristic, and based on this protocol type, HTTP can be established between the indicator proxy component and the edge cluster.
  • the indicator agent component is equivalent to a network access gateway, which can receive the network connection of the edge cluster (including network long connection and network short connection, etc.), and can maintain the network connection and heartbeat with the edge cluster.
  • the central cluster can receive the collected indicator data sent by the edge cluster through the network long connection in the following manner: receive the pending indicator data sent by the edge cluster through the network long connection; the indicator proxy component obtains the indicator of the pending indicator data attribute label, merge and process the pending indicator data with the same indicator attribute label to obtain the collected indicator data.
  • the index attribute label is used to indicate the index attribute corresponding to the index data to be processed, and may be the attribute label of the index data to be processed itself, such as the unit attribute label corresponding to the index data to be processed (such as time attribute label, distance attribute label or rate attribute label, etc.), etc., can also be the attribute label of the associated object of the indicator data to be processed, for example, the indicator data to be processed is the indicator data obtained by monitoring the application program, etc.
  • the attribute tag may be an attribute tag of the user associated with the application program when generating the indicator data to be processed, such as an age tag or a gender tag, etc., which is not limited here.
  • the edge cluster can obtain the index data to be processed and the index information of the index data to be processed.
  • the index information can include f index attribute information, and the index attribute information includes but not limited to the index attribute label, index name and Index collection time, etc., f is a positive integer; the edge cluster sends the obtained index data to be processed and the index information of the index data to be processed to the index agent component.
  • the edge cluster can obtain the application data generated in the application program, perform processing on the application data, and obtain the indicator data to be processed.
  • the index information and index data in the received application data are associated with each other.
  • the edge cluster can obtain the pending index data corresponding to the target index, the target index includes but not limited to round-trip delay (Round-Trip Time, RTT) per second, packet loss rate, and data update frequency.
  • RTT Round-trip delay
  • the indicator data to be processed can be represented by charts, texts or lists, etc., as shown in Table 1 below:
  • Table 1 is a representation of the indicator data to be processed and the indicator information of the indicator data to be processed.
  • Table 1 shows that the central cluster receives the indicator data to be processed based on the indicator agent component, and the indicator data to be processed The number is d, and d is a positive integer.
  • the d indicators to be processed include indicator data 1, indicator data 2, and indicator data 3, among which the indicator name of indicator data 1 is RTT, and the attribute label of the indicator is "attribute label a1 and attribute label b1, etc", the index collection time is collection time 1, ...; the index name of index data 2 is RTT, the index attribute label is "attribute label a2 and attribute label b1, etc", the index collection time is collection time 1, ...;
  • the index name of index data 3 is packet loss rate, the index attribute label is "attribute label a1 and attribute label b2, etc.”, the index collection time is collection time 2, ...; the index name of index data 4 is packet loss rate, and the index attribute label It is "attribute label a2 and attribute label b1, etc.”, and the index collection time is collection time 1, .
  • the indicator name is generally used to reflect the meaning of the monitored sample.
  • http_request_total is used to indicate the total number of HTTP requests received by the current system
  • the indicator attribute label is used to indicate the feature dimension of the current sample (that is, the indicator data to be processed).
  • the central cluster can obtain the indicator attribute labels of the indicator data to be processed based on the indicator agent component, and merge and process the indicator data to be processed with the same indicator attribute label to obtain the collected indicator data.
  • the indicator attribute labels include the age attribute label and gender attribute tags, merge and process the unprocessed index data with the same age attribute tag and gender attribute tag to obtain the collected index data and reduce the storage capacity of the index data.
  • the index data 2 and the index data 4 have the same index attribute label "attribute label a2 and attribute label b1", and the index data 2 and index data 4 in the index data to be processed are merged and processed to obtain the collection index data , the collection index data can be seen in Table 2.
  • the central cluster can merge and process the pending indicator data based on the indicator name and indicator attribute label, that is, merge and process the pending indicator data with the same indicator name and indicator attribute label to obtain the collected indicator data.
  • the collection indicator The data can be shown in Table 3:
  • the central cluster can obtain the cluster resource status. If the cluster resource status is sufficient, the central cluster can directly receive the collection index data sent by the service endpoint in the edge cluster; if the cluster resource status is cluster resource shortage, Then the central cluster can generate the connection random number corresponding to the service endpoint in the edge cluster, and obtain the collection indicator data sent by the service endpoint based on the connection random number. For example, the central cluster can establish a connection with the service endpoint in the edge cluster through the transceiver component, obtain the endpoint information of the service endpoint, and generate a connection random number based on the endpoint information.
  • connection random number is a random number used to indicate that the service endpoint establishes a connection with the central cluster, wherein the connection random number is a random number within a random number interval, and the random number interval is determined based on the connection establishment range , that is to say, the central cluster can generate a connection random number within the random number interval according to the endpoint information, which is used to indicate whether the central cluster receives the indicator data sent by the service endpoint.
  • the endpoint information may include the endpoint identifier of the service endpoint, the endpoint memory space, the endpoint transmission bandwidth, etc., and the central cluster may generate connection random numbers according to one or more elements in the endpoint information.
  • the central cluster can directly use a random number generating function (such as a random function (rand function) or a random seed function (srand), etc.) to process one or more elements in the endpoint information to generate connection random numbers; or, The central cluster can obtain the element hash value corresponding to one or more elements in the endpoint information, and perform a remainder calculation on the element hash value to obtain a connection random number.
  • the specified location range can be obtained from the element hash value
  • the hash value at the location, the hash value at the specified location range is calculated to obtain the connection random number, etc., and there is no limitation here.
  • the sending and receiving component may include an indicator proxy component, or may include an indicator proxy component and a monitoring collection component.
  • the central cluster establishes a connection with the service endpoint in the edge cluster through the indicator proxy component in the sending and receiving component to generate a connection random number.
  • the central cluster can generate connection random numbers based on endpoint information, or can generate connection random numbers based on endpoint information and system network time, or can generate connection random numbers based on hash values of endpoint information, etc., without limitation here.
  • connection establishment range is obtained, wherein the connection establishment range can be regarded as a random number threshold range, which is used to implement the percentage drop request, so that the central cluster can provide services corresponding to the random number located in the connection establishment range
  • the indicator data sent by the endpoint is processed, and the indicator data sent by the service endpoint corresponding to the random number outside the connection establishment range is discarded.
  • the service endpoint in the receiving edge cluster sends Collect indicator data; when the connection random number is not within the connection establishment range, obtain the received historical indicator data sent by the service endpoint in the edge cluster, discard the historical indicator data, that is, discard the service endpoint corresponding to the connection random number
  • an indicator push failure message can be sent to the service endpoint in the edge cluster.
  • the central cluster can determine the connection establishment range based on the index reception ratio. Assuming that the index reception ratio is 70%, and the connection random number ranges from 1 to 100, the connection establishment range is any consecutive number from 1 to 100 70% of the number, for example, the connection establishment range can be 1-70, that is, the central cluster can generate a connection random number corresponding to the service endpoint in the range of 1-100, if the connection random number belongs to 1-70, the receiving edge The collected indicator data sent by the service endpoint in the cluster; if the connection random number does not belong to 1 ⁇ 70, then discard the indicator data associated with the service endpoint in the edge cluster, for example, you can send the indicator push failure to the service endpoint in the edge cluster information.
  • the central cluster can obtain the The historical indicator data sent by the service endpoint B is discarded.
  • the central cluster can only discard the historical indicator data sent by the service endpoint, while maintaining the network connection with the service endpoint. Through this process, the central cluster can have the same probability of receiving the indicator data of each service endpoint.
  • each service endpoint can be discarded according to the percentage, so that when the cluster resource status of the central cluster is in the cluster resource shortage state, it can be used to a certain extent. In this way, the indicator data that needs to be processed can be reduced, and the resource utilization efficiency of the central cluster can be improved.
  • the central cluster can receive the collected indicator data sent by the service endpoints in the edge cluster based on the indicator push volume of the edge cluster, where the indicator push volume refers to the total amount of indicator data pushed by the edge cluster to the central cluster, that is, the The amount of resources occupied by metrics data pushed to the central cluster.
  • the central cluster can monitor the indicator push volume of the edge cluster.
  • the central cluster can receive the collection indicator data sent by the service endpoint in the edge cluster; when the indicator push volume is greater than When the indicator processing volume threshold is set, it means that the amount of indicator data received by the central cluster exceeds the maximum amount of data that can be processed by itself, and the acquired indicator data can be filtered, for example, by sending and receiving components and service endpoints in the edge cluster Establish a connection, obtain the endpoint information of the service endpoint, generate a connection random number based on the endpoint information, and determine whether to receive and collect indicator data based on the connection random number, so as to reduce the amount of indicator data that the central cluster needs to process and reduce the resources of the central cluster pressure.
  • the central cluster can also receive the initial indicator data sent by the edge cluster through the transceiver component, obtain the indicator type to be collected, and generate the collected indicator data based on the indicator data corresponding to the indicator type to be collected obtained from the initial indicator data .
  • the initial indicator data can be regarded as the indicator data sent directly by the edge cluster.
  • the type of the indicator to be collected is used to indicate the type of indicator data that the indicator agent component needs to receive.
  • the central cluster can receive the initial indicator data sent by the edge cluster through the transceiver component to obtain the indicator type to be collected, and will obtain the indicator data corresponding to the indicator type to be collected from the initial indicator data as the indicator data to be processed, and the indicator data to be processed Perform merge processing to obtain the collection index data.
  • each index data in the embodiment of the present application refers to the index data of different stages obtained after the central cluster obtains the index data from the edge cluster and processes them in different components, that is to say, the different index data
  • the name is used to indicate the name of the indicator data obtained after different processing of the indicator data at different stages.
  • the indicator interaction logic can be encapsulated into an indicator static library (lib), which will collect various data obtained by the edge cluster by default. Indicator data, and the indicator data of some indicator types is unnecessary.
  • the central cluster can filter the acquired initial indicator data based on the indicator type to be collected to remove irrelevant indicator data, thereby reducing the amount of data and indicator agent components. Push pressure on metric data.
  • Step S402 pre-aggregate the collected index data based on the sending and receiving component to obtain the pre-aggregated index data, and send the pre-aggregated index data to the coordinating writing component.
  • the index data refers to the index data that needs to be pre-aggregated in the collected index data in the embodiment of the present application, such as collected index data.
  • the sending and receiving component is integrated with a first data pushing function, and the first data pushing function is used to send the pre-aggregated indicator data to the coordination writing component through the sending and receiving component.
  • the sending and receiving component includes an indicator agent component
  • the indicator agent component performs pre-aggregation processing on the collected indicator data, obtains the pre-aggregated indicator data, and sends the pre-aggregated indicator data to the coordination writing component.
  • the sending and receiving component includes an indicator proxy component and a monitoring collection component
  • the indicator proxy component sends the collected indicator data to the monitoring and collection component.
  • the indicator proxy component can freely push and collect indicator data, which is conducive to performance optimization.
  • the monitoring and collection component obtains the index information of the collected index data, obtains the index aggregation attribute, performs index conversion on the collected index data corresponding to the index information matching the index aggregation attribute (that is, executes the index conversion algorithm), and obtains the pre-aggregated index data.
  • the monitoring collection component sends a remote write request to the coordinated write component, and sends pre-aggregated indicator data to the coordinated write component based on the remote write request.
  • the pre-aggregated indicator data is stored based on the monitoring and collection component, and the pre-aggregated indicator data is bypassed to the coordinating writing component based on the remote writing request.
  • the indicator proxy component is integrated with a second data push function
  • the second data push function is used to push the collected indicator data to the monitoring and collection component through the indicator proxy component
  • the monitoring and collection component is integrated with a third data push function
  • the third data push The function is used to send pre-aggregated metric data to the coordinating write component through the monitoring collection component.
  • the index aggregation attribute and index conversion algorithm may be preset in advance and stored in the monitoring collection component. As shown in Table 1 to Table 3 above, when the indicator data is transmitted in each component, it will carry corresponding indicator information.
  • the index aggregation attribute may be any one or multiple index attribute labels of the index attribute tags, or any one or any number of index attribute information among the f index attribute information.
  • the The collected index data with the same index aggregation attribute executes the index conversion algorithm to obtain the pre-aggregated index data and the pre-aggregated index name corresponding to the pre-aggregated index data.
  • the index conversion algorithm can be the mean value algorithm, the maximum value algorithm, the minimum value algorithm The algorithm or the statistical algorithm with the most occurrences are not limited here.
  • index aggregation attributes and index conversion algorithms may be called index pre-aggregation rules, and the index pre-aggregation rules may be updated according to actual needs. For example, there is a group of collection index data whose index name is RTT, assuming that the index aggregation attribute is the gender attribute label in the index attribute label, the index conversion algorithm is the mean algorithm, and the collection index data corresponding to the first gender attribute label is obtained There are "1, 2, 1, 0.5", and the collection index data corresponding to the second gender attribute label is "2, 3, 2, 3".
  • the name of the corresponding pre-aggregated indicator such as "Mean RTT of the second gender”.
  • the indicator agent component can push the collected indicator data to the monitoring and collection component in the following ways: the indicator agent component caches the collected indicator data in the indicator queue to be sent; The indicator data is pushed to the monitoring collection component.
  • the collected index data in the index queue to be sent is split to obtain at least two collection index data groups, based on the second data push function Send at least two collection index data groups in parallel (Period Push) to the monitoring collection component; if the data volume of the collection index data is less than the data push threshold, then directly through the second data push function of the index proxy component and the queue of indicators to be sent, the The collected indicator data is pushed to the monitoring and collection component, thereby improving the efficiency of pushing indicator data.
  • the monitoring and collection component can receive the indicator push request from the indicator agent component, based on the indicator push request, write the received collected indicator data into the indicator agent component, and perform pre-aggregation processing on the collected indicator data to obtain the pre-aggregated indicator data , periodically send the pre-aggregated indicator data to the coordinating write component.
  • Step S403 the coordinating writing component converts the pre-aggregated index data into converted index data in the target storage format, and merges the converted index data to obtain stored index data.
  • the central cluster can receive the remote write request sent by the monitoring and collection component based on the coordinated writing component, and convert the pre-aggregated indicator data into converted indicator data in the target storage format based on the remote write request, wherein,
  • the target storage format refers to the data format supported by the database component in the central cluster. For example, assuming that the database component is M3DB, the target storage format corresponding to the database component is m3 format.
  • the merging of the conversion index data can be implemented in the following manner: based on the coordinated writing component, the conversion index data is down-sampled to obtain the down-sampled index data, and the conversion storage period of the conversion index data is obtained, which will have The downsampled index data with the same converted storage period are merged to obtain the storage index data.
  • the conversion index data may be periodically down-sampled, or the conversion index data may be down-sampled according to the down-sampling method corresponding to the database component, such as the down-sampling method corresponding to m3.
  • the conversion storage period may refer to the collection time or generation time of the conversion index data (the generation time of the collection index data corresponding to the conversion index data) and the like.
  • Step S404 writing the storage index data into the database component, and the database component writes the storage index data into the storage disk.
  • the number of the database components is N, where N is a positive integer.
  • the coordination writing component obtains the storage index name and the storage index attribute label of the storage index data, and generates the storage hash value of the storage index data based on the storage index name and the storage index attribute label,
  • the storage fragment identifier is determined based on the storage hash value, and the storage index data is written into the database component corresponding to the storage fragment identifier.
  • the index information such as the index name and index attribute label of the index data will be synchronously processed.
  • writing the storage index data into the storage disk can be realized in the following manner: the database component performs cold and hot analysis on the storage index data, obtains the hot index data and the cold index data included in the storage index data, and determines the storage index data in the storage disk. The hot data storage area and the cold data storage area, write the hot index data into the hot data storage area, and write the cold index data into the cold data storage area.
  • hot index data refers to online index data that needs to be accessed frequently
  • cold index data refers to index data that is not frequently accessed.
  • hot index data refers to online index data that needs to be accessed frequently
  • cold index data refers to index data that is not frequently accessed.
  • FIG. 5 is a flowchart of a data processing method provided in an embodiment of the present application. As shown in Figure 5, the method is implemented by the central cluster, and the computer equipment in the central cluster can also run index query components, etc.
  • the data processing process includes the following steps:
  • Step S501 receiving index query information sent by a user equipment, and sending an index query request to an index query component based on the index query information.
  • the computer equipment in the central cluster can also run index query components, wherein the number of the database components is N, and N is a positive integer.
  • the number of database components can be set according to requirements, that is, the database components have scalability.
  • the central cluster may send an indicator query request to the indicator query component based on the indicator query information, where the indicator query request includes the indicator query information.
  • the central cluster can also run metrics visualization components. For example, based on the address of the visualization component of the indicator visualization component, the central cluster can receive the indicator query information sent by the user device; obtain the address of the query component of the indicator query component, establish a monitoring panel based on the address of the query component, and obtain the corresponding index query information from the monitoring panel The indicator query statement; based on the indicator query statement, send an indicator query request to the indicator query component.
  • FIG. 6 is a schematic diagram of an index query scenario provided by an embodiment of the present application.
  • the central cluster can receive the indicator query information sent by the user device 601 based on the visualization component address of the indicator visualization component, establish a monitoring panel 603 based on the indicator query information, and generate indicators corresponding to the indicator query information based on the monitoring panel 603
  • the query statement sends an indicator query request to the indicator query component 604 according to the indicator query statement.
  • the indicator query statement may also be generated by the user equipment.
  • the user device can obtain the address of the visualization component of the indicator visualization component, access the address of the visualization component, display the access page of the visualization component, obtain the address of the query component of the indicator query component based on the access page of the visualization component, establish a monitoring panel, and obtain the information in the monitoring panel.
  • an indicator query statement is generated based on the indicator query statement, and an indicator query request is sent to the indicator visualization component based on the indicator query statement, and the central cluster sends the indicator query request to the indicator query component based on the indicator visualization component.
  • the indicator query statement can be regarded as a query expression, and the statement type of the indicator query statement is determined according to the database component.
  • the statement type of the indicator query statement can be Pro Metheus query language type (PromQL, Prometheus Query Language), etc.
  • Step S502 the index query component splits the index query request into M fragmented query requests.
  • the index query component splits the index query request into M fragmented query requests based on the index query information. For example, according to the query index name and query index attribute label included in the index query information, Split the index query request into M shard query requests, where M is a positive integer.
  • Step S503 based on the M fragmented query requests, determine the query database component from the N database components, and obtain the query index fragmented data respectively corresponding to the M fragmented query requests from the query database component.
  • the central cluster initiates a query to the database component in parallel based on M shard query requests.
  • the index query information includes the shard query index names and shard query index attribute labels corresponding to the M shard query requests. .
  • determining the query database components from the N database components can be implemented in the following manner: based on the fragment query index names and fragment query indexes corresponding to the M fragment query requests Attribute label, obtain the fragment query hash values corresponding to M fragment query requests respectively; obtain the fragment query identifiers corresponding to M fragment query hash values respectively, and based on the M fragment query identifiers, query from N databases Among the components, k query database components are determined; k is a positive integer less than or equal to M.
  • the hash algorithm corresponding to the fragmented query hash value is the same as the hash algorithm corresponding to the stored hash value in FIG. 4 above.
  • the central cluster can obtain query index fragmentation data corresponding to M fragmentation query requests from k query database components (that is, M query index fragmentation data), and M query index fragmentation Data is sent to the metrics visualization component. Therefore, the efficiency of data query is improved through fragmented query.
  • Step S504 integrating the M pieces of query index fragment data to obtain an index query result, and sending the index query result to the user equipment.
  • the central cluster can combine M query index fragment data based on the index visualization component to obtain index query data; establish query coordinates based on index query information, and combine query coordinates and index query data to obtain Index query result.
  • the index query result may be icon data or curve coordinates including query coordinates and index query data, etc., or data including index query information and index query data; the index query result is sent to the user equipment so that the user The device displays the query result of the index, so that the user device can visually display the query result of the index and improve the effect of the index data query.
  • the indicator query statement is generated by the user equipment, the user equipment may also display the indicator query result on the monitoring panel.
  • the above-mentioned Figures 4 to 5 are a data processing process provided by the embodiment of the present application, including the collection, storage and query of index data.
  • the embodiment of the present application receives the collected index data sent by the edge cluster through the transceiver component;
  • the data push function in the component performs pre-aggregation processing on the collected indicator data, obtains the pre-aggregated indicator data, and sends the pre-aggregated indicator data to the coordinated writing component; based on the coordinated writing component, the pre-aggregated indicator data is converted into a target storage format
  • the conversion index data is merged and processed to obtain the storage index data; the storage index data is written into the database component, and the storage index data is written into the storage disk based on the database component.
  • the central cluster integrates the sending and receiving components, coordinated writing components, and database components, and integrates the data push function in the sending and receiving components, so that the receiving and receiving components can compress the acquired data to reduce the amount of data. And send the compressed data to the coordinating writing component, which can realize data transmission more efficiently and help optimize data storage performance. At the same time, write the obtained storage index data into the database component, based on the characteristics of the database component, improve Data processing efficiency.
  • FIG. 7 is a schematic structural diagram of a data processing system provided by an embodiment of the present application.
  • the data processing system includes a sending and receiving node, a coordinating write node and a database node, wherein, in the data processing system shown in Figure 7, the sending and receiving node includes an index agent node and a monitoring collection node, the data processing A system refers to a system corresponding to the central cluster 701 .
  • each node in the data processing system refers to a node composed of at least one computer device running the same component in the central cluster, that is, each node is realized by at least one computer device.
  • the cost (including time and energy, etc.) of updating the monitoring collection node is low, and the ability of the monitoring collection node to support pre-aggregation processing is relatively strong, which can improve the monitoring collection node. update efficiency and pre-aggregation efficiency.
  • the sending and receiving node is used to receive the collected index data sent by the edge cluster. It is also used to pre-aggregate the collected index data to obtain the pre-aggregated index data, and send the pre-aggregated index data to the coordinating write node; coordinate writing
  • the node is used to convert the pre-aggregated indicator data into the converted indicator data with the target storage format, and merges the converted indicator data to obtain the stored indicator data;
  • the coordinating write node is also used to write the stored indicator data into the database node Medium; the database node, which is used to write storage metric data to storage disks.
  • the sending and receiving nodes include an indicator proxy node and a monitoring collection node; the indicator proxy node is used to establish a network long connection with the edge cluster, and receive the collection index data sent by the edge cluster through the network long connection; push the collection index data To the monitoring collection node; the monitoring collection node is used to obtain the index information of the collected index data, and obtain the index aggregation attribute; perform index conversion on the collected index data corresponding to the index information matching the index aggregation attribute, and obtain the pre-aggregated index data; Send a remote write request to the coordinating write node, and send pre-aggregated metric data to the coordinating write node based on the remote write request.
  • the indicator proxy node is also used to receive the pending indicator data sent by the edge cluster through the network long connection; the indicator proxy node obtains the indicator attribute label of the indicator data to be processed, and will The indicator data is merged and processed to obtain the collected indicator data; the collected indicator data is cached in the indicator queue to be sent; the collected indicator data in the indicator queue to be sent is pushed to the monitoring collection node.
  • the transceiver node is also used to establish a connection with the service endpoint in the edge cluster, and obtain the endpoint information of the service endpoint, generate a connection random number based on the endpoint information; obtain the connection establishment range; when the connection random number is in the connection establishment When the scope is set, receive the collection indicator data sent by the service endpoint in the edge cluster.
  • the coordinating write node is also used to perform down-sampling processing on the conversion index data to obtain the down-sampling index data; obtain the conversion storage period of the conversion index data, and convert the down-sampling index data with the same conversion storage period Merge processing is performed to obtain storage index data.
  • the number of database nodes is N, and N is a positive integer; the coordinating write node is also used to obtain the storage index name and the storage index attribute label of the storage index data, based on the storage index name and the storage index attribute label, A storage hash value of the storage index data is generated; a storage fragment identifier is determined based on the storage hash value, and the storage index data is written into a database node corresponding to the storage fragment identifier.
  • the database node is also used for the database node to perform thermal analysis and processing on the storage index data to obtain the hot index data and cold index data included in the storage index data; determine the hot data storage area and cold data in the storage disk In the storage area, the hot index data is written into the hot data storage area, and the cold index data is written into the cold data storage area.
  • the transceiver node is also used to receive the initial indicator data sent by the edge cluster through the transceiver node; obtain the indicator type to be collected, and generate the collected indicator data based on the indicator data corresponding to the indicator type to be collected obtained from the initial indicator data. Indicator data.
  • the edge cluster can include one or at least two service endpoints (Service), which can be considered as edge nodes, and a long network connection is established between the service endpoint and the indicator proxy node. Based on the long network connection, each service endpoint Both can batch send (Batch Send) to the indicator proxy node to collect indicator data.
  • the indicator proxy node creates a coroutine pool/thread pool to receive the collected indicator data, and can also divide the large amount of collected indicator data into multiple small requests, and push the small requests to the monitoring collection node in parallel, for example, the indicator proxy node
  • the collection index data is received, and the number of the collection index data is 1 million.
  • the indicator proxy node can be a Metric Proxy system
  • the monitoring collection node can be a Prometheus system, which provides a series of toolsets to facilitate monitoring and querying the running status of services and applications.
  • the monitoring collection node can remotely write pre-aggregated indicator data to the coordinating write node.
  • the monitoring and collecting node can perform pre-aggregation processing on the received collection indicator data to obtain multiple pre-aggregated indicator data
  • the pre-aggregation indicator data may be split to obtain at least two pre-aggregation push requests, and based on the at least two pre-aggregation push requests, the pre-aggregation indicator data is sent to the coordinating write node in parallel.
  • the node parameters of the monitoring collection node can be updated, the node parameters include but not limited to the data cache time of the monitoring collection node and garbage collection (GC, Garbage Collection) parameters, etc., for example, to reduce the data cache time, the data cache time It refers to the retention time of the monitoring collection node for the received indicator data.
  • garbage collection GC, Garbage Collection
  • the GC parameters for example, set the GC parameter to is 30, then when it is detected that the memory allocated for the collection of index data for the (i+1)th time exceeds 30% of the memory allocated for the collection of index data for the i-th time, the GC function is triggered to perform garbage collection to release Monitor the memory of the collection node, thereby improving the operating efficiency of the monitoring collection node, i is a positive integer.
  • the Prometheus system 801 can be referred to FIG. 8 , which is an architecture diagram of a monitoring collection node provided by an embodiment of the present application.
  • the monitoring collection node can monitor Service Discovery (Service Discovery), discover the list of services (targets) that need to be monitored, and monitor services based on the Scrape Discovery function, which can be based on the Scrape Manager (Scrape Manager) ) monitor service changes, update the capture service, and capture indicator data based on the service list.
  • the service list can be used to indicate the service endpoint that the monitoring collection node needs to monitor.
  • the monitoring collection node can use the service list to obtain the obtained Collect indicator data for screening.
  • the monitoring collection node can also include a read-write agent component for reading/writing data, wherein the monitoring collection node can perform local storage and remote storage, and the local storage of the monitoring collection node is through the local disk of the monitoring collection node It is implemented by reading/writing data in the remote storage, and the remote storage is realized by remotely reading data from the remote reading endpoint and remotely writing data from the remote writing endpoint.
  • the monitoring collection node can store the received collection index data based on the read-write proxy component, and can also acquire and process the data stored in the local disk or remote read endpoint based on the read-write proxy component.
  • the monitoring collection node integrates the third data push function in the capture manager, for example, integrates the push interface in the capture manager, and realizes the third data push function based on the push interface, and will process the collected index data
  • the obtained pre-aggregated indicator data is sent to the coordinating write node, so that the monitoring and collecting nodes can freely control the push time and specifications of the indicator data, that is, the monitoring and collecting nodes can implement the pre-aggregated indicator data at any point in time based on actual needs.
  • the pre-aggregated indicator data can be sent to the coordinating write node in parallel (that is, the push method is parallel push), or the pre-aggregated The indicator data is sent serially to the coordinating write node (that is, the push method is serial push), etc.
  • the coordinating writing node may be the m3Coordinator system, and the coordinating writing node is used to convert the pre-aggregated indicator data into converted indicator data in the target storage format, merge the converted indicator data, and obtain the storage indicator Data, write storage index data into the database node, deploy multi-subprocesses on the coordinated write node, and use multi-core to improve throughput.
  • the coordinated write node can also integrate write worker pool policy (writeWorkerPoolPolicy) parameters, Based on this parameter, the coordinating write node can increase the size of the coroutine pool according to the pressure, thereby improving the work performance of the coordinating writing node.
  • the compression option can be added to reduce the amount of data written by the coordinating write node to the database node, and reduce the damage caused by the coordinating write component due to the large amount of outbound packets.
  • the performance of the coordinated write node is further improved.
  • the database node is used to write the storage indicator data to the storage disk.
  • the data processing system may further include an index visualization node and an index query node.
  • the indicator visualization node is used to receive the indicator query information sent by the user equipment, and sends the indicator query request to the indicator query node based on the indicator query information, wherein the indicator query request includes the indicator query information;
  • the indicator query node is used to split the indicator query request M is a query request for shards; M is a positive integer;
  • the index query node is also used to determine the query database node from N database nodes based on M shard query requests, and obtain M scores from the query database nodes Slice query requests corresponding query index fragment data, and send M query index fragment data to the index visualization node;
  • the index visualization node is used to integrate the M query index fragment data to obtain index query results, and Send the index query result to the user device.
  • the indicator visualization node is further configured to receive the indicator query information sent by the user equipment based on the visualization node address of the indicator visualization node; obtain the query node address of the indicator query node, and establish a monitoring panel based on the query node address , obtain the indicator query statement corresponding to the indicator query information from the monitoring panel; send an indicator query request to the indicator query node based on the indicator query statement.
  • the index query information includes the name of the fragment query index and the attribute label of the fragment query index corresponding to the M fragment query requests; the index query node is also used to correspond to the M fragment query requests respectively.
  • the name of the fragment query index and the attribute label of the fragment query index obtain the fragment query hash values corresponding to the M fragment query requests respectively; obtain the fragment query identifiers corresponding to the M fragment query hash values, and based on M shard query identifiers, k query database nodes are determined from N database nodes, k is a positive integer less than or equal to M.
  • the index visualization node is also used to combine M query index slice data to obtain index query data; establish query coordinates based on index query information, and combine query coordinates and index query data to obtain index query result.
  • FIG. 9 is a schematic diagram of another data processing system architecture provided by an embodiment of the present application.
  • the data processing system includes a sending and receiving node, a coordinating writing node, a database node, an index query node, and an index visualization node, etc., wherein the sending and receiving node includes an index agent node.
  • the data processing system refers to the system corresponding to the central cluster 901 .
  • each node in the data processing system refers to a node composed of at least one computer device running the same component in the central cluster.
  • the data processing system integrates the coordinated writing node into the indicator proxy node to reduce performance loss and reduce operation and maintenance costs.
  • the index proxy node obtains the collected index data, performs pre-aggregation processing on the collected index data, obtains the pre-aggregated index data, and sends the pre-aggregated index data to the coordination write
  • the ingress node that is, the function implemented by the monitoring collection node in FIG. 7 is implemented by the indicator proxy node 902 in the architecture shown in FIG. 9 .
  • the embodiment of the present application can be applied to the field of cloud games, and can also be a scenario where index data is collected for any application program.
  • FIG. 10 is a schematic diagram of a cloud game deployment scenario provided by an embodiment of the present application.
  • edge cluster 1 and edge cluster 2 shown in Figure 10 there are one or at least two edge clusters in the cloud game architecture (edge cluster 1 and edge cluster 2 shown in Figure 10), each edge cluster dynamically starts and destroys h instances, h is a positive integer
  • the instance is subject to overall scheduling and resource allocation by the services of the central cluster, and each instance interacts with application services, which can be application services (App, Application), H5 services, or computer webpage (PC Web) services.
  • application services can be application services (App, Application), H5 services, or computer webpage (PC Web) services.
  • each instance may include a scheduling module, an Android container module, an audio and video encoding module, and a streaming media transmission module, etc., wherein the streaming media transmission module can adjust the flow control strategy in real time according to the user's actual network status.
  • the central cluster can obtain the indicator data generated by the cloud game by monitoring the performance and status of the streaming media transmission module, that is, the indicator data generated by each user when using the cloud game, wherein the target indicator can also include network connection indicators , flow control strategy effectiveness indicators and online freeze indicators, etc.
  • the indicator data corresponding to the network connection indicators include but are not limited to mobile hotspots (wifi), the second generation wireless communication technology (2G, The 2nd Generation), the third generation Mobile communication technology (3G, 3rd Generation) and the fourth generation mobile communication system (4G, Fourth Generation Communications System), etc.
  • the collected index data sent by the edge cluster can be obtained, and the collected index data can be processed and stored in the database node.
  • the user device can query the index data in the database node to obtain the cloud game or The running status of applications, etc., and then optimize cloud games or applications.
  • the central cluster can obtain collection index data from edge nodes.
  • the collection index data can include but not limited to game freeze frequency index, game frame transmission rate index, game data synchronization delay index, and game rendering rate index, etc.
  • the central cluster processes the collected index data through the included components, obtains the stored index data, and writes the stored index data into the database component.
  • the user equipment needs to obtain the index data related to the cloud game, it can send the index query information to the central cluster, and the central cluster obtains the index query result by executing the steps shown in Figure 5, and sends the index query result to the user equipment.
  • the user device can determine the running status of the cloud game based on the query result of the indicator. If the running status of the cloud game indicated by the query result of the indicator is abnormal, the abnormal indicator data can be obtained from the query result of the indicator.
  • the index data of the cloud game is optimized, so that the cloud game can better meet the needs of users, and improve the operating efficiency and effect of the cloud game.
  • the user equipment can preset a reasonable index data range, compare the obtained index query result with the reasonable index data range, and when the index query result falls within the reasonable index data range, determine the cloud index indicated by the index query result.
  • the running state of the game is the normal state of the game; when there is data in the index query result that does not belong to the reasonable index data range, it is determined that the running state of the cloud game indicated by the index query result is the game abnormal state.
  • the reasonable index data range is used to indicate the range of index data that should be available when the cloud game can better meet the user's game needs.
  • the user device can optimize the cloud game, for example, update the data transmission algorithm between game devices , or, compress data packets transmitted between game devices, etc., to increase the game frame transmission rate.
  • FIG. 11 is a schematic diagram of a data processing device provided in an embodiment of the present application.
  • the data processing device may be a computer program (including program code, etc.) running on a computer device (running the above-mentioned sending and receiving component, coordinated writing component, database component, indicator query component and indicator visualization component), such as the data processing
  • the device may be an application software; the device may be used to execute corresponding steps in the method provided by the embodiment of the present application.
  • the data processing device 1100 can be used for the computer equipment in the embodiment corresponding to Figure 4, and the device can include: an indicator collection module 11, a data push module 12, a data conversion module 13, a data compression module 14 and Data storage module 15.
  • the indicator collection module 11 is configured to receive the collected indicator data sent by the edge cluster through the transceiver component; the data push module 12 is configured to perform pre-aggregation processing on the collected indicator data based on the transceiver component, obtain the pre-aggregated indicator data, and send the pre-aggregated indicator data Sent to the coordinated writing component; the data conversion module 13 is configured to convert the pre-aggregated index data into converted index data with the target storage format based on the coordinated written component; the data compression module 14 is configured to merge the converted index data, The storage index data is obtained; the data storage module 15 is configured to write the storage index data into the database component, and write the storage index data into the storage disk based on the database component.
  • the transceiver component includes an indicator proxy component and a monitoring collection component;
  • the indicator collection module 11 includes: a connection establishment unit 111 configured to establish a long network connection with the edge cluster through the indicator proxy component included in the transceiver component
  • the indicator collection unit 112 is configured to receive the collection indicator data sent by the edge cluster through the network long connection;
  • the data push module 12 includes: a first push unit 121 configured to push the collection indicator data to the monitoring and collection component based on the indicator proxy component
  • the data aggregation unit 122 is configured to obtain the index information of the collected index data based on the monitoring and collection component, and obtain the index aggregation attribute; carry out index conversion on the index data corresponding to the index information matched with the index aggregation attribute, and obtain the pre-aggregated index data
  • the second push unit 123 is configured to send a remote writing request to the coordination writing component based on the monitoring collection component, and send the pre-aggregated indicator data to the coordination writing component based on the remote writing request.
  • the indicator collection unit 112 includes: a data receiving subunit 1121, configured to receive the pending indicator data sent by the edge cluster through a network long connection; a data merging subunit 1122, configured to obtain the pending Process the index attribute tags of the index data, merge and process the index data to be processed with the same index attribute tags, and obtain the collected index data; the first push unit 121 includes: a queue cache subunit 1211 configured as an index proxy component Cache the collected index data into the index queue to be sent; the queue pushing subunit 1212 is configured to push the collected index data in the index queue to be sent to the monitoring and collection component.
  • the indicator collection module 11 includes: a random generation unit 113 configured to establish a connection with a service endpoint in the edge cluster through a transceiver component, obtain endpoint information of the service endpoint, and generate a connection random number based on the endpoint information;
  • the connection matching unit 114 is configured to acquire a connection establishment range; when the connection random number is within the connection establishment range, receive the collection index data sent by the service endpoint in the edge cluster.
  • the data compression module 14 includes: a down-sampling unit 141 configured to perform down-sampling processing on the conversion index data to obtain the down-sampling index data; an index merging unit 142 configured to obtain conversion and storage of the conversion index data period, the downsampling indicator data with the same converted storage period are combined to obtain the storage indicator data.
  • the number of database components is N, and N is a positive integer;
  • the data storage module 15 includes: a hash generation unit 151 configured to obtain the storage index name and the storage index attribute label of the storage index data, based on the storage The index name and the storage index attribute label generate the storage hash value of the storage index data;
  • the hash storage unit 152 is configured to determine the storage fragment identification based on the storage hash value, and write the storage index data into the database corresponding to the storage fragment identification in the component.
  • the data storage module 15 includes: a cold and hot separation unit 153 configured as a database component to perform cold and hot analysis on the stored index data to obtain the hot index data and cold index data included in the stored index data;
  • the storage unit 154 is configured to determine the hot data storage area and the cold data storage area in the storage disk, write the hot index data into the hot data storage area, and write the cold index data into the cold data storage area.
  • the indicator collection module 11 includes: an indicator screening unit 115 configured to receive the initial indicator data sent by the edge cluster through the transceiver component; obtain the type of indicator to be collected, and based on the initial indicator data obtained from the initial indicator data The indicator data corresponding to the indicator type generates and collects indicator data.
  • the device also includes: an index query module 16 configured to receive the index query information sent by the user equipment in the central cluster, and send the index query information to the index query component based on the index query information An index query request, wherein the index query request includes index query information; the request fragmentation module 17 is configured to split the index query request into M fragmentation query requests based on the index query component, and M is a positive integer; the component determination module 18, It is configured to determine the query database component from the N database components based on the M fragment query requests; the fragment acquisition module 19 is configured to obtain the query index fragment data corresponding to the M fragment query requests respectively from the query database component; The result generation module 20 is configured to integrate the M pieces of query index fragment data to obtain an index query result, and send the index query result to the user equipment.
  • an index query module 16 configured to receive the index query information sent by the user equipment in the central cluster, and send the index query information to the index query component based on the index query information
  • An index query request wherein the index query request includes index query information
  • the indicator query module 16 includes: an information receiving unit 161 configured to receive the indicator query information sent by the user equipment based on the address of the visualized component of the indicator visualization component; a statement acquiring unit 162 configured to acquire the indicator query component address of the query component, and establish a monitoring panel based on the address of the query component, and obtain an indicator query statement corresponding to the indicator query information from the monitoring panel; the request sending unit 163 is configured to send an indicator query request to the indicator query component based on the indicator query statement.
  • the index query information includes the fragment query index names and fragment query index attribute labels corresponding to the M fragment query requests;
  • the component determination module 18 includes: a hash determination unit 181 configured to be based on M Fragmentation query index names and fragmentation query index attribute labels corresponding to the fragmentation query requests respectively, obtain the fragmentation query hash values corresponding to M fragmentation query requests respectively;
  • the component determination unit 182 is configured to obtain M fragmentation Query the fragment query identifiers corresponding to the hash values, and determine k query database components from the N database components based on the M fragment query identifiers, where k is a positive integer less than or equal to M.
  • the result generation module 20 includes: a data integration unit 20a configured to combine M query index fragment data to obtain index query data; a result generation unit 20b configured to establish a query based on index query information Coordinates: combine the query coordinates and index query data to obtain an index query result; the result sending unit 20c is configured to send the index query result to the user equipment, so that the user equipment displays the index query result.
  • An embodiment of the present application provides a data processing device, which can run in a central cluster, and integrates a transceiver component, a coordinated writing component, and a database component in the central cluster, and the transceiver component compresses the acquired data to Reduce the amount of data and send the compressed data to the coordinated writing component, which can realize data transmission more efficiently, which is conducive to performance optimization.
  • the obtained storage index data is written into the database component, which is scalable based on the database component The characteristics make storage scalable and improve data processing efficiency.
  • Fig. 12 is a schematic structural diagram of a computer device (running the above-mentioned sending and receiving component, coordinated writing component, database component, indicator query component and indicator visualization component) provided by the embodiment of the present application.
  • the computer device in this embodiment of the present application may include: one or more processors 1201 , a memory 1202 and an input/output interface 1203 .
  • the processor 1201 , memory 1202 and input/output interface 1203 are connected through a bus 1204 .
  • the memory 1202 is used to store computer programs, the computer programs include program instructions, and the input and output interface 1203 is used to receive data and output data, such as for data interaction between the central cluster and the edge cluster, or for data interaction between the central cluster and user equipment Data exchange between, or data exchange between various components in the central cluster; the processor 1201 is used to execute the program instructions stored in the memory 1202.
  • processor 1201 may perform the following operations:
  • the collection index data sent by the edge cluster is received through the transceiver component; the collection index data is pre-aggregated based on the transceiver component to obtain the pre-aggregated index data, and the pre-aggregated index data is sent to the coordination writing component; the coordination writing component will pre-aggregate
  • the index data is converted into the converted index data with the target storage format, and the converted index data is merged to obtain the stored index data; the stored index data is written into the database component; the database component writes the stored index data into the storage disk.
  • the processor 1201 can be a central processing unit (CPU, Central Processing Unit), and the processor can also be other general-purpose processors, digital signal processors (DSP, Digital Signal Processor), dedicated integrated Circuit (ASIC, Application Specific Integrated Circuit), off-the-shelf programmable gate array (FPGA, Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • DSP digital signal processors
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the memory 1202 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1201 and the input-output interface 1203 .
  • a portion of memory 1202 may also include non-volatile random access memory.
  • memory 1202 may also store device type information.
  • the computer device can execute the implementation provided by each step in FIG. 4 through its built-in functional modules.
  • the implementation provided by each step in FIG. 4 will not be repeated here.
  • the embodiment of the present application provides a computer device, including: a processor, an input and output interface, and a memory, and the processor acquires a computer program in the memory, executes each step of the method shown in FIG. 4 , and performs data processing operations.
  • the transceiver component, coordinated writing component and database component are integrated.
  • the transmitting and receiving component can compress the acquired data to reduce the amount of data, and send the compressed data to the coordinated writing component, which can be more Efficient data transmission is conducive to performance optimization.
  • the obtained storage index data is written into the database component. Based on the scalable characteristics of the database component, the storage is scalable and the data processing efficiency is improved.
  • the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by the processor and executing the data processing method provided by each step in FIG. 4 , which can Refer to the implementation manner provided by each step in FIG. 4 , which will not be repeated here.
  • the description of the beneficial effect of adopting the same method will not be repeated here.
  • a computer program can be deployed to be executed on one computer device, or on multiple computer devices at one site, or distributed across multiple sites and interconnected by a communication network implement.
  • the computer-readable storage medium may be the data processing apparatus provided in any of the foregoing embodiments or an internal storage unit of the computer device, such as a hard disk or memory of the computer device.
  • the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, Flash card (Flash Card), etc.
  • the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device.
  • the computer-readable storage medium is used to store the computer program and other programs and data required by the computer device.
  • the computer-readable storage medium can also be used to temporarily store data that has been output or will be output.
  • the embodiment of the present application also provides a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium.
  • the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various modes in FIG.
  • the receiving and sending components can compress the acquired data to reduce the amount of data, and send the compressed data to the coordinated writing components, which can realize data storage more efficiently. Transmission is conducive to performance optimization.
  • the obtained storage index data is written into the database component. Based on the scalable characteristics of the database component, the storage is scalable and the data processing efficiency is improved.
  • These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions
  • the device implements the functions specified in one or more blocks of the flowchart and/or one or more blocks of the structural schematic diagram.
  • These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby
  • the instructions provide steps for implementing the functions specified in one or more steps of the flowchart and/or one or more blocks in the structural illustration.
  • the modules in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Human Computer Interaction (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种数据处理方法、装置、系统、计算机设备、可读存储介质及计算机程序产品，涉及云技术领域及区块链技术，该方法包括：通过收发组件接收边缘集群发送的采集指标数据（S401）；基于收发组件对采集指标数据进行预聚合处理，得到预聚合指标数据，将预聚合指标数据发送至协调写入组件（S402）；协调写入组件将预聚合指标数据转换成具有目标存储格式的转换指标数据，并对转换指标数据进行归并处理，得到存储指标数据（S403）；将存储指标数据写入数据库组件中，数据库组件将存储指标数据写入存储磁盘（S404）。

Description

一种数据处理方法、装置、系统、计算机设备、可读存储介质及计算机程序产品
相关申请的交叉引用
本申请实施例基于申请号为202110772159.5、申请日为2021年07月08日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请实施例作为参考。
技术领域
本申请涉及计算机技术领域，尤其涉及一种数据处理方法、装置、系统、计算机设备、可读存储介质及计算机程序产品。
背景技术
在一些应用场景(如视频直播点播或云游戏等)中，为了减少延迟，会将服务部署到更靠近用户的地方(即边缘节点)，这些边缘节点组成了边缘集群，在边缘集群上需要监控部署的服务的性能和稳定性等，因此，会采集应用程序中所产生的指标数据，以对服务的性能和稳定性等进行检测。其中，一般是由每个边缘集群中的服务端点定时拉取并存储指标数据，当用户设备需要获取指标数据时，由中央集群从边缘集群的服务端点中拉取所需的指标数据，再反馈至用户设备。由于在服务端点中会产生较多的数据，因此，一般会保存一段时间的指标数据，超过一段时间后会删除数据，以保障服务端点的正常运行，使得数据存储的性能较差，且无法进行有效的数据传输，不利于存储长生命周期的数据。
发明内容
本申请实施例提供了一种数据处理方法、装置、系统、计算机设备、可读存储介质及计算机程序产品，可以通过传输预聚合指标数据实现高效的数据传输。
本申请实施例提供了一种数据处理方法，应用于中央集群，所述中央集群包括多个计算机设备，每个所述计算机设备用于运行收发组件、协调写入组件及数据库组件中的至少一种；
所述方法包括：
通过所述收发组件接收边缘集群发送的采集指标数据；
基于所述收发组件对所述采集指标数据进行预聚合处理，得到预聚合指标数据，并将所述预聚合指标数据发送至所述协调写入组件；
所述协调写入组件将所述预聚合指标数据转换成具有目标存储格式的转换指标数据，并对所述转换指标数据进行归并处理，得到存储指标数据；
将所述存储指标数据写入所述数据库组件中；
所述数据库组件将所述存储指标数据写入存储磁盘。
本申请实施例提供了一种数据处理系统，所述数据处理系统包括收发节点、协调写入节点及数据库节点，所述收发节点、所述协调写入节点及所述数据库节点分别是通过至少一个计算机设备实现的；
所述收发节点，用于接收边缘集群发送的采集指标数据；
所述收发节点，还用于对所述采集指标数据进行预聚合处理，得到预聚合指标数据， 并将所述预聚合指标数据发送至所述协调写入节点；
所述协调写入节点，用于将所述预聚合指标数据转换成具有目标存储格式的转换指标数据，并对所述转换指标数据进行归并处理，得到存储指标数据；
所述协调写入节点，还用于将所述存储指标数据写入所述数据库节点中；
所述数据库节点，用于将所述存储指标数据写入存储磁盘。
本申请实施例提供了一种数据处理装置，应用于计算机设备，计算机设备运行有收发组件、协调写入组件及数据库组件，该装置包括：
指标采集模块，配置为通过所述收发组件接收边缘集群发送的采集指标数据；
数据推送模块，配置为基于所述收发组件对所述采集指标数据进行预聚合处理，得到预聚合指标数据，并将所述预聚合指标数据发送至所述协调写入组件；
数据转换模块，配置为基于协调写入组件将所述预聚合指标数据转换成具有目标存储格式的转换指标数据；
数据压缩模块，配置为对所述转换指标数据进行归并处理，得到存储指标数据；
数据存储模块，配置为将所述存储指标数据写入所述数据库组件中，所述数据库组件将所述存储指标数据写入存储磁盘。
本申请实施例提供了一种计算机设备，包括处理器、存储器、输入输出接口；
处理器分别与存储器和输入输出接口相连，其中，输入输出接口用于接收数据及输出数据，存储器用于存储计算机程序，处理器用于调用该计算机程序，以使包含该处理器的计算机设备执行本申请实施例中的数据处理方法。
本申请实施例提供了一种计算机可读存储介质，计算机可读存储介质存储有计算机程序，该计算机程序适于由处理器加载并执行，以使得具有该处理器的计算机设备执行本申请实施例中的数据处理方法。
本申请实施例提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行本申请实施例中的各种方式中提供的方法。
实施本申请实施例，将具有如下有益效果：
在本申请实施例中，在中央集群中运行有收发组件、协调写入组件及数据库组件等，并通过收发组件对获取到的数据进行预聚合，并将预聚合指标数据传输至协调写入组件，以减少数据量，从而实现高效的数据传输，有利于优化数据存储的性能，提高数据的处理效率；与相关技术不同，本申请实施例先将得到的存储指标数据写入数据库组件中，再并由数据库组件将存储指标数据写入存储磁盘，从而通过数据库组件中转存储指标数据，无需保存一段时间后删除存储指标数据，以提高数据存储的生命周期。
附图说明
图1是本申请实施例提供的一种数据处理的网络交互架构图；
图2是本申请实施例提供的一种数据处理架构图；
图3是本申请实施例提供的一种数据处理场景示意图；
图4是本申请实施例提供的一种数据存储的方法流程图；
图5是本申请实施例提供了一种数据查询的方法流程图；
图6是本申请实施例提供的一种指标查询场景示意图；
图7是本申请实施例提供的一种数据处理系统的架构示意图；
图8是本申请实施例提供的一种监控收集节点的架构图；
图9是本申请实施例提供的另一种数据处理系统架构示意图；
图10是本申请实施例提供的一种云游戏部署场景示意图；
图11是本申请实施例提供的一种数据处理装置示意图；
图12是本申请实施例提供的一种计算机设备的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
例如，本申请实施例可以采用云技术及区块链技术等实现数据的传输及存储等。
其中，云存储(Cloud Storage)是在云计算概念上延伸和发展出来的一个新的概念，分布式云存储系统(以下简称存储系统)是指通过集群应用、网格技术以及分布存储文件系统等功能，将网络中大量各种不同类型的存储设备(存储设备也称之为存储节点)通过应用软件或应用接口集合起来协同工作，共同对外提供数据存储和业务访问功能的一个存储系统。
存储系统的存储方法为：创建逻辑卷，在创建逻辑卷时，为每个逻辑卷分配物理存储空间，该物理存储空间可能是某个存储设备或者某几个存储设备的磁盘组成的。客户端在某一逻辑卷上存储数据，也就是将数据存储在文件系统上，文件系统将数据分成许多部分，每一部分是一个对象，对象不仅包含数据而且还包含数据标识(ID，ID Entity)等额外的信息，文件系统将每个对象分别写入该逻辑卷的物理存储空间，且文件系统会记录每个对象的存储位置信息，从而当客户端请求访问数据时，文件系统能够根据每个对象的存储位置信息让客户端对数据进行访问。
存储系统为逻辑卷分配物理存储空间的过程为：按照对存储于逻辑卷的对象的容量估量(该估量往往相对于实际要存储的对象的容量有很大余量)和独立冗余磁盘阵列(RAID，Redundant Array of Independent Disk)的组别，预先将物理存储空间划分成分条，一个逻辑卷可以理解为一个分条，从而为逻辑卷分配了物理存储空间。本申请实施例可以基于上述的云存储技术实现对采集到的指标数据的存储。
例如，本申请实施例的数据处理系统包括中央集群及边缘集群，该边缘集群可以包括至少一个服务端点，每个服务端点中可能会产生较多的指标数据，因此，本申请实施例也可以基于大数据实现对指标数据的处理。其中，大数据(Big Data)是指无法在一定时间范围内采用常规软件工具进行捕捉、管理和处理的数据集合，是需要新处理模式才能具有更强的决策力、洞察发现力和流程优化能力的海量、高增长率和多样化的信息资产。
在一些实施例中，区块链是分布式数据存储、点对点传输、共识机制、加密算法等计算机技术的新型应用模式。区块链(Block Chain)，本质上是一个去中心化的数据库，是一串使用密码学方法相关联产生的数据块，每一个数据块中包含了一批次网络交易的信息，用于验证其信息的有效性(防伪)和生成下一个区块。区块链可以包括区块链底层平台、平台产品服务层以及应用服务层。
本申请实施例可以基于云技术及区块链技术，实现对指标数据的处理。本申请实施例可以应用于云游戏领域，对云游戏所产生的指标数据进行监测，云游戏(Cloud Gaming)又可称为游戏点播(Gaming on Demand)，是一种以云计算技术为基础的在线游戏技术。云游戏技术使图形处理与数据运算能力相对有限的轻端设备(Thin Client)能运行高品质游戏。在云游戏场景下，游戏并不在玩家游戏终端，而是在云端服务器中运行，并由云端服务器将游戏场景渲染为视频音频流，通过网络传输给玩家游戏终端。玩家游戏终 端无需拥有强大的图形运算与数据处理能力，仅需拥有基本的流媒体播放能力与获取玩家输入指令并发送给云端服务器的能力即可。例如，本申请实施例还可以应用于应用程序监测领域，监测应用程序中所产生的指标数据等，在此不做限制。其中，该应用程序可以是任意一种应用，如视频应用程序、文件编辑应用程序及音乐应用程序等。
在本申请实施例中，请参见图1，图1是本申请实施例提供的一种数据处理的网络交互架构图，本申请实施例可以由中央集群101实现。其中，该中央集群101是关联的应用程序所在的多个计算机设备组成的集群，可以部署与应用程序相关的服务，用于管理一个或多个边缘集群，可以管理各个边缘集群中的服务端点，可以向边缘集群提供服务，该中央集群101可以与边缘集群102进行数据交互，该边缘集群102可以包括一个或至少两个服务端点，如服务端点102a、服务端点102b及服务端点102c等，每个服务端点可以认为是一个边缘节点，其中，边缘集群102包括相较于中央集群更靠近用户的机房或节点，部署服务的机房或节点称为服务端点或边缘节点，通过边缘集群中的服务端点定时拉取存储指标数据，相对于中央集群，边缘集群与用户设备之间的传输距离更近。中央集群101可以从任意一个服务端点中获取采集指标数据，该采集指标数据为服务端点中所产生的指标数据，或者是由服务端点监测到的指标数据。中央集群101可以对获取到的采集指标数据进行压缩处理及转换等，得到存储指标数据(用于存储的指标数据)，并将该存储指标数据写入中央集群所包括的数据库组件中。
需要说明的是，中央集群201包括多个计算机设备，其中每个计算机设备通过运行至少一个具有特定功能模块的组件(例如收发组件(具有收发数据的功能)、协调写入组件(具有写入数据的功能)、数据库组件(具有将数据写入磁盘的功能)、指标查询组件(具有查询指标的功能)及指标可视化组件(具有可视化指标的功能))来完成本申请实施例所提供的数据处理方法。其中，运行有相同组件的至少一个计算机设备构成中央集群中的一个节点，例如运行有收发组件的至少一个计算机设备构成中央集群中的收发节点，运行有协调写入组件的至少一个计算机设备构成中央集群中的协调写入节点，运行有数据库组件的至少一个计算机设备构成中央集群中的数据库节点，运行有指标查询组件的至少一个计算机设备构成中央集群中的指标查询节点，运行有指标可视化组件的至少一个计算机设备构成中央集群中的指标可视化节点。
在一些实施例中，中央集群101可以接收用户设备(如用户设备103a、用户设备103b或用户设备103c等)发送的指标查询信息，基于该指标查询信息从数据库组件中获取到指标查询结果，将该指标查询结果发送至该用户设备。
请参见图2，图2是本申请实施例提供的一种数据处理架构图。如图2所示，该数据处理系统包括中央集群201、边缘集群202及用户设备203，其中，该中央集群201包括多个计算机设备，每个计算机设备用于运行收发组件2011、协调写入组件2012、数据库组件2013、指标查询组件2014及指标可视化组件2015中的至少一种，其中，该边缘集群202可以包括至少一个服务端点。
例如，通过收发组件2011接收边缘集群202发送的采集指标数据，收发组件2011对采集指标数据进行预聚合处理，得到预聚合指标数据，并将预聚合指标数据发送至协调写入组件2012，协调写入组件2012将预聚合指标数据转换成具有目标存储格式的转换指标数据，并对转换指标数据进行归并处理，得到存储指标数据，将存储指标数据写入数据库组件2013中，数据库组件2013将存储指标数据写入存储磁盘。中央集群201还可以接收用户设备203发送的指标查询信息，基于该指标查询信息从数据库组件2013中获取到指标查询结果，将该指标查询结果发送至该用户设备203。
在一些实施例中，参见图3，图3是本申请实施例提供的一种数据处理场景示意图。如图3所示，中央集群301可以通过收发组件接收边缘集群302发送的采集指标数据， 该收发组件对采集指标数据进行预聚合处理，得到预聚合指标数据，将预聚合指标数据发送至协调写入组件。基于协调写入组件对预聚合指标数据进行格式转换，具体是将该预聚合指标数据转换成具有目标存储格式的转换指标数据，对转换指标数据进行归并处理，得到存储指标数据。
在一些实施例中，中央集群301可以将该存储指标数据写入数据库组件中，可以基于该数据库组件将存储指标数据写入存储磁盘。通过以上过程，在中央集群中集成了收发组件、协调写入组件及数据库组件等，并该收发组件可以对获取到的数据进行压缩，以减少数据量，并将压缩后的数据发送至协调写入组件，可以更为高效地实现数据的传输，有利于优化数据存储性能，同时将得到的存储指标数据写入数据库组件中，基于数据库组件将存储指标数据写入存储磁盘的特性，提高数据的处理效率。
在一些实施例中，中央集群301还可以接收用户设备303发送的指标查询信息，基于指标查询信息向指标查询组件发送指标查询请求，该指标查询请求包括指标查询信息。中央集群301可以基于指标查询组件将指标查询请求拆分为M个分片查询请求，M为正整数。基于M个分片查询请求，从N个数据库组件中获取M个分片查询请求分别对应的查询指标分片数据，将M个查询指标分片数据进行整合，得到指标查询结果，将指标查询结果发送至用户设备303。
可以理解的是，本申请实施例中所提及的中央集群、边缘集群及用户设备均可以是一种计算机设备，其中，中央集群可以是由一个计算机设备或至少两个计算机设备组成，边缘集群可以是由一个计算机设备或至少两个计算机设备组成，本申请实施例中的计算机设备包括但不限于终端设备或服务器。换句话说，计算机设备可以是服务器或终端设备，也可以是服务器和终端设备组成的系统。其中，以上所提及的终端设备可以是一种电子设备，包括但不限于手机、平板电脑、台式电脑、笔记本电脑、掌上电脑、车载设备、增强现实/虚拟现实(AR/VR，Augmented Reality/Virtual Reality)设备、头盔显示器、智能电视、可穿戴设备、智能音箱、数码相机、摄像头及其他具备网络接入能力的移动互联网设备(MID，Mobile Internet Device)，或者火车、轮船、飞行等场景下的终端设备等。其中，以上所提及的服务器可以是独立的物理服务器，也可以是多个物理服务器构成的服务器集群或者分布式系统，还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、车路协同、内容分发网络(CDN，Content Delivery Network)、以及大数据和人工智能平台等基础云计算服务的云服务器。
例如，本申请实施例中所涉及的数据可以存储在计算机设备中，或者可以基于云存储技术对该数据进行存储，在此不做限制。
在一些实施例中，请参见图4，图4是本申请实施例提供的一种数据存储的方法流程图。如图4所示，该方法由中央集群实现，中央集群包括多个计算机设备，每个计算机设备用于运行收发组件、协调写入组件、数据库组件、指标查询组件及指标可视化组件中中的至少一种，该数据存储过程包括如下步骤：
步骤S401，通过收发组件接收边缘集群发送的采集指标数据。
在本申请实施例中，该边缘集群可以包括至少一个服务端点，中央集群可以通过收发组件接收边缘集群中任意一个服务端点或任意多个(即至少两个)服务端点发送的采集指标数据。其中，该采集指标数据是指采集到的指标所对应的具体数据，该指标包括但不限于每秒的往返时延(RTT，Round-Trip Time)、丢包率及数据更新频率等，可以反映应用程序的性能，该采集指标数据是边缘集群中的一个服务端点或任意多个服务端点，从关联的用户设备(即使用应用程序的用户设备)中采集到的指标对应的数据，例如，针对指标“RTT”，边缘集群采集到该指标“RTT”对应的采集指标数据可以是“0.5秒” 等。
其中，收发组件是指可以从边缘集群中获取数据，并将获取到的数据发送至其他组件的组件，即该收发组件是指可以接收数据及推送数据的组件。
在一些实施例中，该收发组件可以包括指标代理组件，中央集群可以通过该指标代理组件与边缘集群建立网络长连接，通过网络长连接接收边缘集群发送的采集指标数据。或者，该收发组件包括指标代理组件及监控收集组件，中央集群可以通过收发组件所包括的指标代理组件与边缘集群建立网络长连接，通过网络长连接接收边缘集群发送的采集指标数据，并将采集指标数据发送至监控收集组件。
例如，该指标代理组件相当于一个接入网关，如MetricProxy组件，该指标代理组件可以接收边缘集群的传输控制协议(TCP，Transmission Control Protocol)连接，保持与边缘集群之间的连接和心跳，即实现与边缘集群建立网络长连接，可以接收该边缘集群发送的采集指标数据，并可以将处理后的采集指标数据推送至中央集群中的其他组件(如监控收集组件等)。
例如，该监控收集组件可以认为是集成有预聚合功能的组件，如Prometheus组件，可以接收指标代理组件的指标推送请求，获取指标代理组件推送的采集指标数据，对该采集指标数据进行预聚合处理，例如，可以将获取到的数据缓存至该监控收集组件中。
例如，该网络长连接是指指标代理组件与边缘集群之间保持长时间(设定时间阈值)的网络连接，可以发送多个数据包，如传输控制协议(TCP，Transmission Control Protocol)长连接或超文本传输协议(HTTP，Hypertext Transfer Protocol)长连接等，在此不做限制，其中，可以通过心跳保持该网络长连接，例如，以边缘集群中的一个服务端点为例，该服务端点与指标代理组件之间通过发送心跳数据包，保持服务端点与指标代理组件之间的网络长连接。例如，由于HTTP长连接每次会有建链断链的开销。
为了减少建链断链等带来的开销，提高吞吐量，可以使用除HTTP长连接之外的长连接(如TCP长连接等)建立指标代理组件与边缘集群之间的网络长连接，换句话说，不同协议类型的长连接具有不同的特性，在建立指标代理组件与边缘集群建立网络长连接时，可以获取符合实际需求的目标连接特性，获取该目标连接特性所对应的协议类型，基于该协议类型在指标代理组件与边缘集群之间建立网络长连接，其中，该目标连接特性是指根据需要达到的连接效果的需求(例如建链断链开销小、连接维持开销小)所确定的特性。例如，若该目标连接特性为建链断链的开销小的特性(即需要达到建链断链开销小的需求)，则可以将TCP确定为该目标连接特性所对应的协议类型，基于该协议类型在指标代理组件与边缘集群之间建立TCP长连接，使得指标代理组件与边缘集群之间的连接不会经常建链断链，可以减少用于建链断链的开销；若该目标连接特性为连接维持开销小的特性(即需要达到连接维持开销小的需求)，则可以将HTTP确定为该目标连接特性所对应的协议类型，基于该协议类型在指标代理组件与边缘集群之间建立HTTP长连接，使得指标代理组件与边缘集群之间的连接在空闲时可以断开等，在此不做限制。其中，该指标代理组件相当于一个网络接入网关，可以接收边缘集群的网络连接(包括网络长连接及网络短连接等)，可以保持与边缘集群之间的网络连接及心跳。
在一些实施例中，中央集群通过网络长连接接收边缘集群发送的采集指标数据可以通过以下方式实现：通过网络长连接接收边缘集群发送的待处理指标数据；指标代理组件获取待处理指标数据的指标属性标签，将具有相同的指标属性标签的待处理指标数据进行归并处理，得到采集指标数据。
其中，该指标属性标签用于表示该待处理指标数据对应的指标属性，可以是该待处理指标数据自身的属性标签，如该待处理指标数据对应的单位属性标签(如时间属性标签、距离属性标签或速率属性标签等)等，也可以是该待处理指标数据的关联对象的属 性标签，例如，该待处理指标数据是对应用程序等进行监测得到的指标数据，该待处理指标数据的指标属性标签可以是该应用程序在产生待处理指标数据时关联的用户的属性标签，如年龄标签或性别标签等，在此不做限制。
例如，边缘集群可以获取待处理指标数据，以及待处理指标数据的指标信息，该指标信息可以包括f个指标属性信息，指标属性信息包括但不限于待处理指标数据的指标属性标签、指标名称及指标采集时间等，f为正整数；边缘集群将获取到的待处理指标数据及该待处理指标数据的指标信息发送至指标代理组件。例如，边缘集群可以获取应用程序中所产生的应用数据，对该应用数据进行打点处理，得到待处理指标数据，该打点处理是指对应用程序中所产生的应用数据进行整理，也就是将获取到的应用数据中的指标信息及指标数据等进行关联处理。其中，边缘集群可以获取目标指标所对应的待处理指标数据，该目标指标包括但不限于每秒的往返时延(Round-Trip Time，RTT)、丢包率及数据更新频率等。其中，该待处理指标数据可以通过图表、文本或列表等进行表示，如下表1所示：
表1
Figure PCTCN2022097848-appb-000001
如上表1所示，表1是待处理指标数据及该待处理指标数据的指标信息的一种表现形式，表1表示中央集群基于指标代理组件接收到待处理指标数据，该待处理指标数据的数量为d个，d为正整数，d个待处理指标数据包括指标数据1、指标数据2及指标数据3等，其中，指标数据1的指标名称为RTT，指标属性标签为“属性标签a1及属性标签b1等”，指标采集时间为采集时间1，…；指标数据2的指标名称为RTT，指标属性标签为“属性标签a2及属性标签b1等”，指标采集时间为采集时间1，…；指标数据3的指标名称为丢包率，指标属性标签为“属性标签a1及属性标签b2等”，指标采集时间为采集时间2，…；指标数据4的指标名称为丢包率，指标属性标签为“属性标签a2及属性标签b1等”，指标采集时间为采集时间1，…。例如，该待处理指标数据可以通过“<指标名称>{<指标属性标签>＝<待处理指标数据>，…}”的格式进行表示，也就是该待处理指标数据可以是以指标名称进行分组表示的，例如，表1中的待处理指标数据可以表示为“<RTT>{<属性标签a1，属性标签b1…>＝<指标数据1>，<属性标签a2，属性标签b1…>＝<指标数据2>…}；<丢包率>{<属性标签a1，属性标签b2…>＝<指标数据3>，<属性标签a2，属性标签b1…>＝<指标数据4>…}”。其中，指标名称一般用于反映被监控样本的含义，例如，http_request_total用于表示当前系统接收到的HTTP请求总量；指标属性标签用于表示当前样本(即待处理指标数据)的特征维度，通过这些维度可以对样本数据进行过滤及聚合等处理。
例如，中央集群可以基于指标代理组件获取待处理指标数据的指标属性标签，将具有相同的指标属性标签的待处理指标数据进行归并处理，得到采集指标数据，例如，该指标属性标签包括年龄属性标签及性别属性标签，将具有相同的年龄属性标签及性别属性标签的待处理指标数据进行归并处理，得到采集指标数据，减少指标数据的存储量。其中，以表1中的待处理指标数据为例，得到的采集指标数据可以参见表2所示：
表2
Figure PCTCN2022097848-appb-000002
如表1可知，指标数据2与指标数据4具有相同的指标属性标签“属性标签a2及属性标签b1”，将待处理指标数据中的指标数据2及指标数据4进行归并处理，得到采集指标数据，该采集指标数据可以参见表2所示。例如，中央集群可以基于指标名称及指标属性标签对待处理指标数据进行归并处理，即，将具有相同的指标名称及指标属性标签的待处理指标数据进行归并处理，得到采集指标数据，此时，由于指标数据2及指标数据4的指标名称不同，因此，不会对该指标数据2及指标数据4进行归并处理，该采集指标数据可以通过“<指标名称>{<指标属性标签>＝<待处理指标数据，…>，…}”进行表示，其中，“<待处理指标数据，…>”表示一组具有相同的指标名称及指标属性标签的待处理指标数据，在这一情况下，采集指标数据可以如表3所示：
表3
Figure PCTCN2022097848-appb-000003
例如，中央集群可以获取集群资源状态，若该集群资源状态为集群资源充足状态，则中央集群可以直接接收边缘集群中的服务端点发送的采集指标数据；若该集群资源状态为集群资源缺乏状态，则中央集群可以生成边缘集群中的服务端点对应的连接随机数，基于该连接随机数获取该服务端点所发送的采集指标数据。例如，中央集群可以通过收发组件与边缘集群中的服务端点建立连接，并获取服务端点的端点信息，基于端点信息生成连接随机数。
其中，该连接随机数是一个用于表示服务端点与中央集群建立连接的随机数，其中，该连接随机数位于随机数区间内的一个随机数，该随机数区间是基于连接建立范围所确定的，也就是说，中央集群可以根据端点信息生成一个位于随机数区间内的连接随机数，用于表示中央集群是否接收该服务端点发送的指标数据。该端点信息可以包括服务端点的端点标识、端点内存空间及端点传输带宽等，中央集群可以根据端点信息中的一个或任意多个元素生成连接随机数。例如，中央集群可以直接采用随机数生成函数(如随机函数(rand函数)或随机种子函数(srand)等)，对端点信息中的一个或任意多个元素进行处理，生成连接随机数；或者，中央集群可以获取端点信息中的一个或任意多个元素对应的元素哈希值，对该元素哈希值进行取余计算，得到连接随机数，例如，可以从元素哈希值中获取指定位置范围处的哈希值，对指定位置范围处的哈希值进行取余计算，得到连接随机数等，在此不做限制。其中，收发组件可以包括指标代理组件，或者可以 包括指标代理组件及监控收集组件，两种情况下，中央集群均是通过收发组件中的指标代理组件与边缘集群中的服务端点建立连接，生成连接随机数。例如，中央集群可以根据端点信息生成连接随机数，或者可以根据端点信息及系统网络时间生成连接随机数，或者可以根据端点信息的哈希值生成连接随机数等，在此不做限制。
在一些实施例中，获取连接建立范围，其中，该连接建立范围可以认为是一种随机数阈值范围，用于实现百分比丢弃请求，使得中央集群可以对位于连接建立范围的随机数所对应的服务端点发送的指标数据进行处理，对位于连接建立范围外的随机数所对应的服务端点发送的指标数据进行丢弃，例如，当连接随机数位于连接建立范围时，接收边缘集群中的服务端点发送的采集指标数据；当连接随机数不位于连接建立范围时，获取接收到的边缘集群中的服务端点所发送的历史指标数据，丢弃该历史指标数据，即丢弃与该连接随机数对应的服务端点相关联的所有指标数据，以保障指标数据的完整性，例如，可以向边缘集群中的服务端点发送指标推送失败消息。
举例来说，中央集群可以基于指标接收比例确定连接建立范围，假定该指标接收比例为70％，该连接随机数的范围为1～100，则该连接建立范围为1～100中的任意连续的70％的数，例如该连接建立范围可以是1～70，即，中央集群可以生成一个位于1～100的该服务端点对应的连接随机数，若该连接随机数属于1～70，则接收边缘集群中的服务端点发送的采集指标数据；若连接随机数不属于1～70，则丢弃与边缘集群中的服务端点所关联的指标数据，例如，可以向边缘集群中的服务端点发送指标推送失败消息。例如，存在服务端点A及服务端点B，假定服务端点A对应的连接随机数位于连接建立范围，服务端点B对应的连接随机数不位于连接建立范围，则中央集群可以获取该服务端点B所发送的历史指标数据，丢弃该服务端点B所发送的历史指标数据。其中，当服务端点对应的连接随机数不位于连接建立范围时，中央集群可以只丢弃该服务端点发送的历史指标数据，而保持与该服务端点之间的网络连接。通过该过程，可以使得中央集群接收每个服务端点的指标数据的概率相同，同时，可以对各个服务端点按照百分比进行丢弃，使得中央集群的集群资源状态为集群资源缺乏状态时，可以在一定程度上减少需要处理的指标数据，提高中央集群的资源利用效率。
例如，中央集群可以基于边缘集群的指标推送量，接收边缘集群中的服务端点发送的采集指标数据，其中，该指标推送量是指边缘集群推送到中央集群的指标数据的总量，即边缘集群推送到中央集群的指标数据所占的资源数量。例如，中央集群可以监测边缘集群的指标推送量，当该指标推送量小于或等于指标处理量阈值时，则中央集群可以接收边缘集群中的服务端点发送的采集指标数据；当该指标推送量大于指标处理量阈值时，则表示中央集群接收到的指标数据的数据量超过自身可以处理的最大数据量，可以对获取到的指标数据进行筛选，例如，可以通过收发组件与边缘集群中的服务端点建立连接，获取服务端点的端点信息，根据端点信息生成连接随机数，基于连接随机数确定是否接收采集指标数据的过程，以减少中央集群需要进行处理的指标数据的数据量，减少中央集群的资源压力。
在一些实施例中，中央集群还可以通过收发组件接收边缘集群发送的初始指标数据，获取待采集指标类型，并基于从初始指标数据中获取的待采集指标类型对应的指标数据，生成采集指标数据。
其中，该初始指标数据可以认为是由边缘集群直接发送的指标数据。其中，该待采集指标类型用于表示指标代理组件需要接收的指标数据的类型。例如，中央集群可以通过收发组件接收边缘集群发送的初始指标数据，得到待采集指标类型，将从初始指标数据中获取待采集指标类型对应的指标数据，作为待处理指标数据，并对待处理指标数据进行归并处理，得到采集指标数据。其中，本申请实施例中的各个指标数据是指中央集 群从边缘集群中获取到指标数据后，在不同的组件中进行处理后得到的不同阶段的指标数据，也就是说，不同的指标数据的名称用于表示指标数据在不同阶段，经过不同处理后得到的指标数据的名称。
例如，为了减少客户端(即边缘集群中的服务端点)接入的复杂性，可以将指标交互逻辑封装成一个指标静态库(lib)，该静态数据库会默认收集边缘集群所获取到的各种指标数据，而有些指标类型的指标数据是不必要的，中央集群可以基于待采集指标类型对获取到的初始指标数据进行筛选，以去掉不相关的指标数据，从而减少数据量，减少指标代理组件对指标数据的推送压力。
步骤S402，基于收发组件对采集指标数据进行预聚合处理，得到预聚合指标数据，将预聚合指标数据发送至协调写入组件。
在本申请实施例中，当指标属性标签组合的指标数据的数据量大于数据量阈值时，在查询并聚合该指标属性标签对应的指标数据时，会产生较大的耗时，预聚合是指在对指标数据写入前或写入后对该指标属性标签对应的指标数据进行聚合计算，存储为另一个指标名称，以减少查询耗时，并节省资源。其中，该指标数据是指本申请实施例中采集指标数据中需要进行预聚合处理的指标数据，如采集指标数据等。其中，收发组件集成有第一数据推动功能，第一数据推动功能用于通过收发组件将预聚合指标数据发送至协调写入组件。
其中，若该收发组件包括指标代理组件，则指标代理组件对采集指标数据进行预聚合处理，得到预聚合指标数据，将预聚合指标数据发送至协调写入组件。若该收发组件包括指标代理组件及监控收集组件，指标代理组件将采集指标数据发送至监控收集组件。指标代理组件可以自由推送采集指标数据，有利于性能优化。监控收集组件获取采集指标数据的指标信息，并获取指标聚合属性，对与指标聚合属性匹配的指标信息所对应的采集指标数据进行指标转换(即执行指标转换算法)，得到预聚合指标数据。监控收集组件向协调写入组件发送远程写入请求，基于远程写入请求将预聚合指标数据发送至协调写入组件。其中，基于监控收集组件对预聚合指标数据进行存储，并基于远程写入请求将预聚合指标数据旁路发送至协调写入组件。
其中，该指标代理组件集成有第二数据推送功能，第二数据推送功能用于通过指标代理组件将采集指标数据推送至监控收集组件，监控收集组件集成有第三数据推送功能，第三数据推送功能用于通过监控收集组件将预聚合指标数据发送至协调写入组件。
其中，该指标聚合属性及指标转换算法可以是提前预设的，存储于监控收集组件中。由上述表1至表3所示，指标数据在各个组件中进行传输时会携带对应的指标信息。
其中，该指标聚合属性可以是指标属性标签中的任意一个或任意多个标签类型的指标属性标签，也可以是f个指标属性信息中的任意一个或任意多个指标属性信息，例如，可以对具有相同的指标聚合属性的采集指标数据执行指标转换算法，得到预聚合指标数据及该预聚合指标数据对应的预聚合指标名称，该指标转换算法可以是均值算法、极大值算法、极小值算法或出现次数最多的统计算法等，在此不做限制。
例如，可以将指标聚合属性及指标转换算法等称为指标预聚合规则，该指标预聚合规则可以根据实际需求进行更新。举例来说，存在一组指标名称为RTT的采集指标数据，假定指标聚合属性为指标属性标签中的性别属性标签，该指标转换算法为均值算法，获取到第一性别属性标签对应的采集指标数据有“1，2，1，0.5”，第二性别属性标签对应的采集指标数据有“2，3，2，3”，对与指标聚合属性相匹配的指标信息所对应的采集指标数据，执行指标转换算法，得到预聚合指标数据，可以得到第一性别属性标签对应的预聚合指标数据为“(1+2+1+0.5)/4＝1.125”，第二性别属性标签对应的预聚合指标数据为“(2+3+2+3)/4＝2.5”，例如，可以获取到第一性别属性标签对应的预聚合指标名称， 如“第一性别RTT均值”，以及第二性别属性标签对应的预聚合指标名称，如“第二性别RTT均值”。
在一些实施例中，指标代理组件将采集指标数据推送至监控收集组件可以通过以下方式实现：指标代理组件将采集指标数据缓存至待发送指标队列中；指标代理组件将待发送指标队列中的采集指标数据推送至监控收集组件。
例如，若采集指标数据的数据量大于或等于数据推送阈值，则基于该数据推送阈值对待发送指标队列中的采集指标数据进行拆分，得到至少两个采集指标数据组，基于第二数据推送功能将至少两个采集指标数据组并行发送(Period Push)至监控收集组件；若采集指标数据的数据量小于数据推送阈值，则直接通过指标代理组件的第二数据推送功能及待发送指标队列，将采集指标数据推送至监控收集组件，从而提高对指标数据的推送效率。其中，该监控收集组件可以接收指标代理组件的指标推送请求，基于该指标推送请求将接收到的采集指标数据写入指标代理组件，并对该采集指标数据进行预聚合处理，得到预聚合指标数据，将该预聚合指标数据周期性发送至协调写入组件。
步骤S403，协调写入组件将预聚合指标数据转换成具有目标存储格式的转换指标数据，并对转换指标数据进行归并处理，得到存储指标数据。
在本申请实施例中，中央集群可以基于协调写入组件接收监控收集组件发送的远程写入请求，基于该远程写入请求将预聚合指标数据转换成具有目标存储格式的转换指标数据，其中，该目标存储格式是指中央集群中的数据库组件所支持存储的数据格式，例如，假定该数据库组件为M3DB，则该数据库组件对应的目标存储格式为m3格式。
在一些实施例中，对转换指标数据进行归并处理可以通过以下方式实现：基于协调写入组件对转换指标数据进行降采样处理，得到降采样指标数据，获取转换指标数据的转换存储期限，将具有相同的转换存储期限的降采样指标数据进行合并处理，得到存储指标数据。
其中，可以对转换指标数据进行周期性降采样处理，或者可以根据数据库组件对应的降采样方式对转换指标数据进行降采样处理，如m3对应的降采样方式。其中，该转换存储期限可以是指该转换指标数据的采集时间或产生时间(该转换指标数据对应的采集指标数据的产生时间)等。
步骤S404，将存储指标数据写入数据库组件中，数据库组件将存储指标数据写入存储磁盘。
例如，该数据库组件的数量为N，N为正整数。在将存储指标数据写入数据库组件中时，协调写入组件获取存储指标数据的存储指标名称及存储指标属性标签，基于存储指标名称及存储指标属性标签，生成存储指标数据的存储哈希值，基于存储哈希值确定存储分片标识，将存储指标数据写入存储分片标识对应的数据库组件中。
其中，在对指标数据进行预聚合处理、归并处理或转换处理等时，会对指标数据的指标名称及指标属性标签等指标信息进行同步处理。
在一些实施例中，将存储指标数据写入存储磁盘可以通过以下方式实现：数据库组件对存储指标数据进行冷热分析处理，得到存储指标数据包括的热指标数据及冷指标数据，确定存储磁盘中的热数据存储区域及冷数据存储区域，将热指标数据写入热数据存储区域，将冷指标数据写入冷数据存储区域。
其中，热指标数据是指需要被频繁访问的在线类指标数据，冷指标数据是指不经常访问的指标数据。例如，通过对存储指标数据进行冷热分析处理，实现对存储指标数据的冷热分离，可以使得该存储指标数据的存储更为规律，便于更好地管理指标数据，可以对不使用的存储指标数据进行清理，从而减少数据库组件的资源使用量。
在一些实施例中，请参见图5，图5是本申请实施例提供的一种数据处理方法流程 图。如图5所示，该方法由中央集群实现，中央集群中的计算机设备还可以运行指标查询组件等，该数据处理过程包括如下步骤：
步骤S501，接收用户设备发送的指标查询信息，基于指标查询信息向指标查询组件发送指标查询请求。
在本申请实施例中，该中央集群中的计算机设备还可以运行包括指标查询组件，其中，该数据库组件的数量为N，N为正整数，可根据需求设置数据库组件的数量，即数据库组件具有可扩展性。中央集群可以基于指标查询信息向指标查询组件发送指标查询请求，该指标查询请求包括所述指标查询信息。
在一些实施例中，该中央集群还可以运行指标可视化组件。例如，中央集群可以基于指标可视化组件的可视化组件地址，接收用户设备发送的指标查询信息；获取指标查询组件的查询组件地址，并基于查询组件地址建立监控面板，从监控面板中获取指标查询信息对应的指标查询语句；基于指标查询语句向指标查询组件发送指标查询请求。
例如，参见图6，图6是本申请实施例提供的一种指标查询场景示意图。如图6所示，中央集群可以基于指标可视化组件的可视化组件地址，接收用户设备601发送的指标查询信息，基于该指标查询信息建立监控面板603，基于该监控面板603生成指标查询信息对应的指标查询语句，根据指标查询语句向指标查询组件604发送指标查询请求。
例如，该指标查询语句也可以是由用户设备生成的。例如，用户设备可以获取指标可视化组件的可视化组件地址，访问该可视化组件地址，显示可视化组件访问页面，基于该可视化组件访问页面获取指标查询组件的查询组件地址，建立监控面板，获取该监控面板中所配置的查询参数，根据该查询参数生成指标查询语句，基于该指标查询语句向指标可视化组件发送指标查询请求，中央集群基于该指标可视化组件向指标查询组件发送该指标查询请求。
其中，该指标查询语句可以认为是一种查询表达式，该指标查询语句的语句类型是根据数据库组件所确定的，例如，该数据库组件为M3DB组件，则该指标查询语句的语句类型可以是普罗米修斯查询语言类型(PromQL，Prometheus Query Language)等。
步骤S502，指标查询组件将指标查询请求拆分为M个分片查询请求。
在本申请实施例中，指标查询组件基于指标查询信息，将该指标查询请求拆分为M个分片查询请求，例如，可以根据指标查询信息中所包括的查询指标名称及查询指标属性标签，将指标查询请求拆分为M个分片查询请求，M为正整数。
步骤S503，基于M个分片查询请求，从N个数据库组件中确定查询数据库组件，并从查询数据库组件中获取M个分片查询请求分别对应的查询指标分片数据。
在本申请实施例中，中央集群基于M个分片查询请求，并行向数据库组件发起查询，该指标查询信息包括M个分片查询请求分别对应的分片查询指标名称及分片查询指标属性标签。
在一些实施例中，基于M个分片查询请求，从N个数据库组件中确定查询数据库组件可以通过以下方式实现：基于M个分片查询请求分别对应的分片查询指标名称及分片查询指标属性标签，获取M个分片查询请求分别对应的分片查询哈希值；获取M个分片查询哈希值分别对应的分片查询标识，并基于M个分片查询标识，从N个数据库组件中确定k个查询数据库组件；k为小于或等于M的正整数。
其中，该分片查询哈希值对应的哈希算法与上述图4中的存储哈希值对应的哈希算法相同。例如，中央集群可以基于指标查询组件，从k个查询数据库组件中获取M个分片查询请求分别对应的查询指标分片数据(即M个查询指标分片数据)，将M个查询指标分片数据发送至指标可视化组件。从而，通过分片查询的方式提高数据查询的效率。
步骤S504，将M个查询指标分片数据进行整合，得到指标查询结果，将指标查询 结果发送至用户设备。
在本申请实施例中，中央集群可以基于指标可视化组件，将M个查询指标分片数据进行组合，得到指标查询数据；基于指标查询信息建立查询坐标，将查询坐标及指标查询数据进行组合，得到指标查询结果。
其中，该指标查询结果可以是包括查询坐标与指标查询数据的图标数据或曲线坐标等，也可以是包括指标查询信息及指标查询数据的数据等；将指标查询结果发送至用户设备，以使用户设备显示指标查询结果，使得用户设备可以直观地展示查询到的指标查询结果，提高指标数据查询的效果。例如，若该指标查询语句是由用户设备生成的，则用户设备还可以在监控面板中显示指标查询结果。
上述图4至图5是本申请实施例所提供的一种数据处理过程，包括对指标数据的采集存储及查询等过程，本申请实施例通过收发组件接收边缘集群发送的采集指标数据；基于收发组件中的数据推送功能，对采集指标数据进行预聚合处理，得到预聚合指标数据，将预聚合指标数据发送至协调写入组件；基于协调写入组件将预聚合指标数据转换成具有目标存储格式的转换指标数据，对转换指标数据进行归并处理，得到存储指标数据；将存储指标数据写入数据库组件中，基于数据库组件将存储指标数据写入存储磁盘。通过以上过程，在中央集群中集成了收发组件、协调写入组件及数据库组件等，并在收发组件中集成数据推送功能，使得该收发组件可以对获取到的数据进行压缩，以减少数据量，并将压缩后的数据发送至协调写入组件，可以更为高效地实现数据的传输，有利于优化数据存储性能，同时将得到的存储指标数据写入数据库组件中，基于数据库组件的特性，提高数据的处理效率。
参见图7，图7是本申请实施例提供的一种数据处理系统的架构示意图。如图7所示，该数据处理系统包括收发节点、协调写入节点及数据库节点，其中，在图7所示的数据处理系统中，该收发节点包括指标代理节点及监控收集节点，该数据处理系统是指中央集群701对应的系统。其中，该数据处理系统中的各个节点是指上述中央集群中运行有相同组件的至少一个计算机设备所构成的节点，即每个节点是通过至少一个计算机设备实现的。其中，图7所示的数据处理系统的结构中，对监控收集节点更新的成本(包括时间及精力等)较低，该监控收集节点支持预聚合处理的能力较强，可以提高对监控收集节点的更新效率及预聚合效率。
收发节点，用于接收边缘集群发送的采集指标数据收发节点，还用于对采集指标数据进行预聚合处理，得到预聚合指标数据，并将预聚合指标数据发送至协调写入节点；协调写入节点，用于将预聚合指标数据转换成具有目标存储格式的转换指标数据，并对转换指标数据进行归并处理，得到存储指标数据；协调写入节点，还用于将存储指标数据写入数据库节点中；数据库节点，用于将存储指标数据写入存储磁盘。
在一些实施例中，收发节点包括指标代理节点及监控收集节点；指标代理节点，用于与边缘集群建立网络长连接，并通过网络长连接接收边缘集群发送的采集指标数据；将采集指标数据推送至监控收集节点；监控收集节点，用于获取采集指标数据的指标信息，并获取指标聚合属性；对与指标聚合属性匹配的指标信息所对应的采集指标数据进行指标转换，得到预聚合指标数据；向协调写入节点发送远程写入请求，基于远程写入请求将预聚合指标数据发送至协调写入节点。
在一些实施例中，指标代理节点，还用于通过网络长连接接收边缘集群发送的待处理指标数据；指标代理节点获取待处理指标数据的指标属性标签，将具有相同的指标属性标签的待处理指标数据进行归并处理，得到采集指标数据；将采集指标数据缓存至待发送指标队列中；将待发送指标队列中的采集指标数据推送至监控收集节点。
在一些实施例中，收发节点，还用于与边缘集群中的服务端点建立连接，并获取服 务端点的端点信息，基于端点信息生成连接随机数；获取连接建立范围；当连接随机数位于连接建立范围时，接收边缘集群中的服务端点发送的采集指标数据。
在一些实施例中，协调写入节点，还用于对转换指标数据进行降采样处理，得到降采样指标数据；获取转换指标数据的转换存储期限，将具有相同的转换存储期限的降采样指标数据进行合并处理，得到存储指标数据。
在一些实施例中，数据库节点的数量为N，N为正整数；协调写入节点，还用于获取存储指标数据的存储指标名称及存储指标属性标签，基于存储指标名称及存储指标属性标签，生成存储指标数据的存储哈希值；基于存储哈希值确定存储分片标识，将存储指标数据写入存储分片标识对应的数据库节点中。
在一些实施例中，数据库节点，还用于数据库节点对存储指标数据进行冷热分析处理，得到存储指标数据包括的热指标数据及冷指标数据；确定存储磁盘中的热数据存储区域及冷数据存储区域，将热指标数据写入热数据存储区域，将冷指标数据写入冷数据存储区域。
在一些实施例中，收发节点，还用于通过收发节点接收边缘集群发送的初始指标数据；获取待采集指标类型，并基于从初始指标数据中获取的待采集指标类型对应的指标数据，生成采集指标数据。
例如，该边缘集群可以包括一个或至少两个服务端点(Service)，该服务端点可以认为是边缘节点，在服务端点与指标代理节点之间建立网络长连接，基于该网络长连接，各个服务端点均可以向指标代理节点批量发送(Batch Send)采集指标数据。指标代理节点创建一个协程池/线程池，接收采集指标数据，还可以将大数据量的采集指标数据切分为多个小请求，将小请求并行推送至监控收集节点，例如，指标代理节点接收到采集指标数据，该采集指标数据的数量为100万个，对采集指标数据进行切分，得到100个小请求，每个小请求包括1万个采集指标数据，基于协程池/线程池将多个小请求并行推送至监控收集节点。其中，该指标代理节点可以是Metric Proxy系统，该监控收集节点可以是普罗米修斯(Prometheus)系统，该Prometheus提供了一系列的工具集，方便监测和查询服务和应用内部的运行状态。
在一些实施例中，监控收集节点可以向协调写入节点远程写入预聚合指标数据，例如，监控收集节点可以对接收到的采集指标数据进行预聚合处理，得到多个预聚合指标数据，还可以对预聚合指标数据进行拆分，得到至少两个预聚合推送请求，基于至少两个预聚合推送请求，将预聚合指标数据并行发送至协调写入节点中。例如，可以对监控收集节点的节点参数进行更新，该节点参数包括但不限于监控收集节点的数据缓存时间及垃圾收集(GC，Garbage Collection)参数等，例如，减少数据缓存时间，该数据缓存时间是指监控收集节点对接收到的指标数据的保留时长，通过减少数据缓存时间，减少监控收集节点的空间占用率，提高监控收集节点的运行效率；还可以调整GC参数，例如，将GC参数置为30，则当检测到第(i+1)次为采集指标数据等分配的内存，比第i次为采集指标数据等分配的内存超过30％，则触发GC功能，进行垃圾收集，以释放监控收集节点的内存，进而提高监控收集节点的运行效率，i为正整数。
其中，该Prometheus系统801可以参见图8，图8是本申请实施例提供的一种监控收集节点的架构图。如图8所示，监控收集节点可以监测服务发现(Service Discovery)，发现需要监控的服务(targets)列表，基于抓取发现服务(Scrape Discovery)功能监控服务，可以基于抓取管理器(Scrape Manager)监控服务的变化，更新抓取服务，并可以基于服务列表抓取指标数据，例如，该服务列表可以用于表示监控收集节点需要监控的服务端点，监控收集节点可以基于服务列表对获取到的采集指标数据进行筛选。
例如，该监控收集节点还可以包括读写代理组件，用于读/写数据，其中，监控收集 节点可以进行本地存储及远程存储，监控收集节点的本地存储是通过从该监控收集节点的本地磁盘中读/写数据实现的，远程存储是通过从远程读取端点中进行远程读取数据，从远程写入端点中远程写入数据实现的。例如，监控收集节点可以基于该读写代理组件，对接收到的采集指标数据进行存储，还可以基于该读写代理组件对本地磁盘或远程读取端点中存储的数据进行获取并处理等。
例如，该监控收集节点在抓取管理器中集成第三数据推送功能，例如，是在抓取管理器中集成推送接口，基于该推送接口实现第三数据推送功能，将针对采集指标数据进行处理得到的预聚合指标数据发送至协调写入节点，使得监控收集节点可以自由控制指标数据的推送时间及推送规格等，即监控收集节点可以基于实际需要，在任意一个时间点实现对预聚合指标数据发送至协调写入组件，同时可以以任意的数据量及推送方式实现对预聚合指标数据的推送，例如，对预聚合指标数据进行全量推送(即数据量为预聚合指标数据的全部)，或者进行拆分推送(即数据量为预聚合指标数据拆分后对应的数据量)等；可以将预聚合指标数据并行发送至协调写入节点(即推送方式为并行推送方式)，或者将预聚合指标数据串行发送至协调写入节点(即推送方式为串行推送方式)等。
在一些实施例中，该协调写入节点可以是m3Coordinator系统，该协调写入节点用于将预聚合指标数据转换成具有目标存储格式的转换指标数据，对转换指标数据进行归并处理，得到存储指标数据，将存储指标数据写入数据库节点中，对该协调写入节点进行多子进程部署，利用多核，以提高吞吐量，该协调写入节点还可以集成写入工作池策略(writeWorkerPoolPolicy)参数，基于该参数使得协调写入节点可以根据压力增加协程池的大小，从而提高协调写入节点的工作性能。例如，还可以在协调写入节点向数据库节点写入指标数据时，增加压缩选项，以减少协调写入节点写入到数据库节点的数据量，减少协调写入组件由于出包量过大造成机器打挂(即机器僵住或死机等)的情况，进一步提高协调写入节点的性能。该数据库节点，用于将存储指标数据写入存储磁盘。
其中，该数据处理系统还可以包括指标可视化节点及指标查询节点。指标可视化节点，用于接收用户设备发送的指标查询信息，基于指标查询信息向指标查询节点发送指标查询请求，其中，指标查询请求包括指标查询信息；指标查询节点，用于将指标查询请求拆分为M个分片查询请求；M为正整数；该指标查询节点，还用于基于M个分片查询请求，并从N个数据库节点中确定查询数据库节点，从查询数据库节点中获取M个分片查询请求分别对应的查询指标分片数据，将M个查询指标分片数据发送至指标可视化节点；该指标可视化节点，用于将M个查询指标分片数据进行整合，得到指标查询结果，并将指标查询结果发送至用户设备。
在一些实施例中，其中，指标可视化节点，还用于基于指标可视化节点的可视化节点地址，接收用户设备发送的指标查询信息；获取指标查询节点的查询节点地址，并基于查询节点地址建立监控面板，从监控面板中获取指标查询信息对应的指标查询语句；基于指标查询语句向指标查询节点发送指标查询请求。
在一些实施例中，其中，指标查询信息包括M个分片查询请求分别对应的分片查询指标名称及分片查询指标属性标签；指标查询节点，还用于基于M个分片查询请求分别对应的分片查询指标名称及分片查询指标属性标签，获取M个分片查询请求分别对应的分片查询哈希值；获取M个分片查询哈希值分别对应的分片查询标识，并基于M个分片查询标识，从N个数据库节点中确定k个查询数据库节点，k为小于或等于M的正整数。
在一些实施例中，指标可视化节点，还用于将M个查询指标分片数据进行组合，得到指标查询数据；基于指标查询信息建立查询坐标，将查询坐标及指标查询数据进行 组合，得到指标查询结果。
在一些实施例中，参见图9，图9是本申请实施例提供的另一种数据处理系统架构示意图。如图9所示，该数据处理系统包括收发节点、协调写入节点、数据库节点、指标查询节点及指标可视化节点等，其中，该收发节点包括指标代理节点。该数据处理系统是指中央集群901对应的系统。其中，该数据处理系统中的各个节点是指上述中央集群中运行有相同组件的至少一个计算机设备所构成的节点。其中，数据处理系统将协调写入节点集成至指标代理节点中，以减少性能损耗，减轻运维成本。其中，在图9所示的数据处理系统的架构中，指标代理节点在获取到采集指标数据，对采集指标数据进行预聚合处理，得到预聚合指标数据，将该预聚合指标数据发送至协调写入节点，即图7中的监控收集节点所实现的功能，在图9所示架构中，是由指标代理节点902实现的。
其中，本申请实施例可以应用于云游戏领域，也可以是对任意一个应用程序进行指标数据采集的场景。举例来说，参见图10，图10是本申请实施例提供的一种云游戏部署场景示意图。如图10所示，云游戏架构中存在一个或至少两个边缘集群(如图10所示的边缘集群1和边缘集群2)，每个边缘集群动态启动和销毁h个实例，h为正整数该实例受中央集群的服务进行总体调度和资源分配等，每个实例与应用服务进行交互，该应用服务可以是应用程序服务(App，Application)、H5服务或电脑网页(PC Web)服务等。其中，每个实例中可以包括调度模块、安卓容器模块、音视频编码模块及流媒体传输模块等，其中，该流媒体传输模块可以根据用户实际的网络状态，实时调整流控策略。中央集群可以通过监控该流媒体传输模块的性能和状态，得到云游戏所产生的指标数据，即，各个用户在使用该云游戏时所产生的指标数据，其中，目标指标还可以包括网络连接指标、流控策略有效性指标及线上卡顿指标等，该网络连接指标对应的指标数据包括但不限于行动热点(wifi)、第二代无线通信技术(2G，The 2nd Generation)、第三代移动通信技术(3G，3rd Generation)及第四代移动通信系统(4G，Fourth Generation Communications System)等。
通过本申请实施例中的方案，可以获取边缘集群发送的采集指标数据，并对采集指标数据进行处理后存储至数据库节点中，用户设备可以通过查询该数据库节点中的指标数据，得到云游戏或应用程序等的运行情况，进而对云游戏或应用程序等进行优化。
以云游戏为例，中央集群可以从边缘节点获取采集指标数据，该采集指标数据可以包括但不限于游戏卡顿频率指标、游戏帧传输速率指标、游戏数据同步时延指标及游戏渲染速率指标等指标所对应的指标数据，基于上述图4所示的各个步骤，中央集群通过包含的组件将该采集指标数据进行处理，得到存储指标数据，将存储指标数据写入数据库组件中。当用户设备需要获取该云游戏相关的指标数据时，可以向中央集群发送指标查询信息，中央集群通过执行图5中所示的各个步骤，获取到指标查询结果，将该指标查询结果发送至用户设备。
用户设备可以基于该指标查询结果确定云游戏的运行状态，若该指标查询结果所指示的云游戏的运行状态为游戏异常状态，则可以从指标查询结果中获取出现异常的指标数据，可以基于异常的指标数据对云游戏进行优化处理，使得该云游戏可以更好地满足用户需求，提高云游戏的运行效率及运行效果等。其中，用户设备可以预设有合理指标数据范围，将获取到的指标查询结果与合理指标数据范围进行对比，当指标查询结果属于合理指标数据范围内时，则确定该指标查询结果所指示的云游戏的运行状态为游戏正常状态；当指标查询结果中存在不属于合理指标数据范围的数据时，则确定该指标查询结果所指示的云游戏的运行状态为游戏异常状态。其中，该合理指标数据范围用于表示云游戏可以较好地满足用户游戏需求时，应该具备的指标数据所属的范围，例如，假定游戏帧传输速率指标对应的合理指标数据范围为0.5秒/帧以下，当指标查询结果中的游 戏帧传输速率的值小于0.5秒/帧时，则确定该指标查询结果所指示的云游戏的运行状态为游戏正常状态；当指标查询结果中的游戏帧传输速率的值大于或等于0.5秒/帧时，则确定该指标查询结果所指示的云游戏的运行状态为游戏异常状态，用户设备可以对云游戏进行优化，例如，更新游戏设备之间的数据传输算法，或者，压缩游戏设备之间传输的数据包等，以提高游戏帧传输速率。
在一些实施例中，请参见图11，图11是本申请实施例提供的一种数据处理装置示意图。该数据处理装置可以是运行于计算机设备(运行有上述的收发组件、协调写入组件、数据库组件、指标查询组件及指标可视化组件)中的一个计算机程序(包括程序代码等)，例如该数据处理装置可以为一个应用软件；该装置可以用于执行本申请实施例提供的方法中的相应步骤。如图11所示，该数据处理装置1100可以用于图4所对应实施例中的计算机设备，该装置可以包括：指标采集模块11、数据推送模块12、数据转换模块13、数据压缩模块14及数据存储模块15。
指标采集模块11，配置为通过收发组件接收边缘集群发送的采集指标数据；数据推送模块12，配置为基于收发组件对采集指标数据进行预聚合处理，得到预聚合指标数据，并将预聚合指标数据发送至协调写入组件；数据转换模块13，配置为基于协调写入组件将预聚合指标数据转换成具有目标存储格式的转换指标数据；数据压缩模块14，配置为对转换指标数据进行归并处理，得到存储指标数据；数据存储模块15，配置为将存储指标数据写入数据库组件中，基于数据库组件将存储指标数据写入存储磁盘。
在一些实施例中，该收发组件包括指标代理组件及监控收集组件；该指标采集模块11，包括：连接建立单元111，配置为通过收发组件所包括的指标代理组件，与边缘集群建立网络长连接；指标采集单元112，配置为通过网络长连接接收边缘集群发送的采集指标数据；该数据推送模块12，包括：第一推送单元121，配置为基于指标代理组件将采集指标数据推送至监控收集组件；数据聚合单元122，配置为基于监控收集组件获取采集指标数据的指标信息，并获取指标聚合属性；对与指标聚合属性匹配的指标信息所对应的采集指标数据进行指标转换，得到预聚合指标数据；第二推送单元123，配置为基于监控收集组件向协调写入组件发送远程写入请求，基于远程写入请求将预聚合指标数据发送至协调写入组件。
在一些实施例中，该指标采集单元112，包括：数据接收子单元1121，配置为通过网络长连接接收边缘集群发送的待处理指标数据；数据归并子单元1122，配置为基于指标代理组件获取待处理指标数据的指标属性标签，将具有相同的指标属性标签的待处理指标数据进行归并处理，得到采集指标数据；该第一推送单元121，包括：队列缓存子单元1211，配置为基于指标代理组件将采集指标数据缓存至待发送指标队列中；队列推送子单元1212，配置为将待发送指标队列中的采集指标数据推送至监控收集组件。
在一些实施例中，该指标采集模块11，包括：随机生成单元113，配置为通过收发组件与边缘集群中的服务端点建立连接，并获取服务端点的端点信息，基于端点信息生成连接随机数；连接匹配单元114，配置为获取连接建立范围；当连接随机数位于连接建立范围时，接收边缘集群中的服务端点发送的采集指标数据。在一些实施例中，该数据压缩模块14，包括：降采样单元141，配置为对转换指标数据进行降采样处理，得到降采样指标数据；指标归并单元142，配置为获取转换指标数据的转换存储期限，将具有相同的转换存储期限的降采样指标数据进行合并处理，得到存储指标数据。
在一些实施例中，数据库组件的数量为N，N为正整数；该数据存储模块15，包括：哈希生成单元151，配置为获取存储指标数据的存储指标名称及存储指标属性标签，基于存储指标名称及存储指标属性标签，生成存储指标数据的存储哈希值；哈希存储单元152，配置为基于存储哈希值确定存储分片标识，将存储指标数据写入存储分片标识对 应的数据库组件中。
在一些实施例中，该数据存储模块15，包括：冷热分离单元153，配置为数据库组件对存储指标数据进行冷热分析处理，得到存储指标数据包括的热指标数据及冷指标数据；冷热存储单元154，配置为确定存储磁盘中的热数据存储区域及冷数据存储区域，将热指标数据写入热数据存储区域，将冷指标数据写入冷数据存储区域。
在一些实施例中，该指标采集模块11，包括：指标筛选单元115，配置为通过收发组件接收边缘集群发送的初始指标数据；获取待采集指标类型，并基于从初始指标数据中获取的待采集指标类型对应的指标数据，生成采集指标数据。
在一些实施例中，数据库组件的数量为N，N为正整数；该装置还包括：指标查询模块16，配置为中央集群接收用户设备发送的指标查询信息，基于指标查询信息向指标查询组件发送指标查询请求，其中，指标查询请求包括指标查询信息；请求分片模块17，配置为基于指标查询组件将指标查询请求拆分为M个分片查询请求，M为正整数；组件确定模块18，配置为基于M个分片查询请求，从N个数据库组件中确定查询数据库组件；分片获取模块19，配置为从查询数据库组件中获取M个分片查询请求分别对应的查询指标分片数据；结果生成模块20，配置为将M个查询指标分片数据进行整合，得到指标查询结果，并将指标查询结果发送至用户设备。
在一些实施例中，该指标查询模块16，包括：信息接收单元161，配置为基于指标可视化组件的可视化组件地址，接收用户设备发送的指标查询信息；语句获取单元162，配置为获取指标查询组件的查询组件地址，并基于查询组件地址建立监控面板，从监控面板中获取指标查询信息对应的指标查询语句；请求发送单元163，配置为基于指标查询语句向指标查询组件发送指标查询请求。
在一些实施例中，指标查询信息包括M个分片查询请求分别对应的分片查询指标名称及分片查询指标属性标签；该组件确定模块18，包括：哈希确定单元181，配置为基于M个分片查询请求分别对应的分片查询指标名称及分片查询指标属性标签，获取M个分片查询请求分别对应的分片查询哈希值；组件确定单元182，配置为获取M个分片查询哈希值分别对应的分片查询标识，并基于M个分片查询标识，从N个数据库组件中确定k个查询数据库组件，k为小于或等于M的正整数。
在一些实施例中，该结果生成模块20，包括：数据整合单元20a，配置为将M个查询指标分片数据进行组合，得到指标查询数据；结果生成单元20b，配置为基于指标查询信息建立查询坐标，将查询坐标及指标查询数据进行组合，得到指标查询结果；结果发送单元20c，配置为将指标查询结果发送至用户设备，以使用户设备显示指标查询结果。
本申请实施例提供了一种数据处理装置，该装置可以运行于中央集群中，在中央集群中集成了收发组件、协调写入组件及数据库组件等，收发组件对获取到的数据进行压缩，以减少数据量，并将压缩后的数据发送至协调写入组件，可以更为高效地实现数据的传输，有利于性能优化，同时将得到的存储指标数据写入数据库组件中，基于数据库组件可扩展的特性使得存储可扩展，提高数据的处理效率。
参见图12，图12是本申请实施例提供的一种计算机设备(运行有上述的收发组件、协调写入组件、数据库组件、指标查询组件及指标可视化组件)的结构示意图。如图12所示，本申请实施例中的计算机设备可以包括：一个或多个处理器1201、存储器1202和输入输出接口1203。该处理器1201、存储器1202和输入输出接口1203通过总线1204连接。存储器1202用于存储计算机程序，该计算机程序包括程序指令，输入输出接口1203用于接收数据及输出数据，如用于中央集群与边缘集群之间进行数据交互，或者用于中央集群与用户设备之间进行数据交互，或者用于中央集群中的各个组件之间进行数 据交互；处理器1201用于执行存储器1202存储的程序指令。
其中，该处理器1201可以执行如下操作：
通过收发组件接收边缘集群发送的采集指标数据；基于收发组件对采集指标数据进行预聚合处理，得到预聚合指标数据，并将预聚合指标数据发送至协调写入组件；协调写入组件将预聚合指标数据转换成具有目标存储格式的转换指标数据，并对转换指标数据进行归并处理，得到存储指标数据；将存储指标数据写入数据库组件中；数据库组件将存储指标数据写入存储磁盘。
在一些可行的实施方式中，该处理器1201可以是中央处理单元(CPU，Central Processing Unit)，该处理器还可以是其他通用处理器、数字信号处理器(DSP，Digital Signal Processor)、专用集成电路(ASIC，Application Specific Integrated Circuit)、现成可编程门阵列(FPGA，Field-Programmable Gate Array)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
该存储器1202可以包括只读存储器和随机存取存储器，并向处理器1201和输入输出接口1203提供指令和数据。存储器1202的一部分还可以包括非易失性随机存取存储器。例如，存储器1202还可以存储设备类型的信息。
具体实现中，该计算机设备可通过其内置的各个功能模块执行如该图4中各个步骤所提供的实现方式，可参见该图4中各个步骤所提供的实现方式，在此不再赘述。
本申请实施例通过提供一种计算机设备，包括：处理器、输入输出接口、存储器，通过处理器获取存储器中的计算机程序，执行该图4中所示方法的各个步骤，进行数据处理操作。在中央集群中集成了收发组件、协调写入组件及数据库组件等，收发组件可以对获取到的数据进行压缩，以减少数据量，并将压缩后的数据发送至协调写入组件，可以更为高效地实现数据的传输，有利于性能优化，同时将得到的存储指标数据写入数据库组件中，基于数据库组件可扩展的特性使得存储可扩展，提高数据的处理效率。
本申请实施例还提供一种计算机可读存储介质，该计算机可读存储介质存储有计算机程序，该计算机程序适于由该处理器加载并执行图4中各个步骤所提供的数据处理方法，可参见该图4中各个步骤所提供的实现方式，在此不再赘述。另外，对采用相同方法的有益效果描述，也不再进行赘述。对于本申请实施例所涉及的计算机可读存储介质实施例中未披露的技术细节，请参照本申请实施例的描述。作为示例，计算机程序可被部署为在一个计算机设备上执行，或者在位于一个地点的多个计算机设备上执行，又或者，在分布在多个地点且通过通信网络互连的多个计算机设备上执行。
该计算机可读存储介质可以是前述任一实施例提供的数据处理装置或者该计算机设备的内部存储单元，例如计算机设备的硬盘或内存。该计算机可读存储介质也可以是该计算机设备的外部存储设备，例如该计算机设备上配备的插接式硬盘，智能存储卡(SMC，Smart Media Card)，安全数字(SD，Secure Digital)卡，闪存卡(Flash Card)等。在一些实施例中，该计算机可读存储介质还可以既包括该计算机设备的内部存储单元也包括外部存储设备。该计算机可读存储介质用于存储该计算机程序以及该计算机设备所需的其他程序和数据。该计算机可读存储介质还可以用于暂时地存储已经输出或者将要输出的数据。
本申请实施例还提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行图4中的各种方式中所提供的方法，实现了在中央集群中集成了收发组件、协调写入组件及数据库组件等，该收发组件可以对获取到的数据进行压缩，以减少数据 量，并将压缩后的数据发送至协调写入组件，可以更为高效地实现数据的传输，有利于性能优化，同时将得到的存储指标数据写入数据库组件中，基于数据库组件可扩展的特性使得存储可扩展，提高数据的处理效率。
本申请实施例的说明书和权利要求书及附图中的术语“第一”、“第二”等是用于区别不同对象，而非用于描述特定顺序。此外，术语“包括”以及它们任何变形，意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、装置、产品或设备没有限定于已列出的步骤或模块，而是还包括没有列出的步骤或模块，或还包括对于这些过程、方法、装置、产品或设备固有的其他步骤单元。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在该说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本申请实施例提供的方法及相关装置是参照本申请实施例提供的方法流程图和/或结构示意图来描述的，可由计算机程序指令实现方法流程图和/或结构示意图的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。这些计算机程序指令可提供到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或结构示意图一个方框或多个方框中指定的功能的装置。这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或结构示意图一个方框或多个方框中指定的功能。这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或结构示意一个方框或多个方框中指定的功能的步骤。
本申请实施例方法中的步骤可以根据实际需要进行顺序调整、合并和删减。
本申请实施例装置中的模块可以根据实际需要进行合并、划分和删减。
以上所揭露的仅为本申请较佳实施例而已，当然不能以此来限定本申请之权利范围，因此依本申请权利要求所作的等同变化，仍属本申请所涵盖的范围。

Claims (17)

  1. 一种数据处理方法，应用于中央集群，所述中央集群包括多个计算机设备，每个所述计算机设备用于运行收发组件、协调写入组件及数据库组件中的至少一种；
    所述方法包括：
    通过所述收发组件接收边缘集群发送的采集指标数据；
    基于所述收发组件对所述采集指标数据进行预聚合处理，得到预聚合指标数据，并将所述预聚合指标数据发送至所述协调写入组件；
    所述协调写入组件将所述预聚合指标数据转换成具有目标存储格式的转换指标数据，并对所述转换指标数据进行归并处理，得到存储指标数据；
    将所述存储指标数据写入所述数据库组件中；
    所述数据库组件将所述存储指标数据写入存储磁盘。
  2. 如权利要求1所述的方法，其中，
    所述收发组件包括指标代理组件及监控收集组件；
    所述通过所述收发组件接收边缘集群发送的采集指标数据，包括：
    通过所述指标代理组件与所述边缘集群建立网络长连接，并通过所述网络长连接接收所述边缘集群发送的采集指标数据；
    所述对所述采集指标数据进行预聚合处理，得到预聚合指标数据，包括：
    所述指标代理组件将所述采集指标数据推送至所述监控收集组件；
    所述监控收集组件获取所述采集指标数据的指标信息，并获取指标聚合属性；
    对与所述指标聚合属性匹配的指标信息所对应的采集指标数据进行指标转换，得到预聚合指标数据；
    所述将所述预聚合指标数据发送至所述协调写入组件，包括：
    所述监控收集组件向所述协调写入组件发送远程写入请求，基于所述远程写入请求将所述预聚合指标数据发送至所述协调写入组件。
  3. 如权利要求2所述的方法，其中，所述通过所述网络长连接接收所述边缘集群发送的采集指标数据，包括：
    通过所述网络长连接接收所述边缘集群发送的待处理指标数据；
    所述指标代理组件获取所述待处理指标数据的指标属性标签，将具有相同的所述指标属性标签的待处理指标数据进行归并处理，得到所述采集指标数据；
    所述指标代理组件将所述采集指标数据推送至所述监控收集组件，包括：
    所述指标代理组件将所述采集指标数据缓存至待发送指标队列中；
    将所述待发送指标队列中的所述采集指标数据推送至所述监控收集组件。
  4. 如权利要求1所述的方法，其中，所述通过所述收发组件接收边缘集群发送的采集指标数据，包括：
    通过所述收发组件与边缘集群中的服务端点建立连接，并获取所述服务端点的端点信息，基于所述端点信息生成连接随机数；
    获取连接建立范围；
    当所述连接随机数位于所述连接建立范围时，接收所述边缘集群中的服务端点发送的采集指标数据。
  5. 如权利要求1所述的方法，其中，所述对所述转换指标数据进行归并处理，得到存储指标数据，包括：
    对所述转换指标数据进行降采样处理，得到降采样指标数据；
    获取所述转换指标数据的转换存储期限，将具有相同的所述转换存储期限的降采样指标数据进行合并处理，得到存储指标数据。
  6. 如权利要求1所述的方法，其中，
    所述数据库组件的数量为N，N为正整数；
    所述将所述存储指标数据写入所述数据库组件中，包括：
    获取所述存储指标数据的存储指标名称及存储指标属性标签，基于所述存储指标名称及所述存储指标属性标签，生成所述存储指标数据的存储哈希值；
    基于所述存储哈希值确定存储分片标识，将所述存储指标数据写入所述存储分片标识对应的所述数据库组件中。
  7. 如权利要求1所述的方法，其中，所述数据库组件将所述存储指标数据写入存储磁盘，包括：
    所述数据库组件对所述存储指标数据进行冷热分析处理，得到所述存储指标数据包括的热指标数据及冷指标数据；
    确定所述存储磁盘中的热数据存储区域及冷数据存储区域，将所述热指标数据写入所述热数据存储区域，将所述冷指标数据写入所述冷数据存储区域。
  8. 如权利要求1所述的方法，其中，所述通过所述收发组件接收边缘集群发送的采集指标数据，包括：
    通过所述收发组件接收所述边缘集群发送的初始指标数据；
    获取待采集指标类型，并基于从所述初始指标数据中获取的所述待采集指标类型对应的指标数据，生成采集指标数据。
  9. 如权利要求1所述的方法，其中，
    所述计算机设备还可以运行指标查询组件；
    所述数据库组件的数量为N，N为正整数；
    所述方法还包括：
    所述中央集群接收用户设备发送的指标查询信息，基于所述指标查询信息向所述指标查询组件发送指标查询请求，其中，所述指标查询请求包括所述指标查询信息；
    所述指标查询组件将所述指标查询请求拆分为M个分片查询请求，M为正整数；
    基于所述M个分片查询请求，从N个数据库组件中确定查询数据库组件，并从所述查询数据库组件中获取所述M个分片查询请求分别对应的查询指标分片数据；
    将M个查询指标分片数据进行整合，得到指标查询结果，并将所述指标查询结果发送至所述用户设备。
  10. 如权利要求9所述的方法，其中，
    所述计算机设备还可以运行指标可视化组件；
    所述中央集群接收用户设备发送的指标查询信息，基于所述指标查询信息向所述指标查询组件发送指标查询请求，包括：
    基于所述指标可视化组件的可视化组件地址，接收用户设备发送的指标查询信息；
    获取所述指标查询组件的查询组件地址，并基于所述查询组件地址建立监控面板，从所述监控面板中获取所述指标查询信息对应的指标查询语句；
    基于所述指标查询语句向所述指标查询组件发送指标查询请求。
  11. 如权利要求9所述的方法，其中，
    所述指标查询信息包括M个分片查询请求分别对应的分片查询指标名称及分片查询指标属性标签；
    所述基于所述M个分片查询请求，从N个数据库组件中确定查询数据库组件，包括：
    基于所述M个分片查询请求分别对应的分片查询指标名称及分片查询指标属性标签，获取所述M个分片查询请求分别对应的分片查询哈希值；
    获取M个分片查询哈希值分别对应的分片查询标识，并基于M个分片查询标识，从N个数据库组件中确定k个查询数据库组件，k为小于或等于M的正整数。
  12. 如权利要求9所述的方法，其中，所述将M个查询指标分片数据进行整合，得到指标查询结果，包括：
    将M个查询指标分片数据进行组合，得到指标查询数据；
    基于所述指标查询信息建立查询坐标，将所述查询坐标及所述指标查询数据进行组合，得到指标查询结果。
  13. 一种数据处理系统，所述数据处理系统包括收发节点、协调写入节点及数据库节点，所述收发节点、所述协调写入节点及所述数据库节点分别是通过至少一个计算机设备实现的；
    所述收发节点，用于接收边缘集群发送的采集指标数据；
    所述收发节点，还用于对所述采集指标数据进行预聚合处理，得到预聚合指标数据，并将所述预聚合指标数据发送至所述协调写入节点；
    所述协调写入节点，用于将所述预聚合指标数据转换成具有目标存储格式的转换指标数据，并对所述转换指标数据进行归并处理，得到存储指标数据；
    所述协调写入节点，还用于将所述存储指标数据写入所述数据库节点中；
    所述数据库节点，用于将所述存储指标数据写入存储磁盘。
  14. 一种数据处理装置，应用于计算机设备，所述计算机设备运行有收发组件、协调写入组件及数据库组件；
    所述数据处理装置包括：
    指标采集模块，配置为通过所述收发组件接收边缘集群发送的采集指标数据；
    数据推送模块，配置为基于所述收发组件对所述采集指标数据进行预聚合处理，得到预聚合指标数据，并将所述预聚合指标数据发送至所述协调写入组件；
    数据转换模块，配置为基于协调写入组件将所述预聚合指标数据转换成具有目标存储格式的转换指标数据；
    数据压缩模块，配置为对所述转换指标数据进行归并处理，得到存储指标数据；
    数据存储模块，配置为将所述存储指标数据写入所述数据库组件中，所述数据库组件将所述存储指标数据写入存储磁盘。
  15. 一种计算机设备，包括处理器、存储器、输入输出接口；
    所述处理器分别与所述存储器和所述输入输出接口相连，其中，所述输入输出接口用于接收数据及输出数据，所述存储器用于存储计算机程序，所述处理器用于调用所述计算机程序，以使得所述计算机设备执行权利要求1-12任一项所述的方法。
  16. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序适于由处理器加载并执行，以使得具有所述处理器的计算机设备执行权利要求1-12任一项所述的方法。
  17. 一种计算机程序产品，包括计算机指令，所述计算机指令使得计算机设备执行权利要求1-12任一项所述的方法。
PCT/CN2022/097848 2021-07-08 2022-06-09 一种数据处理方法、装置、系统、计算机设备、可读存储介质及计算机程序产品 WO2023279912A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/128,144 US20230237064A1 (en) 2021-07-08 2023-03-29 Data processing method, apparatus, and system, computer device, readable storage medium, and computer program product

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110772159.5A CN113220715B (zh) 2021-07-08 2021-07-08 一种数据处理方法、系统、计算机及可读存储介质
CN202110772159.5 2021-07-08

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/128,144 Continuation US20230237064A1 (en) 2021-07-08 2023-03-29 Data processing method, apparatus, and system, computer device, readable storage medium, and computer program product

Publications (1)

Publication Number Publication Date
WO2023279912A1 true WO2023279912A1 (zh) 2023-01-12

Family

ID=77081205

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/097848 WO2023279912A1 (zh) 2021-07-08 2022-06-09 一种数据处理方法、装置、系统、计算机设备、可读存储介质及计算机程序产品

Country Status (3)

Country Link
US (1) US20230237064A1 (zh)
CN (1) CN113220715B (zh)
WO (1) WO2023279912A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116303475A (zh) * 2023-05-17 2023-06-23 吉奥时空信息技术股份有限公司 一种多源指标数据智能存储的管理方法及装置

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113220715B (zh) * 2021-07-08 2021-10-08 腾讯科技（深圳）有限公司 一种数据处理方法、系统、计算机及可读存储介质
CN114489491A (zh) * 2021-12-30 2022-05-13 北京云测信息技术有限公司 一种数据加载的方法、装置、电子设备及存储介质
CN115904910B (zh) * 2022-12-23 2024-03-12 北京云端智度科技有限公司 一种边缘节点状态感知方法、系统、介质及设备
CN116560949B (zh) * 2023-07-10 2023-09-29 南瑞轨道交通技术有限公司 一种数据采集与接收的实现方法、系统、设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102316025A (zh) * 2011-07-27 2012-01-11 中国科学院计算机网络信息中心 网络流量控制方法、装置、系统及服务器
US20130346533A1 (en) * 2012-06-25 2013-12-26 Microsoft Corporation Near-real time distributed usage aggregation system
CN109977077A (zh) * 2019-03-25 2019-07-05 腾讯科技（深圳）有限公司 模型文件存储方法、装置、可读存储介质和计算机设备
CN112671602A (zh) * 2020-12-14 2021-04-16 北京金山云网络技术有限公司 边缘节点的数据处理方法、装置、系统、设备和存储介质
CN113220715A (zh) * 2021-07-08 2021-08-06 腾讯科技（深圳）有限公司 一种数据处理方法、系统、计算机及可读存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5292335B2 (ja) * 2010-02-18 2013-09-18 日本電信電話株式会社 接続先ノード選択方法及び装置及びプログラム
CN102291269A (zh) * 2011-09-29 2011-12-21 迈普通信技术股份有限公司 一种数据归并处理方法
CN106156040A (zh) * 2015-03-26 2016-11-23 阿里巴巴集团控股有限公司 多维度数据管理方法及装置
CN111552689B (zh) * 2020-03-30 2022-05-03 平安医疗健康管理股份有限公司 一种基金审计的去重指标计算方法、装置及设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102316025A (zh) * 2011-07-27 2012-01-11 中国科学院计算机网络信息中心 网络流量控制方法、装置、系统及服务器
US20130346533A1 (en) * 2012-06-25 2013-12-26 Microsoft Corporation Near-real time distributed usage aggregation system
CN109977077A (zh) * 2019-03-25 2019-07-05 腾讯科技（深圳）有限公司 模型文件存储方法、装置、可读存储介质和计算机设备
CN112671602A (zh) * 2020-12-14 2021-04-16 北京金山云网络技术有限公司 边缘节点的数据处理方法、装置、系统、设备和存储介质
CN113220715A (zh) * 2021-07-08 2021-08-06 腾讯科技（深圳）有限公司 一种数据处理方法、系统、计算机及可读存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116303475A (zh) * 2023-05-17 2023-06-23 吉奥时空信息技术股份有限公司 一种多源指标数据智能存储的管理方法及装置
CN116303475B (zh) * 2023-05-17 2023-08-08 吉奥时空信息技术股份有限公司 一种多源指标数据智能存储的管理方法及装置

Also Published As

Publication number Publication date
CN113220715A (zh) 2021-08-06
CN113220715B (zh) 2021-10-08
US20230237064A1 (en) 2023-07-27

Similar Documents

Publication Publication Date Title
WO2023279912A1 (zh) 一种数据处理方法、装置、系统、计算机设备、可读存储介质及计算机程序产品
US11836533B2 (en) Automated reconfiguration of real time data stream processing
US9634915B2 (en) Methods and computer program products for generating a model of network application health
CN110019240B (zh) 一种业务数据交互方法、装置及系统
US8868727B2 (en) Methods and computer program products for storing generated network application performance data
Chaczko et al. Availability and load balancing in cloud computing
CN105335513B (zh) 一种分布式文件系统及文件存储方法
CN106709003A (zh) 基于Hadoop的海量日志数据处理方法
CN112714192B (zh) 数据同步方法、装置、计算机可读介质及电子设备
CN114201421B (zh) 一种数据流处理方法、存储控制节点及可读存储介质
CN113010565A (zh) 基于服务器集群的服务器实时数据处理方法及系统
US8502710B2 (en) Methods and computer program products for providing a compressed circular buffer for efficient storage of network performance data
CN112162829B (zh) 一种边缘计算场景下的资源监控数据预处理系统
CN104219298A (zh) 集群系统及其数据备份的方法
WO2021027331A1 (zh) 基于图数据的全量关系计算方法、装置、设备及存储介质
CN117278628B (zh) 数据传输方法、装置、系统、计算机设备和存储介质
US9893972B1 (en) Managing I/O requests
CN112650755A (zh) 数据存储方法、查询数据的方法、数据库、以及可读介质
Zhang et al. Efficient online surveillance video processing based on spark framework
CN113467718B (zh) 一种数据处理方法、装置、电子设备及存储介质
CN115022585A (zh) 告警视频抓取的方法和装置
JP2023031248A (ja) エッジコンピューティングネットワーク、データ伝送方法、装置、機器、及び記憶媒体
CN111294231B (zh) 资源管理方法及系统
WO2017024976A1 (zh) 实时信息的展现方法和装置
CN110191346B (zh) 云端的管理方法、装置和计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22836667

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE