WO2017092600A1 - Procédé et dispositif de comptage de pointeurs - Google Patents

Procédé et dispositif de comptage de pointeurs Download PDF

Info

Publication number
WO2017092600A1
WO2017092600A1 PCT/CN2016/107017 CN2016107017W WO2017092600A1 WO 2017092600 A1 WO2017092600 A1 WO 2017092600A1 CN 2016107017 W CN2016107017 W CN 2016107017W WO 2017092600 A1 WO2017092600 A1 WO 2017092600A1
Authority
WO
WIPO (PCT)
Prior art keywords
indicator
level
name
structured
lowest
Prior art date
Application number
PCT/CN2016/107017
Other languages
English (en)
Chinese (zh)
Inventor
王逸
武翀
刘键
方孝健
封仲淹
Original Assignee
阿里巴巴集团控股有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 阿里巴巴集团控股有限公司 filed Critical 阿里巴巴集团控股有限公司
Publication of WO2017092600A1 publication Critical patent/WO2017092600A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0201—Market modelling; Market analysis; Collecting market data
    • G06Q30/0202—Market predictions or forecasting for commercial activities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network

Definitions

  • the present application relates to the field of real-time computing technologies, and in particular, to an indicator statistical method and an indicator statistical device.
  • Storm is a distributed open source real-time computing system under the apache community, using clojure language (Clojure is a Lisp language running on Java (Java is an object-oriented programming language that can write cross-platform applications) platform (Lisp is a programming language known for its expressiveness and power)).
  • Storm can be used in "stream processing” to process messages in real time; it can also be used for “continuous computation", which continuously processes data streams and outputs the results to users in the form of streams. It can also be used in "Remote Procedure Call Protocol" to perform operations in parallel.
  • JStorm is a real-time computing system based on Storm, which is compatible with Storm.
  • embodiments of the present application have been made in order to provide an indicator statistical method and a corresponding indicator statistical apparatus that overcome the above problems or at least partially solve the above problems.
  • an indicator statistical method including:
  • a structured indicator name is created for each level node corresponding to the indicator; wherein, the hierarchical relationship between the nodes of each level is determined by the name of the structured indicator;
  • the application also discloses an indicator statistical device, comprising:
  • a structured identifier creation module configured to create a structured indicator name for each level node corresponding to the indicator for an indicator of a topology operation; wherein, the hierarchical relationship between the nodes of each level is determined by the structured indicator name;
  • the bottom layer indicator monitoring module is configured to perform statistics on the lowest level data corresponding to the indicator, and under the name of the corresponding lowest level structured indicator;
  • the layer-by-layer summary module is used to summarize the statistics under the name of the lowest-level structured indicator according to the hierarchical relationship between the names of the structured indicators, and to summarize them under the structural indicator name of the previous level.
  • a structured indicator name is created for each level node corresponding to the indicator, and the structured indicator name is used to determine the hierarchical relationship between the hierarchical nodes. Then, the embodiment of the present application monitors the lowest-level data of the indicator, and collects the data under the name of the lowest-level structured indicator, and then summarizes the data based on the lowest level statistical data according to the hierarchical relationship between the structured indicator names. Under the name of the structural indicator of the previous level.
  • the embodiment of the present application can easily perform statistics on indicators of each level by hierarchical relationship of structured indicator names, which is simple in logic, can reduce system consumption, and can be conveniently implemented because of hierarchical relationships constructed according to structured indicator names. Add or remove one or several levels to facilitate expansion.
  • FIG. 3 is a structural block diagram of an embodiment of an indicator statistical device of the present application.
  • FIG. 4 is a structural block diagram of an embodiment of an indicator statistical system of the present application.
  • a jstorm or storm real-time computing system is taken as an example to introduce related terms involved in the embodiments of the present application.
  • Topology A topology job that is an application running on a jstorm or storm system. After a topology job is submitted to the storm or jstorm real-time computing system, it can run without interruption.
  • a topology consists of multiple components, each called a component.
  • the components of storm and jstorm are divided into spout and bolt.
  • the spout component represents the source of the processed data, such as a spout that can fetch data from an external message component, or retrieve data from a database.
  • spout can continuously acquire data from any external data source, and The data is sent downstream, such as a bolt, and Bolt receives the data from Spout and processes it.
  • Task represents a logical processing unit, which is an instance of the implemented spout/bolt.
  • a component may include multiple Tasks.
  • Stream Data stream. Stream is the smallest unit of indicator statistics in jstorm and storm. A Task may include multiple Streams.
  • jstorm and storm real-time computing systems have a hierarchical structure, such as stream ⁇ task ⁇ component ⁇ topology. The data of each indicator is counted in the stream.
  • One of the core concepts of the embodiments of the present application is that, for a real-time computing system, since it has a hierarchical structure when processing data, and for real-time computing systems, it is possible to quickly perform statistics on all levels of indicators, for each level of hierarchical nodes of an indicator.
  • the structured indicator name is created for each hierarchical node, and the hierarchical relationship between the hierarchical nodes is determined by the structured indicator name itself. Therefore, only the data of the lowest-level structured indicator name needs to be counted, and the data can be summarized step by step according to the hierarchical relationship between the structured indicator names, and the data of each level corresponding to the index can be obtained.
  • the hierarchical relationship of the structured indicator names can be used to simply count the indicators of each level, the logic is simple, the system consumption can be reduced, and the hierarchical relationship established according to the structured indicator name can be convenient. Add or remove one or several levels for easy expansion.
  • FIG. 1 a flow chart of steps of an embodiment of an indicator statistical method of the present application is shown, which may specifically include the following steps:
  • Step 110 Create, for an indicator of a topology job, a structured indicator name for each level node corresponding to the indicator; wherein, the hierarchical relationship between the hierarchical nodes is determined by the structured indicator name;
  • the process of real-time processing of data by jstorm is taken as an example, and jstorm can first receive a topology topology, that is, start an application. Then, the embodiment of the present application needs to count various indicators of each level in the topology processing process, such as the amount of sent messages (Emitted) of each level, the level of transmission per second (TPS), and the like. Then, the embodiment of the present application may create a structured indicator name for each level node corresponding to the indicator, and determine a hierarchical relationship between the nodes of each level by using the structured indicator name.
  • Topology, component, task, and stream indicate the location of the node identifier, and name indicates the location of the indicator identifier.
  • the topology node of topology is tp1.
  • the topology includes a hierarchical node component whose node identifier is spout.
  • the component includes two hierarchical nodes Task0 and Task1.
  • Task0 includes two hierarchical nodes Steam0, Steam1.
  • Under Task0 there are two hierarchical nodes Steam2 and Steam3.
  • the statistical indicator is identified as Emitted.
  • Tp1@spout@Task0@Stream0@Emitted indicates the value of the Emitted value of Stream0.
  • the structured indicator name for Steam1 is:
  • Tp1@spout@Task0@Stream1@Emitted indicates the value of the Emitted value of Stream1.
  • the name of the structured indicator corresponding to Task0 is:
  • Tp1@spout@Task0@@Emitted indicates the Emitted value of the statistics Task0.
  • the structured indicator name for Steam2 is:
  • Tp1@spout@Task1@Stream2@Emitted indicates the value of the Emitted value of Stream2.
  • Tp1@spout@Task1@Stream3@Emitted indicates the value of the Emitted value of Stream3.
  • the name of the structured indicator corresponding to Task1 is:
  • Tp1@spout@Task1@@Emitted indicates the Emitted value of the statistics Task1.
  • the name of the structured indicator corresponding to spout is:
  • Tp1@spout@@@Emitted indicates the Emitted value of the statistics spout.
  • the name of the structured indicator corresponding to tp1 is:
  • Tp1@@@@Emitted indicates the Emitted value of the tp1.
  • the structured indicator name corresponding to Task0 is reduced by the name of the structured indicator corresponding to Steam0 and Steam1, and there is a clear relationship between the superior and the lower.
  • tp1@spout@Task0@Stream0@Emitted and tp1@spout@Task0@Stream1@Emitted are level relationships.
  • tp1@spout@@@Emitted is reduced by the structural index names of Task0 and Task1, with a clear relationship between the superior and the subordinate.
  • tp1@spout@Task0@@Emitted and tp1@spout@Task1@Emitted are level relationships.
  • the above structured indicator names can clarify the hierarchical relationship between nodes at each level.
  • step 110 includes:
  • Sub-step A11 for an indicator of a topology job, the node identifier of the top-level hierarchical node to the lowest-level hierarchical node and the indicator identifier of the indicator are sequentially combined into the lowest-level structured indicator name;
  • the lowest-level node identifier of the top-level hierarchical node to the lowest-level hierarchical node and the indicator are used.
  • the indicator identifiers are sequentially combined into the lowest-level structured indicator names.
  • Steam0 is the lowest level node, and its structured indicator name is set first: tp1@spout@Task0@Stream0@Emitted.
  • Several other lowest level hierarchy nodes are similar.
  • the bottom-level structured indicator name represents the hierarchical path from the top to the bottom.
  • Sub-step A12 based on the lowest-level structured indicator name, set the hierarchical node of the current level in the structured indicator name to be empty for the structured indicator name of each level, and obtain the structural indicator name of the previous level.
  • the upper level of Stream is Task
  • the implementation of this application merges the same Task in the previous level of Stream. For example, if Stream0 in tp1@spout@Task0@Stream0@Emitted is set to null, or tp1@spout@Task0@Stream1@Emitted, the structured indicator name tp1@spout@Task0@@Emitted of the hierarchical node Task0 is obtained. In the same way, the structural index name of Task1 is obtained tp1@spout@Task1@@Emitted.
  • the Task0 in tp1@spout@Task0@@Emitted or Task1 in tp1@spout@Task1@@Emitted Set to null, get the structured index name of spout tp1@spout@@@Emitted. And so on, until the top-level structured indicator name is generated.
  • the symbol @ is used as a separator to facilitate the merging of structured indicator names to generate a hierarchical level of structured indicator names.
  • the symbol similar to @ may not be set.
  • the node identifier of each hierarchical node and the associated hierarchical level may be provided to the real-time computing system, so that the real-time computing system can perform sub-step A12 according to the node identifier of each hierarchical level.
  • sub-step A11 includes:
  • the hierarchical node corresponding to the current hierarchical separator in the structured indicator name is set to be empty, and the upper level structure is obtained.
  • the name of the indicator is
  • the embodiment of the present application in order to prevent the full name of the node identifier and the indicator identifier of each hierarchical node from being directly combined, the duplicate name and the ambiguity occur, and the embodiment of the present application adds a separator between the node identifiers of any level. A separator is also added between the lowest node identifier and the metric identifier. @, as in the above tp1@spout@Task0@Stream0@Emitted, is the added separator. When generating the hierarchical structure name of the previous level, you can only set the indicator ID to be empty and retain the separator to make it easier to determine the hierarchical relationship between the names of each structured indicator.
  • tp1@spout@Task0@@Emitted generated by tp1@spout@Task0@Stream0@Emitted, in the subsequent merge, only need to judge that the 3rd and 4th separators are empty, you can Make sure the tp1@spout@Task0@@Emitted name is the value of Emitted for all Steams under Statistics Task0.
  • the method before step 110, the method further includes:
  • B11 Register the indicator identifier corresponding to the lowest level node to the system.
  • the worker executed by each computing node may register the indicator identifier corresponding to the lowest level node in the system of the computing node according to requirements. For example, register the indicator ID of each level node of the Stream level. Then, the computing node in the embodiment of the present application can automatically generate the structured indicator name of the hierarchical node of each level according to the structural definition of the structured indicator name after the index identifier of the lowest level hierarchical node is registered, for example, for the stream level.
  • Stream0 in step A11, generates tp1@spout@Task0@Stream0@Emitted, etc., and then proceeds to step A12 to generate a structured indicator name until each level node is generated.
  • the above only informs the computing node to register the indicator identifier corresponding to the lowest level hierarchical node to the system, and only Give the compute node a simple notification without reducing the transmission overhead by structuring the metric data to the compute node's hierarchical nodes.
  • the structural definition of the structured indicator name may be configured in a scheduling server of the real-time computing system, and then transmitted to each computing node by the scheduling server.
  • the technician can change the hierarchical structure of the structured metric names configured in the dispatch server as needed to change the hierarchy and change the structured metric names for each level node accordingly.
  • the structure of the structured indicator name can be defined as:
  • the group groups the Emitted statistics according to the requirements of different services. For example, the service A needs to send 10 messages, the structured indicator name is +1, and the service B needs to send 1 message, and the structured indicator name + 1. Then different groups, the corresponding structured indicators have different values.
  • the embodiment of the present invention may also change the structural definition of the foregoing structured indicator name according to actual needs, which is not limited in this application.
  • Step 120 After monitoring the lowest-level data corresponding to the indicator, perform statistics under the name of the corresponding lowest-level structured indicator;
  • a topology is created in jstorm, then the jstorm scheduled system can divide the topology into multiple workers, and each worker represents a process that performs a specific task. .
  • the above workers are distributed on different computing nodes of jstorm's computing cluster and executed in parallel. All the actual data processing work is finally completed in the worker. Therefore, for each computing node, the topology is processed in a hierarchical structure, so in the embodiment of the present application, each computing node obtains the above-mentioned structural index names corresponding to the hierarchical nodes of the indicator, and then in each structured The indicators are counted under the indicator name.
  • each worker can run at least one spout and/or at least one bolt.
  • the spout or bolt is divided into task execution, and the task processes the data in the form of a stream.
  • the embodiment of the present application monitors the data related to the indicator that appears in the stream, for example, for the amount of sent messages (Emitted), monitors that the stream level node stream0 passes a Tuple (Tuple: a basic unit of a message delivery), and corresponds to Steam0. Structured indicator name tp1@spout@Task0@Stream0@Emitted value more New is 1.
  • the system can be notified to monitor the indicators of the lowest level hierarchical node, and no monitoring is performed on the lowest layer.
  • Step 130 Based on the statistical data under the name of the lowest-level structured indicator, according to the hierarchical relationship between the names of the structured indicators, the data is summarized step by step to the structural indicator name of the upper level.
  • the hierarchical structure name corresponding to the structured name is found: tp1@spout@Task0@@Emitted, tp1@spout@@@Emitted, tp1 @@@Emitted, then update the values of the three to 1.
  • step 120 it is monitored that the stream level node stream0 passes a Tuple (Tuple: the basic unit of a message delivery), and the value of the structured indicator name tp1@spout@Task0@Stream0@Emitted corresponding to Steam1 is updated to 2.
  • Tuple the basic unit of a message delivery
  • step 130 according to the hierarchical relationship, the layers are summarized step by step, and the summary order and results are as follows:
  • step 120 the stream level node stream3 is monitored to pass a Tuple (Tuple: the basic unit of a message delivery), and the value of the structured indicator name tp1@spout@Task1@Stream3@Emitted corresponding to Steam0 is updated to 1.
  • Tuple the basic unit of a message delivery
  • step 130 according to the hierarchical relationship, the layers are summarized step by step, and the summary order and results are as follows:
  • the embodiment of the present application aggregates the records under the structural indicator names of the respective computing nodes.
  • the record for compute node 2 is:
  • the real-time computing system counts data within one minute for each structured indicator. You can then continuously output statistics every minute. For example, data in the form of a log.
  • the method further includes:
  • step 140 the statistical data under each structured indicator name is exported to a database for storage.
  • the real-time computing system since the real-time computing system does not have the function of the database, the statistical result is inconvenient to query.
  • the present application can export the statistical data under each structured indicator name to HBase, Hadoop, Hive and other databases.
  • step 140 includes:
  • Sub-step C11 the statistical data under each structured indicator name is exported to the database, and the structured indicator name and time stamp are used as keywords, and the statistical data is used as the key value for storage.
  • the structured indicator name and its statistical data are structured at the end of the time period.
  • the indicator name and timestamp are keyword keys, and the statistics are key values and are stored in the database, such as the HBase database. Then, through the timestamp, it is convenient to find the index values of each level in a period of time. This timestamp is the system time at the end of each time period.
  • the topology identifier of the topology is tp1; tp1 has a component, and the hierarchy identifier is spout; there are 5 tasks under the component, and each task has an id corresponding thereto, and the corresponding hierarchical identifiers are respectively Task0 ⁇ Task4; At the same time, there are 2 streams under each task, and the corresponding level identifiers are Stream0 and Stream1 respectively. Then its hierarchical relationship is:
  • the traditional indicator statistical method does not reflect the above hierarchical relationship and calculation logic. If you need to implement this hierarchical logic, you need to do a lot of additional complex logic judgments and calculations. At the same time, the traditional method also needs to carefully select the indicator name to avoid duplication and result in inaccurate data.
  • the embodiment of the present application in the real-time computing system, has a hierarchical structure when processing data, and the real-time computing system can quickly perform statistics on all the levels of the indicators.
  • the embodiment of the present application A structured indicator name is created for each hierarchical node, and the hierarchical relationship between the hierarchical nodes is determined by the structured indicator name itself, thereby determining the summary relationship. Therefore, only the data of the lowest-level structured indicator name needs to be counted, and the data can be summarized step by step according to the hierarchical relationship between the structured indicator names, and the data of each level corresponding to the index can be obtained.
  • the hierarchical relationship of the structured indicator names can be used to simply perform statistics on the indicators of each level, and the logic is simple, which can reduce system consumption, and is based on the structural indicators.
  • the hierarchical relationship of name construction can easily add or delete one or several levels to facilitate expansion.
  • FIG. 2 a flow chart of steps of a preferred embodiment of an indicator statistical method of the present application is shown. Specifically, the method may include the following steps:
  • Step 210 Each computing node corresponds to an indicator of the lowest level hierarchical node to the system of the computing node.
  • the real-time computing system may adopt a distributed computing system, where the distributed computing system includes a scheduling server and each computing node.
  • the structural definition of the structured indicator name can be configured in the scheduling server of the real-time computing system, and then configured by the scheduling server to each computing node, so that each computing node can name the lowest-level structured indicator according to the above definition. Process it.
  • the worker executed by each computing node can register the index corresponding to the lowest level node in the system of the computing node according to requirements. logo.
  • topology indicator For a topology indicator, it can actually be divided into two parts: 1. The system indicator that has been defined inside the Jstorm calculation framework; 2. The user-defined business-related indicator.
  • the user selects the system indicator, it can be registered in the system of the computing node when the worker is initialized. If the user selects a user-defined business indicator, it can be registered in the system of the computing node when the worker initializes the user code.
  • Register the indicator on each compute node Take jstorm as an example. For a stream, call a global static method in the worker provided by jstorm: registerStreamMetrics (metric related parameter), and then register the stream according to the metric related parameters in the system. Indicator ID. Then, proceed to step 220 to generate the structured indicator names of the layers layer by layer.
  • registerStreamMetrics metric related parameter
  • Step 220 For each index of a topology operation, each computing node uses a delimiter to splice the node identifier of the topmost layer to the lowest level node and the indicator identifier of the index into the lowest structured identifier name;
  • the computing node After each computing node registers the lowest-level indicator identifier, the computing node can generate the lowest-level structured indicator name according to the structured index name and the upper-level relationship of each hierarchical node recorded in the system.
  • Step 230 Each computing node sets, according to the lowest-level structured indicator name, the hierarchical node of the current level in the structured indicator name to the structured indicator name of each level, and obtains the structure of the upper level. The name of the indicator.
  • Step 240 After monitoring the lowest level data corresponding to the indicator, each computing node is at the bottom of the corresponding bottom layer. Statistics under the name of the structured indicator;
  • each computing node summarizes the statistical data under the name of the lowest-level structured indicator according to the hierarchical relationship between the names of the structured indicators, and gradually summarizes them to the structural indicator name of the upper level.
  • Each computing node collects data under the name of each structured indicator according to a time period, for example, a period of 1 minute, and at the end of the time period, sends statistical data of each structured indicator name of the period to the scheduling server.
  • Step 260 The scheduling server acquires statistics of each structured indicator name from each computing node, and performs aggregation.
  • the scheduling server obtains statistics of each structured node to obtain the name of each structured indicator, and then can perform aggregation.
  • Step 270 The scheduling server exports the summarized statistical data under the name of each structured indicator to the database, and uses the structured indicator name and time stamp as keywords and stores the statistical data as a key value.
  • the statistic statistics of each level of each computing node can be aggregated to the scheduling server of the cluster.
  • each computing node aggregates the indicator statistics into the scheduling server every 1 time period, for example, 1 minute, and since the scheduling server does not act as a storage server, the data is continuously covered by new indicator statistics. Therefore, only the aggregated indicator statistics for the most recent time period can be seen.
  • the scheduling server of the embodiment of the present application stores the aggregated indicator statistics in an external database.
  • the structured indicator name + time stamp is the key, and the statistical data is the value and stored in the database.
  • FIG. 3 a structural block diagram of an embodiment of an indicator statistical device of the present application is shown, which may specifically include the following modules:
  • the structured identifier creation module 310 is configured to: for an indicator of a topology job, create a structured indicator name for each level node corresponding to the indicator; wherein, the hierarchical relationship between the hierarchical nodes is determined by the structured indicator name;
  • the bottom layer indicator monitoring module 320 is configured to perform statistics on the bottommost structured indicator name after monitoring the lowest level data corresponding to the indicator;
  • the layer-by-layer summary module 330 is configured to summarize the statistical data under the name of the lowest-level structured indicator according to the hierarchical relationship between the names of the structured indicators, and to summarize them under the structural indicator name of the previous level.
  • the structured identifier creation module 310 includes:
  • the underlying indicator creation sub-module is configured to, for an indicator of a topology operation, sequentially combine the node identifier of the top-level hierarchical node to the lowest-level hierarchical node and the indicator identifier of the indicator into the lowest-level structured indicator name;
  • the upper indicator creation sub-module is configured to set, according to the lowest-level structured indicator name, the hierarchical node of the current level in the structured indicator name to the structured indicator name of each level, and obtain the upper level Structured indicator name.
  • the bottom layer indicator creation submodule includes:
  • the underlying metrics are separated into sub-modules, which are used to index the nodes of the top-level to the lowest-level hierarchical nodes and the indicator identifiers of the indicators, and splicing them into the lowest-level structured indicator names. .
  • the upper layer indicator creation submodule includes:
  • the upper-level indicator separation creation sub-module is configured to set, according to the lowest-level structured indicator name, the hierarchical node corresponding to the current level separator in the structured indicator name to the structured indicator name of each level. , get the name of the structural indicator of the previous level.
  • the method before the structured identifier creation module 310, the method further includes:
  • the registration module is configured to register the indicator identifier corresponding to the lowest level hierarchical node to the system.
  • the method further includes:
  • a data storage module for exporting statistics under each structured indicator name to a database for storage.
  • the data storage module includes:
  • the data storage sub-module is used to export the statistics under each structured indicator name to the database, and use the structured indicator name and time stamp as keywords and store the statistical data as key values.
  • the embodiment of the present application in the real-time computing system, has a hierarchical structure when processing data, and the real-time computing system can quickly perform statistics on all the levels of the indicators.
  • the embodiment of the present application A structured indicator name is created for each hierarchical node, and the hierarchical relationship between the hierarchical nodes is determined by the structured indicator name itself, thereby determining the summary relationship. Therefore, only the data of the lowest-level structured indicator name needs to be counted, and the data can be summarized step by step according to the hierarchical relationship between the structured indicator names, and the data of each level corresponding to the index can be obtained.
  • the hierarchical relationship of the structured indicator names can be used to simply count the indicators of each level, the logic is simple, the system consumption can be reduced, and the hierarchical relationship established according to the structured indicator name can be convenient. Add or remove one or several levels for easy expansion.
  • FIG. 4 a structural block diagram of an embodiment of an indicator statistical system of the present application is shown, which may specifically include:
  • the scheduling server 410 each computing node 420, database 430.
  • Each computer node 420 is exemplarily shown in FIG. 4, and the actual application may be set by the computer node according to the needs of the cluster.
  • Each computer node includes a registration module 421, an underlying metric separation creation module 422, an upper metric separation creation module 423, an underlying metric monitoring module 424, and a layer by layer summary module 425.
  • the dispatch server includes a summary module 411 and a data storage sub-module 412.
  • each computer node may also include other required modules, which are not limited in the embodiment of the present application.
  • the above scheduling server 410 includes:
  • the summary module 411 is configured to obtain statistics of each structured indicator name from each computing node, and perform summary
  • the data storage sub-module 412 is configured to export the statistical data under each structured indicator name to the database 430.
  • the database 430 stores the structured indicator name and time stamp as keywords and stores the statistical data as a key value.
  • Each compute node 420 includes:
  • the registration module 421 is configured to register, to the system of the computing node, an indicator identifier corresponding to the lowest level hierarchical node.
  • the bottom layer indicator separation creation module 422 is configured to use a separator to select a node identifier of the topmost layer to the lowest level node node and an indicator identifier of the indicator, and sequentially splicing into a bottom layer structured indicator name. .
  • the upper-level indicator separation creation module 423 is configured to set, according to the lowest-level structured indicator name, the hierarchical node corresponding to the current level separator in the structured indicator name to the structured indicator name of each level. , get the name of the structural indicator of the previous level.
  • the bottom layer indicator monitoring module 424 is configured to perform statistics on the bottommost structured index name after monitoring the lowest level data corresponding to the indicator;
  • the layer-by-layer summary module 425 is configured to summarize the statistical data under the name of the lowest-level structured indicator according to the hierarchical relationship between the names of the structured indicators, and to summarize them under the structural indicator name of the upper level.
  • the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.
  • embodiments of the embodiments of the present application can be provided as a method, apparatus, or computer program product. Therefore, the embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, embodiments of the present application may be included in one or more of them.
  • a computer program product embodied on a computer usable storage medium including but not limited to disk storage, CD@ROM, optical storage, etc.).
  • the computer device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
  • the memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory.
  • RAM random access memory
  • ROM read only memory
  • Memory is an example of a computer readable medium.
  • Computer readable media includes both permanent and non-persistent, removable and non-removable media.
  • Information storage can be implemented by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, read-only optical read-only memory (CD@ROM), digital versatile disc (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape storage or other magnetic storage devices or any other non-transportable media can be used to store information that can be accessed by a computing device.
  • computer readable media does not include non-persistent computer readable media, such as modulated data signals and carrier waves.
  • Embodiments of the present application are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG.
  • These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing terminal device to produce a machine such that instructions are executed by a processor of a computer or other programmable data processing terminal device
  • Means are provided for implementing the functions specified in one or more of the flow or in one or more blocks of the flow chart.
  • the computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device.
  • the instruction device implements the functions specified in one or more blocks of the flowchart or in a flow or block of the flowchart.

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Strategic Management (AREA)
  • Accounting & Taxation (AREA)
  • Development Economics (AREA)
  • Finance (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Game Theory and Decision Science (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

La présente invention se rapporte au domaine technique du calcul en temps réel et concerne un procédé et un dispositif de comptage de pointeurs. Le procédé consiste à : créer, conformément à des nœuds de niveaux hiérarchiques respectifs et correspondant à un pointeur d'une opération topologique, un nom de pointeur structuré du pointeur, une relation hiérarchique entre les nœuds des niveaux hiérarchiques respectifs pouvant être déterminée au moyen du nom de pointeur structuré (110) ; lors de la détection de données de niveau le plus bas correspondant au pointeur, exécuter une opération de comptage de noms de pointeurs structurés correspondant au niveau le plus bas (120) ; et, sur la base des données issues de l'opération de comptage des noms de pointeurs structurés du niveau le plus bas, agréger, niveau par niveau et conformément à la relation hiérarchique entre les noms de pointeurs structurés respectifs, les données de noms de pointeurs structurés dans des niveaux supérieurs successifs (130). La présente invention permet de réduire les coûts de système, et permet d'ajouter ou de supprimer un ou plusieurs niveaux du fait de la constitution d'une relation hiérarchique en association avec des noms de pointeurs structurés, ce qui permet de faciliter une extension.
PCT/CN2016/107017 2015-12-04 2016-11-24 Procédé et dispositif de comptage de pointeurs WO2017092600A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510886186.XA CN106846021A (zh) 2015-12-04 2015-12-04 一种指标统计方法和装置
CN201510886186.X 2015-12-04

Publications (1)

Publication Number Publication Date
WO2017092600A1 true WO2017092600A1 (fr) 2017-06-08

Family

ID=58796252

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/107017 WO2017092600A1 (fr) 2015-12-04 2016-11-24 Procédé et dispositif de comptage de pointeurs

Country Status (2)

Country Link
CN (1) CN106846021A (fr)
WO (1) WO2017092600A1 (fr)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107832267A (zh) * 2017-11-06 2018-03-23 中国银行股份有限公司 一种统计数据汇总方法及装置
CN109963293A (zh) * 2017-12-25 2019-07-02 中国移动通信集团上海有限公司 一种性能指标优化方法及装置
CN113177725A (zh) * 2021-05-18 2021-07-27 浙江捷创智能技术有限公司 一种可配置树型结构的能源统计方法
CN115965296A (zh) * 2023-03-17 2023-04-14 建信金融科技有限责任公司 考核数据的处理方法、装置、设备、产品及可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060285543A1 (en) * 2003-12-12 2006-12-21 British Telecommunications Public Limited Company Distributed computer system
CN102637200A (zh) * 2012-03-07 2012-08-15 江苏引跑网络科技有限公司 一种使多级关联数据分配到集群相同节点的方法
CN102841891A (zh) * 2011-06-21 2012-12-26 金蝶软件（中国）有限公司 一种树状结构节点的排序方法、装置及查询系统
CN102867059A (zh) * 2012-09-19 2013-01-09 浪潮(北京)电子信息产业有限公司 一种树形结构中数据的处理方法和系统
CN102929587A (zh) * 2012-09-28 2013-02-13 用友软件股份有限公司 数据处理系统和数据处理方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102306199A (zh) * 2011-09-22 2012-01-04 用友软件股份有限公司 数据管理装置和数据管理方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060285543A1 (en) * 2003-12-12 2006-12-21 British Telecommunications Public Limited Company Distributed computer system
CN102841891A (zh) * 2011-06-21 2012-12-26 金蝶软件（中国）有限公司 一种树状结构节点的排序方法、装置及查询系统
CN102637200A (zh) * 2012-03-07 2012-08-15 江苏引跑网络科技有限公司 一种使多级关联数据分配到集群相同节点的方法
CN102867059A (zh) * 2012-09-19 2013-01-09 浪潮(北京)电子信息产业有限公司 一种树形结构中数据的处理方法和系统
CN102929587A (zh) * 2012-09-28 2013-02-13 用友软件股份有限公司 数据处理系统和数据处理方法

Also Published As

Publication number Publication date
CN106846021A (zh) 2017-06-13

Similar Documents

Publication Publication Date Title
US11797618B2 (en) Data fabric service system deployment
Zheng et al. Service-generated big data and big data-as-a-service: an overview
US20190095510A1 (en) Low-latency streaming analytics
WO2017092600A1 (fr) Procédé et dispositif de comptage de pointeurs
JP2019517040A (ja) クラウドプラットフォームベースのクライアントアプリケーション情報統計方法および装置
US9201700B2 (en) Provisioning computer resources on a network
US11755531B1 (en) System and method for storage of data utilizing a persistent queue
US11816511B1 (en) Virtual partitioning of a shared message bus
CN114443437A (zh) 告警根因输出方法、装置、设备、介质和程序产品
TWI726041B (zh) 指標統計方法和裝置
Jayanthi et al. A framework for real-time streaming analytics using machine learning approach
US11853330B1 (en) Data structure navigator
US20230315726A1 (en) Enriching metrics with application execution context
Zhang et al. View-driven Real-time Sensor Data Processing and Servitization System
Wu et al. Stream Processing
CN114254051A (zh) 一种大数据计算方法、装置及大数据平台
CN112988729A (zh) 一种基于storm输电监测数据处理方法
Li et al. Design and implementation of a distributed Complex Event Processing system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16869910

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16869910

Country of ref document: EP

Kind code of ref document: A1