CN110795483A - Data statistical method and device and electronic equipment - Google Patents
Data statistical method and device and electronic equipment Download PDFInfo
- Publication number
- CN110795483A CN110795483A CN201911001137.8A CN201911001137A CN110795483A CN 110795483 A CN110795483 A CN 110795483A CN 201911001137 A CN201911001137 A CN 201911001137A CN 110795483 A CN110795483 A CN 110795483A
- Authority
- CN
- China
- Prior art keywords
- data
- application server
- statistics
- group
- entries
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Abstract
The embodiment of the invention provides a data statistical method, a data statistical device, electronic equipment and a storage medium. The method comprises the following steps: collecting data items which are delivered by each application server and contain data to be counted; wherein each data entry comprises: the identification of the application server, the data type of the data to be counted and the data content of the data to be counted; in response to a request for counting data of any application server, screening data items containing the identification of the application server from the collected data items; and respectively counting the data content in each group of data items with the same data type in the screened data items to obtain the data counting results of each data type of the application server. The data statistical method provided by the embodiment of the invention can realize the statistics of the data to be counted of the new application server on the premise of not developing a new data transmission interface.
Description
Technical Field
The invention relates to the technical field of software, in particular to a data statistical method and device and electronic equipment.
Background
In the field of software technology, data statistics of an application server is generally performed by a data statistics server corresponding to the application server, one data statistics server generally corresponds to a plurality of application servers, and data types of data to be counted of different application servers are generally different.
In the prior art, a method for a data statistics server to count data of a plurality of application servers includes: respectively utilizing a data transmission interface matched with the data type of the data to be counted of each application server to collect the data to be counted sent by each application server; and respectively counting the collected data to be counted of each application server to obtain data counting results of various data types of each application server.
However, the inventor finds that the prior art has at least the following problems in the process of implementing the invention:
when data to be counted of a new application server needs to be counted, the original data output interface may not match with the data type of the data to be counted of the new application server, so that the data to be counted of the new application server cannot be counted by the data counting server, and a new data transmission interface needs to be developed.
Disclosure of Invention
The embodiment of the invention aims to provide a data statistical method, so as to realize the statistics of data to be counted of a new application server on the premise of not developing a new data transmission interface. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a data statistics method, which is applied to a data statistics server, and the method includes:
collecting data items containing data to be counted and delivered by each application server through a preset data transmission interface; wherein each data entry comprises: the identification of the application server, the data type of the data to be counted and the data content of the data to be counted;
in response to a request for counting data of any application server, screening data items containing the identification of the application server from the collected data items;
and respectively counting the data content in each group of data items with the same data type in the screened data items to obtain the data counting results of each data type of the application server.
Optionally, the separately counting data contents in each group of data entries with the same data type in the screened data entries to obtain data statistics results of various data types of the application server includes:
and for each group of data entries with the same data type in the screened data entries, counting the data content of the group of data entries according to the statistical processing logic defined by the data type contained in the group of data entries to obtain the data statistical result of the data type contained in the group of data entries of the application server.
Optionally, the data content contained in each data entry is one or more key-value pairs;
the step of counting the data content of the group of data entries according to the statistical processing logic defined for the data types included in the group of data entries to obtain the data statistical result of the data types included in the group of data entries of the application server includes:
determining each group of key-value pairs containing the same key in the data content of the group of data items;
and according to the statistical processing logic which is respectively corresponding to each key in the statistical processing logic defined for the data types contained in the group of data entries, respectively counting the values of the determined key-value pairs of each group to obtain the statistical values of each key, wherein the statistical values are used as the data statistical results of the data types contained in the group of data entries of the application server.
Optionally, the method further comprises:
storing the collected data items in a preset first database; the first database is used for storing the data items of each application server collected by the data statistics server;
responding to a first query request of data items of any one or more application servers, and acquiring and feeding back the data items requested to be queried by the first query request from the first database;
wherein, the first query request at least carries: an identification of the application server to which the requested query data entry belongs, and/or a data type of the requested query data entry.
Optionally, the method further comprises:
storing the data statistical results of each application server under various data types into a second database corresponding to each application service; each second database is used for storing the data statistical result of the corresponding application server;
responding to a second query request of data statistics results of any one or more application servers, and acquiring and feeding back the data statistics results requested to be queried by the second query request from the second database;
wherein the second query request at least carries: an identification of an application server to which the requested query's statistics belong, and/or a data type of the requested query's statistics.
In a second aspect, an embodiment of the present invention provides a data statistics apparatus, which is applied to a data statistics server, and the apparatus includes:
the collection module is used for collecting data items containing data to be counted and delivered by each application server through a preset data transmission interface; wherein each data entry comprises: the identification of the application server, the data type of the data to be counted and the data content of the data to be counted;
the screening module is used for responding to a request for counting the data of any application server and screening the data items containing the identification of the application server from the collected data items;
and the counting module is used for counting the data content in each group of data items with the same data type in the screened data items respectively to obtain the data counting results of each data type of the application server.
Optionally, the statistics module respectively performs statistics on data contents in each group of data entries with the same data type in the screened data entries to obtain data statistics results of various data types of the application server, where the statistics results include:
and for each group of data entries with the same data type in the screened data entries, counting the data content of the group of data entries according to the statistical processing logic defined by the data type contained in the group of data entries to obtain the data statistical result of the data type contained in the group of data entries of the application server.
Optionally, the data content contained in each data entry is one or more key-value pairs;
the statistical module is used for performing statistics on the data content of the group of data entries according to statistical processing logic defined by the data types contained in the group of data entries to obtain data statistical results of the data types contained in the group of data entries of the application server, and the statistical processing logic comprises the following steps:
determining each group of key-value pairs containing the same key in the data content of the group of data items;
and according to the statistical processing logic which is respectively corresponding to each key in the statistical processing logic defined for the data types contained in the group of data entries, respectively counting the values of the determined key-value pairs of each group to obtain the statistical values of each key, wherein the statistical values are used as the data statistical results of the data types contained in the group of data entries of the application server.
Optionally, the apparatus further comprises: the device comprises a first storage module and a first feedback module;
the first storage module is used for storing the collected data items into a preset first database; the first database is used for storing the data items of each application server collected by the data statistics server;
the first feedback module is used for responding to a first query request of data items of any one or more application servers, and acquiring and feeding back the data items requested to be queried by the first query request from the first database; wherein, the first query request at least carries: an identification of the application server to which the requested query data entry belongs, and/or a data type of the requested query data entry.
Optionally, the apparatus further comprises: the second storage module and the second feedback module;
the second storage module is used for storing the data statistical results of each application server under various data types into a second database corresponding to each application server; each second database is used for storing the data statistical result of the corresponding application server;
the second feedback module is used for responding to a second query request of data statistics results of any one or more application servers, and acquiring and feeding back the data statistics results requested to be queried by the second query request from the second database; wherein the second query request at least carries: an identification of an application server to which the requested query's statistics belong, and/or a data type of the requested query's statistics.
In a third aspect, an embodiment of the present invention provides a data statistics server, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing any one of the data statistical methods when executing the program stored in the memory.
In yet another aspect of the present invention, there is also provided a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to perform any of the above described data statistics methods.
In yet another aspect of the present invention, the present invention further provides a computer program product containing instructions, which when run on a computer, causes the computer to perform any of the above described data statistics methods.
In the data statistical method provided by the embodiment of the invention, a data statistical server collects data items containing data to be statistically transmitted by each application server through a preset data transmission interface; and each data entry collected includes: the identification of the application service, the data type of the data to be counted and the data content of the data to be counted. The data entry collected by the data transmission interface is used for transmitting the data of one data type of one application server at a time, so that the data transmission interface can be matched with the data type of the data to be counted of any application server. Therefore, the scheme can realize the statistics of the data to be counted of the new application server on the premise of not developing a new data transmission interface. Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1(a) is a flow chart of a data statistics method according to an embodiment of the present invention;
FIG. 1(b) is a diagram illustrating an application scenario of the data statistics method shown in FIG. 1 (a);
fig. 2 is a schematic structural diagram of a data statistics apparatus according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data statistics server according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.
In order to realize statistics on data to be counted of a new application server on the premise of not developing a new data transmission interface, the embodiment of the invention provides a data statistics method and device, electronic equipment and a storage medium.
The execution subject of the data statistics method provided by the embodiment of the invention can be a data statistics device, the device can be applied to a data statistics server, and the data statistics server can be communicated with one or more application servers, so that the data statistics of the application servers is realized. In addition, the data statistics server and each communicated application server can form a server cluster, so that a server with comprehensive functions is formed.
First, a data statistical method provided by an embodiment of the present invention will be described in detail. As shown in fig. 1(a), the method may include the steps of:
s10: collecting data items containing data to be counted and delivered by each application server through a preset data transmission interface, wherein each data item comprises: the identification of the application server, the data type of the data to be counted and the data content of the data to be counted.
The data transmission interface is a software interface commonly used by each application server. It will be appreciated that the fields defined in the data transfer interface for transferring data entries correspond one-to-one with the three fields included in the data entries collected by the data transfer interface. In practical applications, the data transmission interface may be a pingback interface. Here, the pingback interface is an interface for notifying the blog system that an article is cited.
It is understood that the application server may generate various data during the operation, and the data to be counted exists in the data. Therefore, the application server can generate the data to be counted as the data item to be delivered to the server.
In this step, each application server may write the generated data entry into a locally preset file; the document is then delivered to a statistics server. It will be appreciated that the data entries in the file are written into the file in the format defined by the data transfer interface described above. After the data statistics server collects the file, all data entries in the file can be obtained from the file, and additional data entries can also be obtained. In practical application, a general log collection program, such as a venus-agent, may be deployed in the data statistics server, and files of each application server are collected by the log collection program; then, the data statistics server acquires data items from each file and sends the acquired data items to a preset kafka cluster. Here, the kafka cluster is a cluster of distributed message queues.
In addition, in the file of each application server, it is reasonable that one data entry may be written per line, or that the written data entries are separated by a specified separator, and so on.
S20: and in response to a request for statistics on data of any application server, screening the collected data entries for data entries containing the identification of the application server.
Here, the request for statistics of the data of the application server may be understood as that the application server has a data statistics requirement, which is predetermined. In practical applications, for an application server with a data statistics requirement, a data statistics program may be set in the data statistics server for the application server, and the data statistics program may listen to the kafka cluster by using Flink, so as to collect data entries of the corresponding application server from the kafka cluster, and then perform step S20 and subsequent steps. Therefore, the data statistics server can realize the statistics of the data of the application server by running the data statistics program set for the application server in the running process. Where Flink is a distributed processing engine for both streaming data and batch data.
In one implementation, in order to ensure the privacy of data, the access authority of data entries may be set for each data statistics program in the data statistics server; that is, each data statistics program of the data statistics server can only access the data entries of the application server corresponding to the data statistics program; in this way, the data statistics server filters the data entry including the identifier of the application server from the collected data entries, which may specifically be: data entries having access rights are screened directly. Of course, when there is no need to ensure the privacy of the data, the data entry of the corresponding identified application server may be filtered from the collected data entries according to the identification of the application server in the manner shown in step S20.
S30: and respectively counting the data content in each group of data items with the same data type in the screened data items to obtain the data counting results of each data type of the application server.
It is understood that the data to be counted of each application server may have a plurality of data types; therefore, the data types included in the data entries screened in step S20 may be various. Therefore, in step S30, data statistics results for each data type can be obtained by performing statistics on the data contents in each set of data entries with the same data type. In addition, during statistics, the data statistics manners of different data types may also be different, and for clarity of the scheme and clarity of layout, a specific implementation manner of performing statistics on the data content in each group of data items with the same data type included in the screened data items is described in the following.
In the data statistical method provided by the embodiment of the invention, a data statistical server collects data items containing data to be statistically transmitted by each application server through a preset data transmission interface; and each data entry collected includes: the identification of the application service, the data type of the data to be counted and the data content of the data to be counted. The data entry collected by the data transmission interface is used for transmitting the data of one data type of one application server at a time, so that the data transmission interface can be matched with the data type of the data to be counted of any application server. Therefore, the scheme can realize the statistics of the data to be counted of the new application server on the premise of not developing a new data transmission interface.
For clarity of the scheme, a specific implementation manner of the data statistics server performing statistics on data contents in each set of data entries with the same data type in the screened data entries is described below.
For example, in an implementation manner, the statistics server separately performs statistics on data contents in each group of data entries with the same data type in the screened data entries to obtain data statistics results of various data types of the application server, and the data statistics results may include:
and for each group of data entries with the same data type in the screened data entries, counting the data content of the group of data entries according to the statistical processing logic defined by the data type contained in the group of data entries to obtain the data statistical result of the data type contained in the group of data entries of the application server.
The statistical processing logic may include, but is not limited to, statistics of values, statistics of numbers, statistics of content categories, and/or logical jumps of data contents. In practical application, each data type can correspond to a message processing object Handler, so that statistical processing on data contents is realized by using the Handler. Here, Handler is a tool used in the android system to implement communication between a main thread and a sub-thread.
It is understood that, in this implementation, the statistical processing logic defined for the data type included in each group of data entries is the statistical processing logic defined in the data statistical program corresponding to the application server. The statistical processing logic defined for each application server may be the same or different for different data types.
In addition, the data content contained in each data entry may be a single numerical value or may be one or more key-value pairs.
When the data content included in each data entry collected by the data statistics server is a single numerical value, in the implementation manner, according to the statistics processing logic defined by the data type included in the group of data entries, performing statistics on the data content of the group of data entries to obtain the data statistics result of the data type included in the group of data entries of the application server, may include:
and calculating statistics of the data types contained in the group of data entries according to a calculation mode defined by the data types contained in the group of data entries, wherein the statistics is used as a data statistics result of the data types contained in the group of data entries of the application server.
It is understood that the calculation method herein is the calculation method defined in the above statistical processing logic.
For example, assume that there is a set of data entries that contain the same type of data, including: data entry 1 and data entry 2; wherein, the data entry 1 is: "A, data type a, 5", data entry 2 is: "A, data type a, 6"; wherein "a" is the identifier of the application server a, and "5" and "6" are both numerical values contained in the data entry; assuming that the calculation method defined by the data types contained in the set of data entries is summation, the statistic of the data types contained in the set of data entries is 11.
In addition, when the data content included in each data entry is one or more key-value pairs, in the foregoing implementation, according to the statistical processing logic defined for the data types included in the group of data entries, performing statistics on the data content of the group of data entries to obtain the data statistical result of the data types included in the group of data entries of the application server, may include:
determining each group of key-value pairs containing the same key in the data content of the group of data items;
and according to the statistical processing logic which is respectively corresponding to each key in the statistical processing logic defined for the data types contained in the group of data entries, respectively counting the values of the determined key-value pairs of each group to obtain the statistical values of each key, wherein the statistical values are used as the data statistical results of the data types contained in the group of data entries of the application server.
It is understood that, in the statistical processing logic defined for the data type included in the set of data entries, the statistical processing logic corresponding to each key may be the same or different.
For example, assume that there is a set of data entries that contain the same type of data, including: data entry 4 and data entry 5; wherein, the data entry 4 is: "B, movie play data, { play time length: 60, scoring: 60} "; data entry 5 is: "B, movie play data, { play time length: 20, scoring: 50 }; wherein, the 'B' is the mark of the application server B, the 'movie play data' is the data type, and two key values in the brace are the data content; assuming that the statistical process corresponding to the key of "play time length" is summation and the statistical process corresponding to the key of "score" is averaging, the statistical value of the key of "play time length" is 80 and the statistical value of the key of "movie score" is 55.
In addition, when a data entry contains multiple key value pairs, multiple key-value pairs may be separated using a predetermined separator, such as a tab. The delimiters used here may be set to be different from delimiters for separating individual data items in a file collected by the data statistics server. In practical applications, each key-value pair in a data entry may be identified by a json (java Script Object notification) string. Here, JSON is a lightweight format for storing and exchanging data.
Based on the above embodiments, it can be seen that the data statistics method provided by the embodiments of the present invention can implement statistics on data of any data type of the application server.
Optionally, with reference to the embodiment corresponding to fig. 1, in a further embodiment provided by the present invention, the data statistics method provided in the embodiment of the present invention may further include:
storing the collected data items in a preset first database; the first database is used for storing data items of each application server collected by the data statistics server;
responding to a first query request for data items of any one or more application servers, and acquiring and feeding back the data items requested to be queried by the first query request from a first database;
wherein, the first query request at least carries: an identification of the application server to which the requested query data entry belongs, and/or a data type of the requested query data entry.
It is understood that the first database stores the total number of data entries of all servers corresponding to the data statistics server. Here, the total number of data entries means data entries including all of the historical collections. Therefore, when the data statistical result of the data statistical program in the data statistical server has an error, the data entries can be inquired according to the total number of the data entries, and the reason why the data statistical result has the error is positioned.
In practical applications, in the data statistics server, the flink may be used to listen to the kafka cluster, so that the full amount of data entries from the kafka cluster are collected and stored in one of the druids according to the original field of the data entry. Here, the pipeline is a platform that can consume and explore real-time data on large data sets.
In this implementation, the first query request may be provided by a data statistics program in the data statistics server. Or may be given by an administrator of the statistics server. Based on the consideration of data privacy, when the first query request is given by a data statistics program in the data statistics server, if a data entry requested to be queried by the data statistics program does not belong to an application server corresponding to the data statistics program, the data statistics server may refuse to feed back the data entry requested to be queried by the first query request. Of course, when there is no need to ensure the privacy of the data, any data statistical program can query all the data entries of the application servers. In addition, when the first query request is given by an administrator of the data statistics server, the first query request may feed back the data entry of the corresponding application server in response to the identification of one or more application servers given by the administrator in the first query request.
Optionally, with reference to the embodiment corresponding to fig. 1, in a further embodiment provided by the present invention, the data statistics method provided in the embodiment of the present invention may further include:
storing the data statistical results of each application server under various data types into a second database corresponding to each application server; each second database is used for storing the data statistical result of the corresponding application server;
responding to a second query request of data statistics results of any one or more application servers, and acquiring and feeding back the data statistics results requested to be queried by the second query request from the second database;
wherein the second query request at least carries: an identification of an application server to which the requested query's statistics belong, and/or a data type of the requested query's statistics.
It can be understood that the data statistical result of the application server is stored in the second database, which is convenient for subsequent operations such as generating a report according to the data statistical result and outputting the report.
In addition, the second query request may be provided by a data statistics program in the data statistics server. Or may be given by an administrator of the statistics server. For consideration of data privacy, reference may be made to an implementation manner of the first query request, which is not described herein again.
For clarity of the scheme, a specific application scenario is taken as an example below to further illustrate the data statistics method provided by the embodiment of the present invention. As shown in fig. 1(b), in the application scenario, a venus-agent is respectively provided in the application server 1, the application server 2, and the application server N, and is used for collecting a file recording data entries in each application server; and each application server respectively sends the file of the application server to a kafka cluster preset in the data statistics server through a preset data transmission interface. In the data statistics server, for each application server, setting a flink data statistics program for the application server, such as the flink data statistics program for the application server 1, the flink data statistics program for the application server 2, and the flink data statistics program for the application server N shown in fig. 1 (b); and the flink data statistical program of each application server is used for collecting data entries of the application server from the kafa cluster and saving the collected data entries to a second database corresponding to the application server. In addition, a full amount of flink data statistical program is also arranged in the data statistical server and used for collecting full amount of data items from the kafka cluster and storing the data items in the first database according to original fields of the data items; therefore, fault location is convenient for the subsequent data statistical result with errors. Wherein the first database is a database constructed based on the pipeline.
As can be seen from fig. 1(b), when the data statistics server needs to count the data of the newly added application server, it is only necessary to set a flink data statistics program for the newly added application server in the data statistics server, and assign a corresponding second database for the application server. And a data transmission interface between the newly added application server and the data statistics server does not need to be re-developed.
Corresponding to the foregoing data statistics method, an embodiment of the present invention further provides a data statistics apparatus, where the apparatus is applied in a data statistics server, as shown in fig. 2, the apparatus may include:
a collecting module 201, configured to collect, through a preset data transmission interface, data items that include data to be counted and are delivered by each application server; wherein each data entry comprises: the identification of the application server, the data type of the data to be counted and the data content of the data to be counted;
a screening module 202, configured to, in response to a request for performing statistics on data of any application server, screen a data entry including an identifier of the application server from the collected data entries;
the statistics module 203 is configured to perform statistics on data contents in each set of data entries with the same data type in the screened data entries, respectively, to obtain data statistics results of various data types of the application server.
Optionally, the statistics module 203 respectively performs statistics on data contents in each group of data entries with the same data type in the screened data entries to obtain data statistics results of various data types of the application server, where the statistics results include:
and for each group of data entries with the same data type in the screened data entries, counting the data content of the group of data entries according to the statistical processing logic defined by the data type contained in the group of data entries to obtain the data statistical result of the data type contained in the group of data entries of the application server.
Optionally, the data content contained in each data entry is one or more key-value pairs;
the statistical module 203, according to the statistical processing logic defined for the data types included in the group of data entries, performs statistics on the data contents of the group of data entries to obtain the data statistical result of the data types included in the group of data entries of the application server, including:
determining each group of key-value pairs containing the same key in the data content of the group of data items;
and according to the statistical processing logic which is respectively corresponding to each key in the statistical processing logic defined for the data types contained in the group of data entries, respectively counting the values of the determined key-value pairs of each group to obtain the statistical values of each key, wherein the statistical values are used as the data statistical results of the data types contained in the group of data entries of the application server.
Optionally, the apparatus further comprises: the device comprises a first storage module and a first feedback module;
the first storage module is used for storing the collected data items into a preset first database; the first database is used for storing the data items of each application server collected by the data statistics server;
the first feedback module is used for responding to a first query request of data items of any one or more application servers, and acquiring and feeding back the data items requested to be queried by the first query request from the first database; wherein, the first query request at least carries: an identification of the application server to which the requested query data entry belongs, and/or a data type of the requested query data entry.
Optionally, the apparatus further comprises: the second storage module and the second feedback module;
the second storage module is used for storing the data statistical results of each application server under various data types into a second database corresponding to each application server; each second database is used for storing the data statistical result of the corresponding application server;
the second feedback module is used for responding to a second query request of data statistics results of any one or more application servers, and acquiring and feeding back the data statistics results requested to be queried by the second query request from the second database; wherein the second query request at least carries: an identification of an application server to which the requested query's statistics belong, and/or a data type of the requested query's statistics.
According to the data statistics device provided by the embodiment of the invention, the data statistics server collects data items containing data to be counted, which are sent by each application server, through a preset data transmission interface; and each data entry collected includes: the identification of the application service, the data type of the data to be counted and the data content of the data to be counted. The data entry collected by the data transmission interface is used for transmitting the data of one data type of one application server at a time, so that the data transmission interface can be matched with the data type of the data to be counted of any application server. Therefore, the scheme can realize the statistics of the data to be counted of the new application server on the premise of not developing a new data transmission interface.
The embodiment of the present invention further provides an electronic device, as shown in fig. 3, which includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, where the processor 301, the communication interface 302, and the memory 303 complete mutual communication through the communication bus 304,
a memory 303 for storing a computer program;
the processor 301 is configured to implement any of the above-described data statistical methods when executing the program stored in the memory 303.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In yet another embodiment of the present invention, there is also provided a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to perform the data statistics method of any of the above embodiments.
In yet another embodiment, there is provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the data statistics method of any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the device, the electronic apparatus and the storage medium embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and the relevant points can be referred to the partial description of the method embodiments.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (11)
1. A data statistics method is applied to a data statistics server, and the method comprises the following steps:
collecting data items containing data to be counted and delivered by each application server through a preset data transmission interface; wherein each data entry comprises: the identification of the application server, the data type of the data to be counted and the data content of the data to be counted;
in response to a request for counting data of any application server, screening data items containing the identification of the application server from the collected data items;
and respectively counting the data content in each group of data items with the same data type in the screened data items to obtain the data counting results of each data type of the application server.
2. The method according to claim 1, wherein the separately counting the data content in each group of data entries with the same data type in the screened data entries to obtain the data statistics results of each data type of the application server comprises:
and for each group of data entries with the same data type in the screened data entries, counting the data content of the group of data entries according to the statistical processing logic defined by the data type contained in the group of data entries to obtain the data statistical result of the data type contained in the group of data entries of the application server.
3. The method of claim 2, wherein the data content contained in each data entry is one or more key-value pairs;
the step of counting the data content of the group of data entries according to the statistical processing logic defined for the data types included in the group of data entries to obtain the data statistical result of the data types included in the group of data entries of the application server includes:
determining each group of key-value pairs containing the same key in the data content of the group of data items;
and according to the statistical processing logic which is respectively corresponding to each key in the statistical processing logic defined for the data types contained in the group of data entries, respectively counting the values of the determined key-value pairs of each group to obtain the statistical values of each key, wherein the statistical values are used as the data statistical results of the data types contained in the group of data entries of the application server.
4. The method of claim 1, further comprising:
storing the collected data items in a preset first database; the first database is used for storing the data items of each application server collected by the data statistics server;
responding to a first query request of data items of any one or more application servers, and acquiring and feeding back the data items requested to be queried by the first query request from the first database;
wherein, the first query request at least carries: an identification of the application server to which the requested query data entry belongs, and/or a data type of the requested query data entry.
5. The method of claim 1, further comprising:
storing the data statistical results of each application server under various data types into a second database corresponding to each application service; each second database is used for storing the data statistical result of the corresponding application server;
responding to a second query request of data statistics results of any one or more application servers, and acquiring and feeding back the data statistics results requested to be queried by the second query request from the second database;
wherein the second query request at least carries: an identification of an application server to which the requested query's statistics belong, and/or a data type of the requested query's statistics.
6. A data statistics apparatus, applied to a data statistics server, the apparatus comprising:
the collection module is used for collecting data items containing data to be counted and delivered by each application server through a preset data transmission interface; wherein each data entry comprises: the identification of the application server, the data type of the data to be counted and the data content of the data to be counted;
the screening module is used for responding to a request for counting the data of any application server and screening the data items containing the identification of the application server from the collected data items;
and the counting module is used for counting the data content in each group of data items with the same data type in the screened data items respectively to obtain the data counting results of each data type of the application server.
7. The apparatus of claim 6, wherein the statistics module respectively performs statistics on data contents in each group of data entries with the same data type in the screened data entries to obtain data statistics results of each data type of the application server, and the statistics module includes:
and for each group of data entries with the same data type in the screened data entries, counting the data content of the group of data entries according to the statistical processing logic defined by the data type contained in the group of data entries to obtain the data statistical result of the data type contained in the group of data entries of the application server.
8. The apparatus of claim 7, wherein the data content contained in each data entry is one or more key-value pairs;
the statistical module is used for performing statistics on the data content of the group of data entries according to statistical processing logic defined by the data types contained in the group of data entries to obtain data statistical results of the data types contained in the group of data entries of the application server, and the statistical processing logic comprises the following steps:
determining each group of key-value pairs containing the same key in the data content of the group of data items;
and according to the statistical processing logic which is respectively corresponding to each key in the statistical processing logic defined for the data types contained in the group of data entries, respectively counting the values of the determined key-value pairs of each group to obtain the statistical values of each key, wherein the statistical values are used as the data statistical results of the data types contained in the group of data entries of the application server.
9. The apparatus of claim 6, further comprising: the device comprises a first storage module and a first feedback module;
the first storage module is used for storing the collected data items into a preset first database; the first database is used for storing the data items of each application server collected by the data statistics server;
the first feedback module is used for responding to a first query request of data items of any one or more application servers, and acquiring and feeding back the data items requested to be queried by the first query request from the first database; wherein, the first query request at least carries: an identification of the application server to which the requested query data entry belongs, and/or a data type of the requested query data entry.
10. The apparatus of claim 6, further comprising: the second storage module and the second feedback module;
the second storage module is used for storing the data statistical results of each application server under various data types into a second database corresponding to each application server; each second database is used for storing the data statistical result of the corresponding application server;
the second feedback module is used for responding to a second query request of data statistics results of any one or more application servers, and acquiring and feeding back the data statistics results requested to be queried by the second query request from the second database; wherein the second query request at least carries: an identification of an application server to which the requested query's statistics belong, and/or a data type of the requested query's statistics.
11. A data statistics server is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for completing mutual communication through the communication bus by the memory;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 5 when executing a program stored in the memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911001137.8A CN110795483A (en) | 2019-10-21 | 2019-10-21 | Data statistical method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911001137.8A CN110795483A (en) | 2019-10-21 | 2019-10-21 | Data statistical method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110795483A true CN110795483A (en) | 2020-02-14 |
Family
ID=69440506
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911001137.8A Pending CN110795483A (en) | 2019-10-21 | 2019-10-21 | Data statistical method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110795483A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101873226A (en) * | 2010-06-21 | 2010-10-27 | 中兴通讯股份有限公司 | Data storage method and device for statistical form system |
| CN104391781A (en) * | 2014-10-24 | 2015-03-04 | 苏州阔地网络科技有限公司 | Processing method and system for log information |
| CN106452933A (en) * | 2015-08-05 | 2017-02-22 | 腾讯科技（北京）有限公司 | Service data statistical method, device and system |
| US20170075954A1 (en) * | 2015-09-16 | 2017-03-16 | International Business Machines Corporation | Identification and elimination of non-essential statistics for query optimization |
-
2019
- 2019-10-21 CN CN201911001137.8A patent/CN110795483A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101873226A (en) * | 2010-06-21 | 2010-10-27 | 中兴通讯股份有限公司 | Data storage method and device for statistical form system |
| CN104391781A (en) * | 2014-10-24 | 2015-03-04 | 苏州阔地网络科技有限公司 | Processing method and system for log information |
| CN106452933A (en) * | 2015-08-05 | 2017-02-22 | 腾讯科技（北京）有限公司 | Service data statistical method, device and system |
| US20170075954A1 (en) * | 2015-09-16 | 2017-03-16 | International Business Machines Corporation | Identification and elimination of non-essential statistics for query optimization |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10977256B2 (en) | System for aggregation and prioritization of IT asset field values from real-time event logs and method thereof | |
| CN110262918B (en) | Process crash analysis method and device, distributed equipment and storage medium | |
| CN110932933B (en) | Network condition monitoring method, computing device and computer storage medium | |
| CN109995612B (en) | Service inspection method and device and electronic equipment | |
| CN111338888B (en) | Data statistics method and device, electronic equipment and storage medium | |
| CN113342689A (en) | Automatic testing method and device for interface, electronic equipment and storage medium | |
| CN112835885B (en) | Processing method, device and system for distributed form storage | |
| CN112948064A (en) | Data reading method and device and data reading system | |
| US11341842B2 (en) | Metering data management system and computer readable recording medium | |
| CN107885634B (en) | Method and device for processing abnormal information in monitoring | |
| CN111694793A (en) | Log storage method and device and log query method and device | |
| CN113010494A (en) | Database auditing method and device and database proxy server | |
| CN112965943A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112732711A (en) | Data storage method and device and electronic equipment | |
| CN110795483A (en) | Data statistical method and device and electronic equipment | |
| CN115037653B (en) | Service flow monitoring method, device, electronic equipment and storage medium | |
| CN111338607A (en) | Method and device for selecting service data for testing and electronic equipment | |
| CN103530297A (en) | Method and device capable of automatically carrying out website analysis | |
| CN114157662A (en) | Cloud platform parameter adaptation method and device, terminal equipment and storage medium | |
| CN114039878A (en) | Network request processing method and device, electronic equipment and storage medium | |
| US20120173495A1 (en) | Computer Readable Medium, Systems, and Methods of Detecting a Discrepancy in a Chain-of-title of an Asset | |
| JP6926646B2 (en) | Inter-operator batch service management device and inter-operator batch service management method | |
| CN108763291B (en) | Data management method and device and electronic equipment | |
| CN111143318A (en) | Information processing method and device, electronic equipment and storage medium | |
| CN111163088B (en) | Message processing method, system and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |