CN110737655B - Method and device for reporting data - Google Patents
Method and device for reporting data Download PDFInfo
- Publication number
- CN110737655B CN110737655B CN201910998716.8A CN201910998716A CN110737655B CN 110737655 B CN110737655 B CN 110737655B CN 201910998716 A CN201910998716 A CN 201910998716A CN 110737655 B CN110737655 B CN 110737655B
- Authority
- CN
- China
- Prior art keywords
- data
- data message
- reported
- target
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Abstract
The embodiment of the disclosure discloses a method and a device for reporting data. One embodiment of the method comprises: the method comprises the steps that a data message is pulled from a preset data message set to serve as a target data message, wherein the preset data message set is used for storing the data message reported by at least one client, and the data message comprises data to be reported and data acquisition time; carrying out pre-check on the target data message to generate a pre-check result; in response to the fact that the pre-check result indicates that the pre-check is passed, performing data repeatability check on the data to be reported in the target data message according to the data acquisition time in the target data message to generate a data repeatability check result; and in response to determining that the data repeatability check result indicates that the target data message is inconsistent with the corresponding data message in the preset database, storing the target data message in the preset database. The implementation mode improves the efficiency of data reporting and saves network resources.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to a method and a device for reporting data.
Background
With the development of cloud computing technology, an IaaS (Infrastructure as a Service) system is also increasingly used. In an IaaS system, accuracy and timeliness of each attribute data of a computing resource are used as important indexes for evaluating performance of a resource management system, and a verification mechanism or an update mechanism is often needed to verify or update whether records of each attribute of the computing resource in an IaaS system are accurate and consistent with actual conditions. In this context, the reporting mechanism comes along. Reporting is a process of collecting, capturing and analyzing the resource attribute of the computing resource by using the agent of the computing resource, and finally informing the IAAS system server. The purpose of reporting is to check and update the attribute of the computing resource in time, and ensure the accuracy of IAAS system data.
There are generally three ways of correlation: one is that a pull command is sent to a managed client through a server, an information acquisition program of a client agent is triggered to execute, and then an acquired information is returned to the server in a primary path; secondly, the client agent reports the information of the client agent to a consumption queue (queue) at regular time and actively, and the server pulls the information from the queue for consumption; and thirdly, a combined reporting mode with client-side timing reporting as a main mode and server-side pulling as an auxiliary mode.
Disclosure of Invention
The embodiment of the disclosure provides a method and a device for reporting data.
In a first aspect, an embodiment of the present disclosure provides a method for reporting data, where the method includes: the method comprises the steps that a data message is pulled from a preset data message set to serve as a target data message, wherein the preset data message set is used for storing the data message reported by at least one client, and the data message comprises data to be reported and data acquisition time; carrying out pre-check on the target data message to generate a pre-check result; in response to the fact that the pre-check result indicates that the pre-check is passed, performing data repeatability check on data to be reported in the target data message according to the data acquisition time in the target data message to generate a data repeatability check result; and storing the target data message into the preset database in response to determining that the data repeatability check result indicates that the target data message is inconsistent with the corresponding data message in the preset database.
In some embodiments, the performing pre-check on the target data message to generate a pre-check result includes: matching the format of data to be reported in the target data message with a preset reported data format to generate a format matching result; and generating a pre-check result according to the format matching result.
In some embodiments, the data message in the data message set further includes an identifier of a source end of the data message; and the pre-check of the target data message to generate a pre-check result includes: verifying the identification of the data source end in the target data message according to a preset data source end verification rule to generate a source end verification result; and generating a pre-check result based on the source side check result.
In some embodiments, the generating a pre-verification result based on the source-side verification result includes: in response to the fact that the verification result of the source end indicates that verification is passed, acquiring a data message which is matched with the identification of the source end of the data in the target data message and is closest to the current time of the data acquisition time interval from a preset database as a matched data message; in response to determining that the data acquisition time in the matching data message is earlier than the data acquisition time in the target data message, generating a pre-verification result indicating that the pre-verification passes.
In some embodiments, the method further comprises: selecting a target client from at least one client; sending information representing the acquired reported data to a target client; and responding to the received data message reported by the target client, and storing the data message reported by the target client into a preset data message set.
In some embodiments, the selecting the target client from the at least one client includes: and selecting a client with the time interval of reporting data larger than a preset time interval from at least one client as a target client.
In some embodiments, the method further comprises: in response to receiving a data message reported by a client meeting an active reporting condition, storing the data message reported by the client meeting the active reporting condition into a preset data message set, wherein the active reporting condition comprises at least one of the following items: and detecting that the data to be reported locally at the client side changes, wherein the time interval from the last data reporting reaches a preset reporting period.
In some embodiments, the predetermined data message set includes a predetermined message queue, the predetermined database includes a Cyclic Redundancy Check (CRC) code corresponding to the data message, and the data repeatability Check includes a CRC Check.
In a second aspect, an embodiment of the present disclosure provides an apparatus for reporting data, where the apparatus includes: the system comprises a pulling unit, a processing unit and a processing unit, wherein the pulling unit is configured to pull a data message from a preset data message set as a target data message, the preset data message set is used for storing the data message reported by at least one client, and the data message comprises data to be reported and data acquisition time; the pre-check unit is configured to perform pre-check on the target data message and generate a pre-check result; the repeated checking unit is configured to respond to the fact that the result of the pre-checking indicates that the pre-checking is passed, and perform data repeatability checking on data to be reported in the target data message according to the data acquisition time in the target data message to generate a data repeatability checking result; a first storage unit configured to store the target data message in a preset database in response to determining that the data repeatability check result indicates that the target data message is inconsistent with a corresponding data message in the preset database.
In some embodiments, the pre-verification unit includes: the first generation module is configured to match the format of data to be reported in the target data message with a preset reported data format and generate a format matching result; and the second generation module is configured to generate a pre-check result according to the format matching result.
In some embodiments, the data message in the data message set further includes an identifier of a source end of the data message; and the pre-verification unit comprises: the third generation module is configured to verify the identifier of the data source end in the target data message according to a preset data source end verification rule to generate a source end verification result; and the fourth generation module is configured to generate a pre-check result based on the source-side check result.
In some embodiments, the fourth generating module includes: an obtaining sub-module configured to obtain, as a matching data message, a data message that matches the identification of the source end of the data in the target data message and is closest in data acquisition time to the current time, from a preset database in response to determining that the source end verification result indicates that the verification passes; a generation submodule configured to generate a pre-check result indicating that the pre-check passes in response to determining that the data acquisition time in the matching data message is earlier than the data acquisition time in the target data message.
In some embodiments, the apparatus further comprises: a selecting unit configured to select a target client from at least one client; the sending unit is configured to send information representing the acquired reported data to the target client; and the second storage unit is configured to store the data message reported by the target client into a preset data message set in response to receiving the data message reported by the target client.
In some embodiments, the selecting unit is further configured to select, as the target client, a client having a reporting time interval greater than a preset time interval from the at least one client.
In some embodiments, the apparatus further comprises: a third storage unit, configured to store, in response to receiving a data message reported by a client satisfying an active reporting condition, the data message reported by the client satisfying the active reporting condition into a preset data message set, where the active reporting condition includes at least one of: and detecting that the data to be reported locally at the client side changes, wherein the time interval from the last data reporting reaches a preset reporting period.
In some embodiments, the preset data message set includes a preset message queue, the preset database includes CRC codes corresponding to the data messages, and the data repeatability check includes CRC check.
In a third aspect, an embodiment of the present disclosure provides a system for reporting data, where the system includes: the client is configured to actively send a data message to the server in response to the fact that the data active reporting condition is met, wherein the data message comprises data to be reported and data acquisition time; the server is configured to respond to the received data message actively sent by the client and store the actively sent data message into a preset data message set; and implementing a method as described in any implementation form of the first aspect.
In some embodiments, the condition for actively reporting data includes detecting that data to be reported locally changes.
In a fourth aspect, an embodiment of the present disclosure provides a server, including: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any implementation of the first aspect.
In a fifth aspect, embodiments of the present disclosure provide a computer-readable medium on which a computer program is stored, which when executed by a processor implements the method as described in any of the implementations of the first aspect.
According to the method and the device for reporting data, provided by the embodiment of the disclosure, a data message is first pulled from a preset data message set as a target data message. The preset data message set is used for storing data messages reported by at least one client. The data message comprises data to be reported and data acquisition time. And then, carrying out pre-check on the target data message to generate a pre-check result. And then, in response to the fact that the pre-check result indicates that the pre-check is passed, performing data repeatability check on the data to be reported in the target data message according to the data acquisition time in the target data message to generate a data repeatability check result. And finally, in response to the fact that the data repeatability check result indicates that the target data message is inconsistent with the corresponding data message in the preset database, storing the target data message in the preset database. Therefore, the real-time performance of data updating is met, and system congestion caused by reporting of a large amount of data is prevented. And further, the efficient reporting of the data is realized, and the network resources and the computing resources of the server are effectively saved.
Drawings
Other features, objects and advantages of the disclosure will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present disclosure may be applied;
fig. 2 is a flow diagram of one embodiment of a method for reporting data in accordance with the present disclosure;
fig. 3 is a schematic diagram of an application scenario of a method for reporting data according to an embodiment of the present disclosure;
fig. 4 is a flow diagram of yet another embodiment of a method for reporting data in accordance with the present disclosure;
fig. 5 is a schematic structural diagram of an embodiment of an apparatus for reporting data according to the present disclosure;
fig. 6 is a timing diagram illustrating interactions among various devices in an embodiment for reporting data according to the present disclosure.
FIG. 7 is a schematic structural diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that, in the present disclosure, the embodiments and features of the embodiments may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates an example architecture 100 to which the disclosed method for reporting data or apparatus for reporting data may be applied.
As shown in fig. 1, system architecture 100 may include terminal devices 101, 102, 103, network 104, and cloud services 105. The cloud services 105 may be provided by the servers 1051, 1052, 1053. Network 104 is used to provide a medium of communication links between terminal devices 101, 102, 103 and cloud service 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The terminal devices 101, 102, 103 interact with the cloud service 105 through the network 104 to receive or transmit messages and the like. The terminal devices 101, 102, 103 may have various communication client applications installed thereon, such as a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, a text editing application reading application, and the like.
The terminal apparatuses 101, 102, and 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices having a display screen and supporting information transceiving, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, desktop computers, and the like. When the terminal apparatuses 101, 102, 103 are software, they can be installed in the electronic apparatuses listed above. It may be implemented as multiple pieces of software or software modules (e.g., software or software modules used to provide distributed services) or as a single piece of software or software module. And is not particularly limited herein.
The cloud service 105 may provide various services, for example, support for various client applications installed on the terminal devices 101, 102, 103. The server 1051 may serve as a server for resource management in the cloud service 105, and the servers 1052 and 1053 may serve as clients for resource management in the cloud service 105. The server 1051 can check and update the resource attribute information reported by the servers 1052 and 1053 in time, so as to ensure the accuracy of the resource data in the cloud service 105.
The server may be hardware or software. When the server is hardware, it may be implemented as a distributed server cluster formed by multiple servers, or may be implemented as a single server. When the server is software, it may be implemented as multiple pieces of software or software modules (e.g., software or software modules used to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.
It should be noted that, the method for reporting data provided by the embodiment of the present disclosure is generally executed by the server 1051, and accordingly, the apparatus for reporting data is generally disposed in the server 1051.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Continuing to refer to fig. 2, a flow 200 of one embodiment of a method for reporting data in accordance with the present disclosure is shown. The method for reporting data comprises the following steps:
In this embodiment, an execution subject (for example, the server 1051 shown in fig. 1) of the method for reporting data may pull a data message from a preset data message set as a target data message in a wired connection manner or a wireless connection manner. The preset data message set may be used to store data messages reported by at least one client. The data message may include data to be reported and data acquisition time. The data to be reported may be determined according to an actual application, for example, resource data managed by a resource management system. The data acquisition time may take various forms, such as a time of day or a time stamp.
In some optional implementations of this embodiment, the preset data message set may include a preset message queue. Therefore, the data messages reported by the client can enter the message queue according to the sequence of the data acquisition time.
In some optional implementations of this embodiment, the data message in the data message set may further include an identifier of the data source end. The identifier of the data source end can be used for uniquely identifying the client end reporting the data message. For example, the identifier of the source end of the data may be an sn (serial number) code or identifier.
In some optional implementation manners of this embodiment, in response to receiving a data message reported by a client that meets an active reporting condition, the execution main body may store the data message reported by the client that meets the active reporting condition into the preset data message set. The active reporting condition may include at least one of the following: and detecting that the data to be reported locally at the client side changes, wherein the time interval from the last data reporting reaches a preset reporting period. Therefore, the client can push the data message to the preset data message set according to the active reporting condition. Step 202, performing pre-check on the target data message to generate a pre-check result.
In this embodiment, the execution subject may perform pre-check on the data message pulled in step 201 in various ways, and generate a pre-check result. The pre-check result may be used to indicate whether the pre-check passes or not. Therefore, a part of data messages which do not meet the requirement of reported data are firstly filtered through the pre-check, the data volume processed by the server can be reduced, and the resource management efficiency is improved.
In some optional implementations of this embodiment, the executing entity may generate the pre-verification result according to the following steps:
firstly, matching the format of data to be reported in the target data message with a preset reported data format to generate a format matching result.
In these implementation manners, the execution main body may match the format of the data to be reported in the target data message pulled in step 201 with a preset reporting data format. The matching with the preset reporting data format may include, but is not limited to, at least one of the following: whether the number of the data items is consistent, whether the fields corresponding to the data items are missing, and whether the values of the data items are abnormal. Then, the execution body may generate a format matching result according to the matching. The format matching result can be used to represent whether the format of the data to be reported in the target data message meets the requirement of the preset reporting data format.
And secondly, generating a pre-check result according to the format matching result.
In some optional implementations of this embodiment, based on the identifier of the data source end included in the data message, the executing body may generate a format matching result according to the following steps:
the first step, according to the preset data source end check rule, the identification of the data source end in the target data message is checked, and a source end check result is generated.
In these implementations, according to a preset data source end verification rule, the execution subject may verify an identifier of the data source end in the target data message pulled in step 201. The preset data source end check rule may include various forms. As an example, the preset data source verification rule may include generating a source verification result indicating that verification fails in response to determining that the identifier of the data source in the target data message conforms to a certain type of format. As yet another example, in response to determining that the identification of the source of the data in the target data message is present in a preset white list, a source verification result is generated indicating that the verification passes.
And secondly, generating a pre-check result based on the source end check result.
In these implementations, the execution subject may generate the pre-verification result in various ways based on the source-side verification result generated in the first step. As an example, the execution subject may directly determine the source side check result as the preamble check result.
Optionally, based on the source side verification result, the execution subject may further generate a pre-verification result according to the following steps:
the first step, in response to the fact that the verification result of the source end indicates that verification is passed, data messages which are matched with the identification of the source end of the data in the target data message and have the data acquisition time closest to the current time are obtained from a preset database to serve as matched data messages.
And a second step of generating a pre-check result indicating that the pre-check passes in response to determining that the data acquisition time in the matching data message is earlier than the data acquisition time in the target data message.
In these implementations, the pre-check result is generated by comparing the data acquisition time in the target data message with the data acquisition time in the matching data message, which can ensure that the data is not falsely modified by earlier data.
And 203, in response to the fact that the pre-check result indicates that the pre-check is passed, performing data repeatability check on the data to be reported in the target data message according to the data acquisition time in the target data message to generate a data repeatability check result.
In this embodiment, in response to determining that the pre-check result indicates that the pre-check is passed, the execution main body may perform data repeatability check on the data to be reported in the target data message according to the data acquisition time in the target data message, and generate a data repeatability check result. The data repeatability check may include various methods for checking data integrity. The data repeatability check result can be used for indicating whether the target data message is consistent with the corresponding data message in the preset database. The preset database may be configured to store the historical data message reported by the at least one client. Typically, the preset database is persistent.
In this embodiment, the execution main body may perform data repeatability check on the data to be reported in the target data message and the reference data. The reference data may be data to be reported in a data message reported last time of data acquisition time in the target data message. In practice, the data message corresponding to the target data message and the reference data message is usually sent by the same client. The data repeatability check may include, but is not limited to, at least one of the following: parity, BCC (Block Check code) Check, md5(Message-Digest Algorithm).
In some optional implementations of this embodiment, the data repeatability check may further include a CRC check.
And step 204, in response to determining that the data repeatability check result indicates that the target data message is inconsistent with the corresponding data message in the preset database, storing the target data message in the preset database.
In this embodiment, in response to determining that the data repeatability check result indicates that the target data message is inconsistent with the corresponding data message in the preset database, the executing entity may store the target data message in the preset database.
In some optional implementation manners of this embodiment, the preset database may further include a CRC code corresponding to the data message. The CRC code may be used to perform the data duplication check.
In some optional implementation manners of this embodiment, the executing body may further compare the target data message with a theoretical calculation value. In response to determining that the data to be reported in the target data message is inconsistent with the data indicated by the theoretical calculation value, the execution main body may further send information prompting to correct the data. The theoretical calculation value is usually generated according to an algorithm preset by the resource management system.
Based on the optional implementation mode, the target data message uploaded by the client serves as an objective actual value, and a theoretical value generated by the resource management system can be corrected, so that the accuracy of the data of the whole resource management system is improved.
In some optional implementation manners of this embodiment, the executing body may further perform various subsequent processes on the data in the preset database.
With continued reference to fig. 3, fig. 3 is a schematic diagram of an application scenario of a method for reporting data according to an embodiment of the present disclosure. In the application scenario of fig. 3, a user 301 uses resources allocated by a cloud service 303 using a terminal 302. Then, the terminal 302 sends a resource usage query request 304 to the cloud service 303. The server 3031 of the cloud service 303 pulls the data message reported by the client 3032 allocated to the resource used by the terminal 302 from the preset data message set as the target data message 305. The preset data message set comprises data messages reported by the clients 3032 and 3033. The target data message 305 may include "200G 11:30: 08". Server 3031 may then perform a pre-check on target data message 305 to generate pre-check result 306 indicating that the check is passed. Then, the server 3031 may generate the data repeatability check result 307 according to the data message "180G 11:25: 08" reported last time by the client 3032 recorded in the database. The server 3031 may then store the target data message 305 in a predetermined database. Optionally, the server 3031 may also send the target data message 305 to the terminal 302.
At present, in one of the prior art, data is usually reported to a server by a server at a regular time or a client at a regular time, which causes poor data reporting purpose and causes repeated data uploading or system congestion. In the method provided by the embodiment of the disclosure, the client actively reports the data message to the preset data message set, and then the server uniformly pulls the data message set, so that the real-time performance of data updating is met, and system congestion caused by batch data reporting is prevented. Moreover, the preposed check and the data repeatability check are used as conditions for storing the reported data into the database, so that the data is efficiently reported, and network resources and computing resources of a server are effectively saved.
With further reference to fig. 4, a flow 400 of yet another embodiment of a method for reporting data is shown. The process 400 of the method for reporting data includes the following steps:
In this embodiment, an execution subject of the method for reporting data (for example, the server 1051 shown in fig. 1) may select a target client from the at least one client through various methods. The target client may be any client specified in advance according to actual application requirements. The target client may also be a client determined according to a rule, for example, data of a client missing from the server.
In some optional implementation manners of this embodiment, the execution main body may select, as the target client, a client whose time interval for reporting data is greater than a preset time interval from at least one client.
In this embodiment, in response to receiving the data message reported by the target client, the execution main body may directly store the data message reported by the target client into a preset data message set to receive a subsequent data verification process. The data message reported by the target client is generally generated based on data collected locally from the client in response to receiving the information characterizing the obtained reported data.
And step 405, performing pre-check on the target data message to generate a pre-check result.
And step 406, in response to the fact that the pre-check result indicates that the pre-check is passed, performing data repeatability check on the data to be reported in the target data message according to the data acquisition time in the target data message to generate a data repeatability check result.
As can be seen from fig. 4, a process 400 of the method for reporting data in this embodiment includes a step of sending information representing that reported data is obtained to a target client, and a step of storing a data message reported by the target client into a preset data message set in response to receiving the data message reported by the target client. Therefore, according to the scheme described in this embodiment, the server can actively acquire the reported data message from the target client, in addition to the active reporting mechanism of the client, so that the problem that the client data stored in the server cannot be updated is avoided under the condition that the active reporting mechanism of the client fails, and the stability of the reporting mechanism is ensured.
With further reference to fig. 5, as an implementation of the methods shown in the above diagrams, the present disclosure provides an embodiment of an apparatus for reporting data, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be specifically applied to various electronic devices.
As shown in fig. 5, the apparatus 500 for reporting data provided in this embodiment includes a pulling unit 501, a pre-parity unit 502, a duplicate parity unit 503, and a first storage unit 504. Wherein, the pulling unit 501 is configured to pull the data message from a preset data message set as the target data message. The preset data message set is used for storing data messages reported by at least one client. The data message comprises data to be reported and data acquisition time. And a pre-check unit 502 configured to perform pre-check on the target data message, and generate a pre-check result. And the repeated checking unit 503 is configured to perform data repeated checking on the data to be reported in the target data message according to the data acquisition time in the target data message in response to the fact that the pre-checking result indicates that the pre-checking is passed, and generate a data repeated checking result. A first storage unit 504 configured to store the target data message in the preset database in response to determining that the data repeatability check result indicates that the target data message is inconsistent with a corresponding data message in the preset database.
In this embodiment, the apparatus 500 for reporting data includes: the specific processing of the pull unit 501, the pre-check unit 502, the repeat check unit 503 and the first storage unit 504 and the technical effects thereof can refer to the related descriptions of step 201, step 202, step 203 and step 204 in the corresponding embodiment of fig. 2, which are not repeated herein.
In some optional implementations of this embodiment, the pre-check unit 502 may include a first generating module (not shown in the figure) and a second generating module (not shown in the figure). The first generating module may be configured to match a format of data to be reported in the target data message with a preset reported data format, and generate a format matching result. The second generation module may be configured to generate a pre-check result according to the format matching result.
In some optional implementations of this embodiment, the data message in the data message set may further include an identifier of the data source end. The pre-check unit 502 may include a third generating module (not shown), and a fourth generating module (not shown). The third generation module may be configured to verify, according to a preset data source end verification rule, an identifier of a data source end in the target data message, and generate a source end verification result. The fourth generation module may be configured to generate a pre-verification result based on the source-side verification result.
In some optional implementations of this embodiment, the fourth generating module may include an obtaining sub-module (not shown in the figure) and a generating sub-module (not shown in the figure). The obtaining sub-module may be configured to, in response to determining that the source end verification result indicates that the verification passes, obtain, from the preset database, a data message that matches the identifier of the source end of the data in the target data message and has a data acquisition time closest to the current time as a matching data message. The generating sub-module may be configured to generate a pre-check result indicating that the pre-check passes in response to determining that the data collection time in the matching data message is earlier than the data collection time in the target data message.
In some optional implementations of the embodiment, the apparatus 500 for reporting data may further include a selecting unit (not shown in the figure), a sending unit (not shown in the figure), and a second storing unit (not shown in the figure). Wherein, the selecting unit may be configured to select a target client from at least one client. The sending unit may be configured to send information characterizing the acquired reported data to the target client. The second storage unit may be configured to store the data message reported by the target client into a preset data message set in response to receiving the data message reported by the target client.
In some optional implementations of the embodiment, the selecting unit may be further configured to select, as the target client, a client whose time interval for reporting data is greater than a preset time interval from at least one client.
In some optional implementation manners of this embodiment, the apparatus 500 for reporting data may further include: and a third storage unit (not shown in the figure), configured to store, in response to receiving the data message reported by the client satisfying the active reporting condition, the data message reported by the client satisfying the active reporting condition into a preset data message set. Wherein the active reporting condition comprises at least one of the following: and detecting that the data to be reported locally at the client side changes, wherein the time interval from the last data reporting reaches a preset reporting period.
In some optional implementation manners of this embodiment, the preset data message set may include a preset message queue, the preset database may include a CRC code corresponding to the data message, and the data repeatability check may include a CRC check.
The apparatus provided by the above embodiment of the present disclosure pulls a data message from a preset data message set as a target data message through the pulling unit 501. The preset data message set is used for storing data messages reported by at least one client. The data message comprises data to be reported and data acquisition time. Then, the pre-check unit 502 performs pre-check on the target data message to generate a pre-check result. Then, the repetition check unit 503 performs data repetition check on the data to be reported in the target data message according to the data acquisition time in the target data message in response to the fact that the pre-check result indicates that the pre-check is passed, and generates a data repetition check result. Finally, the first storage unit 504 stores the target data message into the preset database in response to determining that the data repeatability check result indicates that the target data message is inconsistent with the corresponding data message in the preset database. Therefore, the efficient reporting of the data is realized, and the network resources and the computing resources of the server are effectively saved.
With further reference to fig. 6, a timing sequence 600 of interactions between various devices in one embodiment for reporting data is illustrated. The system for reporting data may include: a client (e.g., servers 1052, 1053 shown in fig. 1), and a server (e.g., server 1051 shown in fig. 1). The client may be configured to actively send the data message to the server in response to determining that the active data reporting condition is satisfied. The data message may include data to be reported and data acquisition time. The server side can be configured to respond to the received data message actively sent by the client side, and store the actively sent data message into a preset data message set; and configured to implement the method for reporting data as described in the foregoing embodiments.
In some optional implementation manners of this embodiment, the condition for actively reporting data may include detecting that data to be locally reported changes.
As shown in fig. 6, in step 601, in response to determining that the data active reporting condition is satisfied, the client actively sends a data message to the server.
In this embodiment, in response to determining that the data active reporting condition is satisfied, the client may actively send the data message to the server. The data message may include data to be reported and data acquisition time. The data active reporting condition may include that an interval duration between the last data reporting time and the previous data reporting time reaches a preset data reporting time interval. In practice, because the installation time of the clients in each batch is usually different, the clients in each batch are not reported at the same time when data reporting is performed according to the preset data reporting time interval. Therefore, the wrong-peak reporting of the data to be reported is realized, the system congestion is effectively avoided, and the utilization rate of network resources is improved.
In some optional implementation manners of this embodiment, the condition for actively reporting data may further include detecting that data to be locally reported changes. The change of the local data to be reported may include various situations. As an example, a virtual machine is installed or destroyed on a host machine, which may be a client.
In step 602, in response to receiving the data message actively sent by the client, the server stores the actively sent data message into a preset data message set.
In step 603, the server pulls the data message from the preset data message set as the target data message.
In step 604, the server performs pre-check on the target data message to generate a pre-check result.
In step 605, in response to determining that the pre-check result indicates that the pre-check is passed, the server performs data repeatability check on the data to be reported in the target data message according to the data acquisition time in the target data message, and generates a data repeatability check result.
In step 606, in response to determining that the data repeatability check result indicates that the target data message is inconsistent with the corresponding data message in the preset database, the server stores the target data message in the preset database.
Step 603, step 604, step 605 and step 606 are respectively consistent with step 201, step 202, step 203 and step 204 in the foregoing embodiment and their optional implementations, and the above description on step 201, step 202, step 203 and step 204 and their optional implementations also applies to step 603, step 604, step 605 and step 606, and is not described herein again.
In the system for reporting data provided in the foregoing embodiment of the present application, in response to determining that a data active reporting condition is satisfied, the client actively sends a data message to the server. And then, in response to the received data message actively sent by the client, the server stores the actively sent data message into a preset data message set. And then, the server side pulls the data message from the preset data message set as a target data message. And then, the server performs pre-check on the target data message to generate a pre-check result. And then, in response to the fact that the pre-check result indicates that the pre-check is passed, the server side conducts data repeatability check on the data to be reported in the target data message according to the data acquisition time in the target data message to generate a data repeatability check result. And finally, in response to the fact that the data repeatability check result indicates that the target data message is inconsistent with the corresponding data message in the preset database, the server stores the target data message in the preset database. Therefore, the data message is actively sent to the server side through the client side, the server side pulls data from the preset data message set, and the data message passing the verification is stored in the preset database after the data message is subjected to pre-verification and data repeatability verification. Therefore, the efficient reporting of the data is realized, and the network resources and the computing resources of the server are effectively saved.
Referring now to FIG. 7, a block diagram of an electronic device (e.g., server 1051 of FIG. 1)700 suitable for use in implementing embodiments of the present disclosure is shown. The server shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 7, electronic device 700 may include a processing means (e.g., central processing unit, graphics processor, etc.) 701 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from storage 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data necessary for the operation of the electronic apparatus 700 are also stored. The processing device 701, the ROM 702, and the RAM703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Generally, the following devices may be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touch pad, keyboard, mouse, etc.; an output device 707 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 708 including, for example, magnetic tape, hard disk, etc.; and a communication device 709. The communication means 709 may allow the electronic device 700 to communicate wirelessly or by wire with other devices to exchange data. While fig. 7 illustrates an electronic device 700 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 7 may represent one device or may represent multiple devices as desired.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via the communication means 709, or may be installed from the storage means 708, or may be installed from the ROM 702. The computer program, when executed by the processing device 701, performs the above-described functions defined in the methods of embodiments of the present disclosure.
It should be noted that the computer readable medium described in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In embodiments of the present disclosure, however, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the server; or may exist separately and not be assembled into the server. The computer readable medium carries one or more programs which, when executed by the server, cause the server to: the method comprises the steps that a data message is pulled from a preset data message set to serve as a target data message, wherein the preset data message set is used for storing the data message reported by at least one client, and the data message comprises data to be reported and data acquisition time; carrying out pre-check on the target data message to generate a pre-check result; in response to the fact that the pre-check result indicates that the pre-check is passed, performing data repeatability check on data to be reported in the target data message according to the data acquisition time in the target data message to generate a data repeatability check result; and storing the target data message into the preset database in response to determining that the data repeatability check result indicates that the target data message is inconsistent with the corresponding data message in the preset database.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor comprises a pulling unit, a preposed checking unit, a repeated checking unit and a first storage unit. Where the names of these elements do not in some cases constitute a limitation on the elements themselves, for example, a pull element may also be described as an "element that pulls a data message from a preset set of data messages as a target data message".
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.
Claims (12)
1. A method for reporting data, comprising:
the method comprises the steps that a data message is pulled from a preset data message set to serve as a target data message, wherein the preset data message set is used for storing the data message reported by at least one client, the data message comprises data to be reported and data acquisition time, and the data to be reported is used for recording various attributes of computing resources in a system;
performing pre-check on the target data message to generate a pre-check result;
in response to the fact that the pre-check result indicates that the pre-check is passed, performing data repeatability check on data to be reported in the target data message according to data acquisition time in the target data message to generate a data repeatability check result, wherein the data repeatability check result is used for indicating whether the target data message is consistent with corresponding data messages in a preset database, and the preset database is used for storing historical data messages reported by the at least one client;
and in response to determining that the data repeatability check result indicates that the target data message is inconsistent with the corresponding data message in a preset database, storing the target data message in the preset database.
2. The method of claim 1, wherein the pre-verifying the target data message, generating a pre-verification result, comprises:
matching the format of the data to be reported in the target data message with a preset reported data format to generate a format matching result;
and generating the pre-check result according to the format matching result.
3. The method of claim 1, wherein the data messages in the set of data messages further include an identification of a source of the data; and
the pre-verifying the target data message to generate a pre-verification result includes:
verifying the identifier of the data source end in the target data message according to a preset data source end verification rule to generate a source end verification result;
and generating the pre-check result based on the source side check result.
4. The method of claim 3, wherein said generating the pre-verification result based on the source-side verification result comprises:
in response to determining that the source end verification result indicates that verification is passed, acquiring a data message which is matched with the identification of the source end of the data in the target data message and has the closest data acquisition time to the current time from the preset database as a matched data message;
generating a pre-check result indicating that the pre-check passes in response to determining that the data acquisition time in the matching data message is earlier than the data acquisition time in the target data message.
5. The method of claim 1, wherein the method further comprises:
selecting a target client from the at least one client;
sending information representing the acquired reported data to the target client;
and responding to the received data message reported by the target client, and storing the data message reported by the target client into the preset data message set.
6. The method of claim 5, wherein the selecting a target client from the at least one client comprises:
and selecting the client with the reporting data time interval larger than the preset time interval from the at least one client as the target client.
7. The method of claim 1, wherein the method further comprises:
in response to receiving a data message reported by a client meeting an active reporting condition, storing the data message reported by the client meeting the active reporting condition into the preset data message set, wherein the active reporting condition comprises at least one of the following items: and detecting that the data to be reported locally at the client side changes, wherein the time interval from the last data reporting reaches a preset reporting period.
8. The method according to one of claims 1 to 7, wherein the predetermined set of data messages comprises a predetermined message queue, the predetermined database comprises Cyclic Redundancy Check (CRC) codes corresponding to the data messages, and the data repetition check comprises a CRC check.
9. An apparatus for reporting data, comprising:
the system comprises a pulling unit, a processing unit and a processing unit, wherein the pulling unit is configured to pull a data message from a preset data message set as a target data message, the preset data message set is used for storing the data message reported by at least one client, the data message comprises data to be reported and data acquisition time, and the data to be reported is used for recording various attributes of computing resources in the system;
the pre-check unit is configured to perform pre-check on the target data message and generate a pre-check result;
the repeated checking unit is configured to respond to the fact that the pre-checking result indicates that the pre-checking is passed, perform data repeated checking on data to be reported in the target data message according to data acquisition time in the target data message, and generate a data repeated checking result, wherein the data repeated checking result is used for indicating whether the target data message is consistent with corresponding data messages in a preset database, and the preset database is used for storing historical data messages reported by the at least one client;
a first storage unit configured to store the target data message in a preset database in response to determining that the data repeatability check result indicates that the target data message is inconsistent with a corresponding data message in the preset database.
10. A system for reporting data, comprising:
the client is configured to actively send a data message to the server in response to the fact that the data active reporting condition is met, wherein the data message comprises data to be reported and data acquisition time;
the server is configured to respond to the data message actively sent by the client and store the actively sent data message into a preset data message set; and to implement the method according to any one of claims 1-8.
11. A server, comprising:
one or more processors;
a storage device having one or more programs stored thereon;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.
12. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910998716.8A CN110737655B (en) | 2019-10-21 | 2019-10-21 | Method and device for reporting data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910998716.8A CN110737655B (en) | 2019-10-21 | 2019-10-21 | Method and device for reporting data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110737655A CN110737655A (en) | 2020-01-31 |
| CN110737655B true CN110737655B (en) | 2021-07-30 |
Family
ID=69270204
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910998716.8A Active CN110737655B (en) | 2019-10-21 | 2019-10-21 | Method and device for reporting data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737655B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111324687A (en) * | 2020-02-17 | 2020-06-23 | 平安科技（深圳）有限公司 | Data processing method and device in knowledge base, computer equipment and storage medium |
| CN113204574B (en) * | 2021-05-27 | 2024-01-26 | 北京深演智能科技股份有限公司 | Data processing method, device, computer readable storage medium and processor |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107122355A (en) * | 2016-02-24 | 2017-09-01 | 阿里巴巴集团控股有限公司 | Data mover system and method |
| CN107809434A (en) * | 2017-11-08 | 2018-03-16 | 济南大陆机电股份有限公司 | A kind of continuous data validation verification system and method |
| CN109714415A (en) * | 2018-12-26 | 2019-05-03 | 北京小米移动软件有限公司 | Data processing method and device |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150331875A1 (en) * | 2014-05-16 | 2015-11-19 | Syntel, Inc. | System and method for validating integrated data recasting objects |
| US20160306864A1 (en) * | 2015-04-14 | 2016-10-20 | Don Estes & Associates, Inc. | Method and system for data synchronization |
| CN105610836B (en) * | 2015-12-31 | 2019-01-15 | 浙江省公众信息产业有限公司 | A kind of method and system of data transmission |
| CN107943808A (en) * | 2016-10-13 | 2018-04-20 | 北京京东尚科信息技术有限公司 | The method and apparatus of processing equipment reported data |
| CN108270875B (en) * | 2018-02-24 | 2021-02-26 | 量化医学研究院（深圳）有限公司 | Data uploading method and front-end processor |
| CN109214855B (en) * | 2018-07-11 | 2020-09-18 | 腾讯科技（北京）有限公司 | Material receiving method, device and system |
| CN108593012A (en) * | 2018-07-19 | 2018-09-28 | 中国地质调查局水文地质环境地质调查中心 | The fast slowdown monitoring system of geological disaster |
-
2019
- 2019-10-21 CN CN201910998716.8A patent/CN110737655B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107122355A (en) * | 2016-02-24 | 2017-09-01 | 阿里巴巴集团控股有限公司 | Data mover system and method |
| CN107809434A (en) * | 2017-11-08 | 2018-03-16 | 济南大陆机电股份有限公司 | A kind of continuous data validation verification system and method |
| CN109714415A (en) * | 2018-12-26 | 2019-05-03 | 北京小米移动软件有限公司 | Data processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110737655A (en) | 2020-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019350B (en) | Data query method and device based on configuration information | |
| CN109241141B (en) | Deep learning training data processing method and device | |
| CN109492013B (en) | Data processing method, device and system applied to database cluster | |
| CN110262807B (en) | Cluster creation progress log acquisition system, method and device | |
| CN112765152B (en) | Method and apparatus for merging data tables | |
| CN109766127B (en) | Method for updating application version information | |
| CN110737655B (en) | Method and device for reporting data | |
| CN113505520A (en) | Method, device and system for supporting heterogeneous federated learning | |
| CN111324786A (en) | Method and device for processing consultation problem information | |
| CN107704357B (en) | Log generation method and device | |
| CN111338834A (en) | Data storage method and device | |
| CN109828830B (en) | Method and apparatus for managing containers | |
| CN109299124B (en) | Method and apparatus for updating a model | |
| CN109409419B (en) | Method and apparatus for processing data | |
| CN108011936B (en) | Method and device for pushing information | |
| CN110928594A (en) | Service development method and platform | |
| CN112860662A (en) | Data blood relationship establishing method and device, computer equipment and storage medium | |
| CN109413212B (en) | Method and apparatus for processing requests | |
| CN109597819B (en) | Method and apparatus for updating a database | |
| CN111324470A (en) | Method and device for generating information | |
| CN110110032B (en) | Method and device for updating index file | |
| CN112825525A (en) | Method and apparatus for processing transactions | |
| CN110730109A (en) | Method and apparatus for generating information | |
| CN111125163A (en) | Method and apparatus for processing data | |
| CN111581356B (en) | User behavior path analysis method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: Room 221, 2nd floor, Block C, 18 Kechuang 11th Street, Daxing Economic and Technological Development Zone, Beijing, 100176 Patentee after: Jingdong Technology Holding Co.,Ltd. Address before: Room 221, 2nd floor, Block C, 18 Kechuang 11th Street, Daxing Economic and Technological Development Zone, Beijing, 100176 Patentee before: Jingdong Digital Technology Holding Co.,Ltd. Address after: Room 221, 2nd floor, Block C, 18 Kechuang 11th Street, Daxing Economic and Technological Development Zone, Beijing, 100176 Patentee after: Jingdong Digital Technology Holding Co.,Ltd. Address before: Room 221, 2nd floor, Block C, 18 Kechuang 11th Street, Daxing Economic and Technological Development Zone, Beijing, 100176 Patentee before: JINGDONG DIGITAL TECHNOLOGY HOLDINGS Co.,Ltd. |