TW201828097A - Data processing method and apparatus capable of performing a real-time processing under the premise of ensuring the integrity of data statistics - Google Patents

Abstract

The invention discloses a data processing method. When receiving the to-be-processed data in the current system time window, it determines a system time window corresponding to the to-be-processed data according to the data time of the to-be-processed data. If the data time is within the range of the current system time window, it caches the to-be-processed data with a system receiving time earlier than the synchronization time point, and immediately processes the to-be-processed data with the system receiving time later than the synchronization time point. If the data time is within the range of the previous system time window, it immediately processes the to-be-processed data with the system receiving time earlier than the closing time point, and discards the to-be-processed data with the system receiving time later than the closing time point. Therefore, it is able to perform real-time processing under the premise of ensuring the integrity of data statistics thereby improving the accuracy and real-time performance of the data processing results.

Description

 Data processing method and device  

The present invention relates to the field of communication technologies, and in particular, to a data processing method, and to a data processing device.

As a new data processing method, streaming computing can calculate the dynamically generated data in real time and feed back the results in time. At present, streaming computing has been widely used in financial banking applications, Internet applications, and Internet of Things applications. It is mainly used to collect statistics on data within a certain time interval, that is, to collect statistics on data within a predetermined time window.

In the prior art, the time window in which the stream computing exists is divided into two types: "system time window" and "data time window", and their respective characteristics and disadvantages are as follows:

 (1) System time window  

The system time window counts the data of the data generation end based on the system time. Firstly, the statistical time of the data is divided into multiple system time windows according to the preset time interval, and then the system time window statistical result is output. If the data is processed only according to the system time window, the data has a certain delay from the generating end to the statistical end. The data processed in the system time window is often not equal to the generated data generated by the generating end in the preset time interval. Data, which makes the results of the stream calculations affected.

 (2) Data time window  

The data time window counts the data at the data generation end based on the data time. In the actual statistical process, due to the incomplete synchronization of the clocks of the devices at the data generating end and the speed of the data collection and transmission process, the data received by the statistical terminal is not necessarily strictly increased according to the order of the data time. Therefore, the problem of data disorder occurs in the statistical results based on the data time window. In the actual statistical process, it is easy to appear that the edge of the current data time window will have some data spanning into another data time window, so that the result of the stream calculation is affected.

Once the data is out of order, only the data that guarantees the current data time window will reach the statistics end, and the accuracy of the stream calculation can be guaranteed. Therefore, the current actual statistical process only closes the current data time window when the data received by the statistical terminal exceeds the data time window and reaches the preset value, or the data received by the statistical terminal reaches the preset data time window. .

In the process of implementing the present invention, the inventors have found that the prior art has at least the following problems:

(1) The data calculated by the stream is usually unstable. When the amount of data in a statistical item is small, the data window of the next data has not reached the statistical end. The statistical result of the current data time window is not output in time, resulting in data. Statistical delays have affected the results of the stream calculation.

(2) The same-level computing units of the stream computing are generally concurrently multiple. When the statistical results of the same-level computing unit are required to be output in sequence, due to the different processing progress of different computing units, the data of different computing units will appear. When the data time is not synchronized, the result of the stream calculation is affected.

It can be seen that the prior art can not output the statistical data in time when the real-time statistical data is carried out for the stream computing, and the statistical data cannot be synchronized globally, thereby affecting the accuracy of the data processing result. Sex.

The invention provides a data processing method, which solves the problem that both the statistical data integrity and the real-time statistical data processing cannot be balanced by setting the closing time point and the synchronization time point for each system time window of the data statistics system in advance. .

The method is applied to a data statistics system, wherein each system time window of the data statistics system is preset with a closing time point and a synchronization time point, the closing time point is before the synchronization time point, and is located in the After the start time point of the system time window, the method includes: when receiving the to-be-processed data in the current system time window, determining a system time window corresponding to the to-be-processed data according to the data time of the to-be-processed data; If the data time is within the current system time window, the system receives the data to be processed before the synchronization time point, and the system receiving time is to be processed after the synchronization time point. The data is processed immediately; if the data time is within the range of the previous system time window, the pending data of the system receiving time before the closing time point is processed immediately, and the system receiving time is The to-be-processed data after the closing time point is discarded.

Preferably, the system time window corresponding to the to-be-processed data is determined according to the data time of the to-be-processed data, specifically: acquiring the data of the to-be-processed data according to the time stamp of the data to be processed in the data time window Time; determining whether the data time is after the start time point of the current system time window; if yes, confirming that the to-be-processed data is within the current system time window; if not, confirming that the pending data is in the previous system Within the time window.

Preferably, the method further includes: when the system time of the data statistics system reaches the closing time point, outputting the processing result of the data to be processed whose data time is within the range of the previous system time window, And closing the data time window corresponding to the previous system time window.

Preferably, the method further comprises: when the system time of the data statistics system reaches the synchronization time point, processing the cached time before the synchronization time point and the data time is within the current system time window range The data to be processed and the processing result is output.

Preferably, the closing time point is set according to a time difference between the system time and the data time; the synchronization time point is based on a time consuming time of closing the time window and between the devices in the data statistics system Clock difference setting.

Correspondingly, the present invention also provides a data processing device, which is applied to a data statistics system, wherein each system time window of the data statistics system is preset with a closing time point and a synchronization time point, the closing time Pointing before the synchronization time point and after the start time point of the system time window, the device includes: a determining module, when the pending data is received in the current system time window, according to the to-be-processed The data time of the data determines a system time window corresponding to the to-be-processed data; the first processing module waits for the system receiving time to be before the synchronization time point within the current system time window within the data time Processing the data for cache, and processing the pending data of the system receiving time after the synchronization time point; the second processing module is within the range of the previous system time window Processing the pending data of the system receiving time before the closing time point, and processing the Receiving the system time data to be processed is discarded after the closing time point.

Preferably, the determining module is specifically configured to: obtain a data time of the to-be-processed data according to a time stamp of the data to be processed in the data time window; and determine whether the data time is in a current system time window. After the start time point; if yes, confirm that the to-be-processed data is within the current system time window; if not, confirm that the pending data is within the previous system time window.

Preferably, the device further includes: a first output module, when the system time of the data statistics system reaches the closing time point, the data to be processed within the range of the previous system time window The processing result is outputted, and the data time window corresponding to the previous system time window is closed.

Preferably, the device further includes: a second output module, when the system time of the data statistics system reaches the synchronization time point, processing the cached time before the synchronization time point and the data time is in the current system The data to be processed within the time window range and output the processing result.

Preferably, the closing time point is set according to a time difference between the system time and the data time; the synchronization time point is based on a time consuming time of closing the time window and between the devices in the data statistics system Clock difference setting.

It can be seen that, by applying the technical solution of the present invention, in the process of stream computing for realizing and completely outputting the calculation result, the method of setting the closing time point and the synchronization time point for each system time window of the data statistics system in advance is set. The global synchronization of statistical data can be processed in a timely manner on the premise of ensuring the integrity of the data, thereby improving the accuracy and real-time performance of the data processing results.

S101~S103‧‧‧Steps

301‧‧‧Determining modules

302‧‧‧First Processing Module

303‧‧‧Second processing module

1 is a schematic flowchart of a data processing method according to the present invention; FIG. 2 is a schematic flowchart of a data processing method according to a specific embodiment of the present invention; and FIG. 3 is a schematic structural diagram of a data processing device according to the present invention.

In view of the problems in the prior art, the present invention provides a data processing method for segmenting a processing time by using a closing time point and a synchronization time point in each system time window, and using the closing time point and the synchronization time point as nodes to process the data. Processing and output effectively improve the accuracy and real-time performance of data processing results.

The closing time point is the closing time of the data time window corresponding to the previous system time window, and also serves as the starting synchronization time of the data time window corresponding to the current system time window. The synchronization time point is the end synchronization time of the data time window corresponding to the current system time window. Therefore, the system time window is divided into different processing and output stages by using the closing time point and the synchronization time point as nodes, and the real-time performance of the processing result output can be ensured while ensuring the integrity of the data processing.

In a preferred embodiment of the present invention, the closing time point is set according to a time difference between the system time and the data time, and the synchronization time point is based on a time consumption of closing the time window and the data statistics system. The clock difference setting between each device.

For example, the data statistics system needs to process all the data generated by the corresponding device in the system time window of 0~1min, assuming that the timestamp of the corresponding data time window is all data of 10:04. In the actual statistics process, if the data with the timestamp of 10:04 may arrive at 1min10s of the system time window of 1~2min at the latest, set the 1min11s to the closing time point to ensure the corresponding time window of 0~1min. All pending data arrive at the data statistics system. If the sum of the time difference between the 0~1min system time window and the clock difference between the devices in the data statistics system is 2s, the 1min13s is set as the synchronization time point, and the received 1~2min system time window corresponds to Processing data is synchronized globally during this time period.

FIG. 1 is a schematic flowchart diagram of a data processing method according to the present invention. The method is applied to a data statistics system, wherein each system time window of the data statistics system is preset with a closing time point and a synchronization time point. Wherein the closing time point of the same system time window is before the synchronization time point and after the starting time point, the method comprising the steps of:

S101. When receiving the to-be-processed data in the current system time window, determine a system time window corresponding to the to-be-processed data according to the data time of the to-be-processed data.

In the actual application scenario, because the clock difference of the device at the data generating end and the speed of the data transmission process are affected, the data time of the data to be processed by the data statistics system may not be strictly increased according to the order of the data time. Therefore, it is easy for the pending data belonging to the previous system time window to cross the current system time window. That is to say, in the data to be processed received by the current system time window, there may be some pending data corresponding to the previous system time window. This will not only affect the integrity of the data of the previous system time window, but also interfere with the data processing of the current system time window.

For example, the data statistics system needs to process all the data generated by the corresponding device in the system time window of 0~1min, assuming that the timestamp of the corresponding data time window is all data of 10:04. In the actual statistical process, the data type with a time stamp of 10:04 may arrive in the system time window of 1~2min. This will not only affect the integrity of the data processing of the system time window of 0~1min, but also interfere with the data processing of the system time window of 1~2min.

Therefore, in order to prevent the result of the disorder of the data to be processed from affecting the calculation result, in the preferred embodiment of the present invention, when the data to be processed is received in the current system time window, it is determined according to the data time of the data to be processed. The system time window corresponding to the to-be-processed data is specifically determined as follows: a) obtaining the data time of the to-be-processed data according to the time stamp of the data to be processed in the data time window; b) determining the data time Whether it is after the start time point of the current system time window; c) if yes, confirm that the pending data is within the current system time window; d) If not, confirm that the pending data is in the previous system time window range within.

It should be noted that the timestamp of the above-mentioned data to be processed is only an example provided by the preferred embodiment of the present invention. On the basis of this, the data time of the data to be processed can also be represented by selecting other forms, so that the present invention is applicable to more. The field of application, these improvements are within the scope of the invention.

S102. If the data time is within the current system time window, the system receiving time is cached by the data to be processed before the synchronization time point, and the system receiving time is after the synchronization time point. The pending data is processed immediately.

Specifically, if the received data corresponding to the current system time window is received, the processed data is processed separately by using the synchronization time point as a demarcation point.

Before the synchronization time point, the data statistics system not only needs to process and output the data to be processed whose data time is within the range of the previous system time window, but also needs to synchronize the data to be processed within the current system time window within the current system time window. Considering the statistical and computational efficiency and preventing the confusion of the statistics and calculation process, only the pending data before the synchronization time point in the system reception time is cached.

After the synchronization time point, since the data time window corresponding to the previous system time window is closed, and the data to be processed within the current system time window range has also completed the global synchronization, the system receiving time is at the synchronization time. The pending data after the point is processed immediately.

S103. If the data time is within the range of the previous system time window, the pending data of the system receiving time before the closing time point is processed immediately, and the system receiving time is turned off. The pending data after the time point is discarded.

Specifically, if the received data corresponding to the previous system time window is received, the processed data is processed separately by using the closing time point as the demarcation point.

Before the closing time point, due to the clock difference of the device at the data generating end and the speed of the data transmission process, there may be some pending data corresponding to the previous system time window reaching the data statistics system in the current system time window. These materials play an important role in ensuring the integrity and accuracy of data processing in the previous system time window. Therefore, the to-be-processed data of the system receiving time before the closing time point is processed immediately.

After the closing time point, since the data time window corresponding to the previous system time window has been closed, the pending data arriving at the data statistics system at this moment will affect the global synchronization of the data to be processed within the current system time window. It may even cause confusion in the statistics and calculation process of the data to be processed within the current system time window. Therefore, the pending data of the system receiving time after the closing time point is discarded.

In a preferred embodiment of the present invention, the output node is the result of the processing by the closing time point and the synchronization time point, respectively.

Specifically, when the system time of the data statistics system reaches the closing time point, the processing result of the to-be-processed data whose data time is within the range of the previous system time window is output, and is closed with the previous A data time window corresponding to a system time window.

In a preferred embodiment of the present invention, when the system time of the data statistics system reaches the synchronization time point, the process that is cached before the synchronization time point and the data time is within the current system time window range is processed. Process the data and output the processing results.

It can be seen that, by applying the technical solution of the present invention, by setting the closing time point and the synchronization time point in advance for each system time window of the data statistics system, the closing time point and the synchronization time point are respectively used as nodes, and are closed. The time point and the synchronization time point are segmented processing and output for the node to process the data, and the global synchronization process of the data to be processed is also added, and the data can be processed in time under the premise of ensuring the completeness of the data statistics, thereby improving the data processing. The accuracy and real-time of the results.

In order to further illustrate the technical idea of the present invention, the technical solution of the present invention will be described in conjunction with the specific application scenario shown in FIG.

When the data calculated by the stream is counted, because there is a difference between the system time and the data time of the data, it is easy to see that some data at the edge of the current data time window crosses into another data time window, thereby The result of the stream calculation is affected. Since the difference between the system time and the data time of the data is often not very large, part of the data at the edge of the current data time window generally only spans to the adjacent data time window (ie, the next data time window), even if there is a pole When a small amount of data corresponding to the current data time window spans the non-adjacent data time window (ie, the data time window after the next data time window), the data that is skipped is discarded when the corresponding data time window data processing node is reached. Just fine. In the stream calculation, the possibility of the above situation is extremely low. In addition, even if the above situation occurs, the selection of a very small amount of data is discarded, and the statistical processing result of the current data time window data is not affected.

Therefore, in the specific application scenario, in the process of stream computing for real-time and complete output calculation results, a data processing method is proposed. The data time window is used as the time basis for data statistics, and the system time window is closed as the data time window. The basis for global synchronization.

The method pre-sets the closing time point and the synchronization time point in each system time window. The closing time point is set according to the time difference between the system time and the data time, and the synchronization time point is set according to the time consumption of the closing time window and the clock difference between the devices in the data statistics system.

Specifically, the application scenario takes 0-1 as the previous system time window, and 1-2 is the current system time window. The previous system time window presets the closing time point ct0 and the synchronization time point st0, and the current system time window presets the closing time point ct1 and the synchronization time point st1.

The method specifically includes the following steps:

Step 1 1~ct1, the data statistics system receives the data to be processed, and judges according to the data time of the data to be processed. If the data time is within the range of the current system time window 1-2, the data to be processed is fast. If the data time is within the range of 0-1 of the previous system time window, the data to be processed is processed and output immediately.

Specifically, the to-be-processed data received in this time period corresponds to the previous system time window 0-1 and the current system time window 1-2. If the data to be processed corresponds to the previous system time window 0-1, the data is processed and output immediately; if the data to be processed corresponds to the current system time window 1-2, only the data is cached and not processed.

Step 2: At time ct1, outputting the processing result of all the data to be processed within the range of the previous system time window 0-1, and closing the data time window corresponding to the previous system time window 0-1 .

Specifically, when the time ct1 is reached, the to-be-processed data corresponding to the previous system time window 0-1 is no longer cached.

Step 3 ct1~st1, receiving the data to be processed, and determining according to the data time of the data to be processed. If the data time is within the range of the current system time window 1-2, the data to be processed is cached, and The data to be processed is globally synchronized for the cache; if the data time is within the range of 0-1 of the previous system time window, the pending data is discarded.

Specifically, the to-be-processed data received in this time period corresponds to the previous system time window 0-1 and the current system time window 1-2. If the data to be processed corresponds to the previous system time window 0-1, the data to be processed is discarded; if the data to be processed corresponds to the current system time window 1-2, the data to be processed is cached for global synchronization without processing.

Step 4 st1~2, receiving the data to be processed and processing, and also outputting the above processing result in real time for 1~ct1 cache.

Specifically, the to-be-processed data received during this time period corresponds to the current system time window 1-2. The received data to be processed is processed along with the data to be processed within the range of the current system time window 1-2, together with the data time cached in the previous time period of 1~st1, and the processing result is output in real time. In the technical solution of the specific application scenario described above, in the process of calculating the streaming stream that requires real-time and complete outputting of the calculation result, the statistics are set by closing the time point and the synchronization time point in advance for each system time window of the data statistics system. The global synchronization of the data can be processed in a timely manner on the premise of ensuring the integrity of the data, thereby improving the accuracy and real-time performance of the data processing results.

It should be noted that the content of the above specific application scenario is only an example provided by the preferred embodiment of the present invention, and more application fields may be included on the basis of the above, so that the technical solution has a wider application. Improvements are within the scope of the invention.

In order to achieve the above technical purpose, the present invention also provides a data processing device. As shown in FIG. 3, the device is applied to a data statistics system, wherein each system time window of the data statistics system is preset with a closing time point. And a synchronization time point, wherein the closing time point of the same system time window is before the synchronization time point and after the starting time point. The device includes: a determining module, when receiving the to-be-processed data in the current system time window, determining a system time window corresponding to the to-be-processed data according to the data time of the to-be-processed data; the first processing module is The data time is within a range of the current system time window, the system receiving time is cached by the data to be processed before the synchronization time point, and the data to be processed after the system receiving time is after the synchronization time point Instant processing; the second processing module, in the range of the data time of the previous system time window, the processing data of the system receiving time before the closing time point is processed immediately, and The pending data of the system receiving time after the closing time point is discarded.

In a specific application scenario, the determining module is specifically configured to: obtain a data time of the to-be-processed data according to a time stamp of the data to be processed in the data time window; and determine whether the data time is in a current system. After the start time point of the time window; if yes, confirm that the to-be-processed data is within the current system time window; if not, confirm that the pending data is within the previous system time window.

In a specific application scenario, the device further includes: a first output module, when the system time of the data statistics system reaches the closing time point, the data time is within the range of the previous system time window The processing result of the to-be-processed data is output, and the data time window corresponding to the previous system time window is closed.

In a specific application scenario, the device further includes: a second output module, when the system time of the data statistics system reaches the synchronization time point, processing the data cached before the synchronization time point The data to be processed whose time is within the current system time window and outputs the processing result.

In a specific application scenario, the closing time point is set according to a time difference between the system time and the data time; the synchronization time point is based on time consumption of closing the time window and each of the data statistics systems The clock difference setting between devices.

Through the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be implemented by hardware, or by means of a software plus a necessary universal hardware platform. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a USB, a mobile hard disk, etc.), including several The instructions are for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various implementation scenarios of the present invention.

Those skilled in the art can understand that the drawings are only a schematic diagram of a preferred implementation scenario, and the modules or processes in the drawings are not necessarily required to implement the present invention.

A person skilled in the art may understand that the modules in the device in the implementation scenario may be distributed in the device that implements the scenario according to the implementation scenario description, or may be correspondingly changed in one or more devices different from the implementation scenario. The modules of the above implementation scenarios may be combined into one module, or may be further split into multiple sub-modules.

The above-mentioned serial numbers of the present invention are merely for description, and do not represent the advantages and disadvantages of the implementation scenario.

The above disclosure is only a few specific implementation scenarios of the present invention, but the present invention is not limited thereto, and any changes that can be made by those skilled in the art should fall within the protection scope of the present invention.

Claims (10)

 A data processing method is applied to a data statistics system, wherein a closing time point and a synchronization time point are set in advance for each system time window of the data statistics system, and the closing time point is before the synchronization time point and is located After the start time point of the system time window, the method includes: when receiving the to-be-processed data in the current system time window, determining a system time window corresponding to the to-be-processed data according to the data time of the to-be-processed data; The data time is within the current system time window, and the data to be processed of the system receiving time before the synchronization time point is cached, and the pending data of the system receiving time after the synchronization time point is processed immediately; The data time is within the range of the previous system time window, and the pending data of the system receiving time before the closing time point is processed immediately, and the pending data of the system receiving time after the closing time point is discarded. .    The method of claim 1, wherein the system time window corresponding to the to-be-processed data is determined according to the data time of the to-be-processed data, specifically: according to the time stamp of the data to be processed in the data time window, Obtaining the data time of the to-be-processed data; determining whether the data time is after the start time point of the current system time window; if yes, confirming that the to-be-processed data is within the current system time window; if not, confirming the pending The data is within the range of the previous system time window.    The method of claim 1, further comprising: when the system time of the data statistics system reaches the closing time point, the data to be processed within the range of the previous system time window The processing result is output, and the data time window corresponding to the previous system time window is closed.    The method of claim 1, wherein the method further comprises: when the system time of the data statistics system reaches the synchronization time point, processing the cached time before the synchronization time point and the data time is at the current system time. The data to be processed within the window range, and output the processing result.    The method of any one of claims 1-4, wherein the closing time point is set according to a time difference between the system time and the data time; the synchronization time point is based on the consumption of the closing time window And the clock difference setting between each device in the data statistics system.    A data processing device is applied to a data statistics system, wherein a closing time point and a synchronization time point are set in advance for each system time window of the data statistics system, and the closing time point is before the synchronization time point and is located After the start time point of the system time window, the device includes: a determining module, when receiving the to-be-processed data in the current system time window, determining a system time corresponding to the to-be-processed data according to the data time of the to-be-processed data a first processing module, in the data system time, within the current system time window, the system receives the data to be processed before the synchronization time point, and the system receiving time is at the synchronization time point. The data to be processed is processed immediately; the second processing module processes the data to be processed immediately before the closing time in the range of the previous system time window during the data time, and The pending data of the system receiving time after the closing time point is discarded.    The device of claim 6, wherein the determining module is configured to: obtain a data time of the to-be-processed data according to a time stamp of the data to be processed in the data time window; and determine whether the data time is After the start time point of the current system time window; if yes, confirm that the pending data is within the current system time window; if not, confirm that the pending data is within the previous system time window.    The device of claim 6, further comprising: a first output module, wherein the data time is in the range of the previous system time window when the system time of the data statistics system reaches the closing time point The processing result of the to-be-processed data is output, and the data time window corresponding to the previous system time window is closed.    The device of claim 6, further comprising: a second output module, when the system time of the data statistics system reaches the synchronization time point, processing the cached time before the synchronization time point and The data to be processed within the current system time window and the processing result is output.    The device of any one of claims 6-9, wherein the closing time point is set according to a time difference between the system time and the data time; the synchronization time point is based on the consumption of the closing time window And the clock difference setting between each device in the data statistics system.