TW201828097A - Data processing method and apparatus capable of performing a real-time processing under the premise of ensuring the integrity of data statistics - Google Patents
Data processing method and apparatus capable of performing a real-time processing under the premise of ensuring the integrity of data statistics Download PDFInfo
- Publication number
- TW201828097A TW201828097A TW106102697A TW106102697A TW201828097A TW 201828097 A TW201828097 A TW 201828097A TW 106102697 A TW106102697 A TW 106102697A TW 106102697 A TW106102697 A TW 106102697A TW 201828097 A TW201828097 A TW 201828097A
- Authority
- TW
- Taiwan
- Prior art keywords
- data
- time
- processed
- time window
- time point
- Prior art date
Links
Landscapes
- Synchronisation In Digital Transmission Systems (AREA)
Abstract
Description
本發明涉及通訊技術領域,特別涉及一種資料處理方法,本發明同時還涉及一種資料處理設備。 The present invention relates to the field of communication technologies, and in particular, to a data processing method, and to a data processing device.
作為一種新的資料處理方式,串流計算能夠對動態產生的資料進行實時計算並及時反饋結果。目前,串流計算已廣泛應用於金融銀行業應用、網際網路應用和物聯網應用等領域中,主要用於對一定時間間隔內的資料進行統計,即對於預定時間窗口內的資料進行統計。 As a new data processing method, streaming computing can calculate the dynamically generated data in real time and feed back the results in time. At present, streaming computing has been widely used in financial banking applications, Internet applications, and Internet of Things applications. It is mainly used to collect statistics on data within a certain time interval, that is, to collect statistics on data within a predetermined time window.
在現有技術中,串流計算所存在時間窗口分為“系統時間窗口”以及“資料時間窗口”這兩種,其各自的特點以及缺點如下: In the prior art, the time window in which the stream computing exists is divided into two types: "system time window" and "data time window", and their respective characteristics and disadvantages are as follows:
系統時間窗口基於系統時間對資料產生端的資料進行統計,首先根據預設的時間間隔將資料的統計時間劃分為多個系統時間窗口,然後輸出系統時間窗口統計結果。若僅按照系統時間窗口對資料進行處理的話,資料從產生端 到統計端是有一定延遲性,在系統時間窗口內處理的資料往往不等於實際過程中產生端在預設時間間隔內所產生的資料,這使得串流計算的結果會受到影響。 The system time window counts the data of the data generation end based on the system time. Firstly, the statistical time of the data is divided into multiple system time windows according to the preset time interval, and then the system time window statistical result is output. If the data is processed only according to the system time window, the data has a certain delay from the generating end to the statistical end. The data processed in the system time window is often not equal to the generated data generated by the generating end in the preset time interval. Data, which makes the results of the stream calculations affected.
資料時間窗口基於資料時間對資料產生端的資料進行統計。在實際統計過程中,受到資料產生端的設備的時鐘不完全同步以及資料採集傳輸過程的快慢程度的影響,統計端接收到資料的資料時間未必是按照資料時間的先後順序嚴格遞增的。因此基於資料時間窗口的統計結果中就會出現資料亂序的問題。在實際統計過程中很容易出現當前資料時間窗口的邊緣會有部分資料跨越到另一資料時間窗口中的情況,使得串流計算的結果受到影響。 The data time window counts the data at the data generation end based on the data time. In the actual statistical process, due to the incomplete synchronization of the clocks of the devices at the data generating end and the speed of the data collection and transmission process, the data received by the statistical terminal is not necessarily strictly increased according to the order of the data time. Therefore, the problem of data disorder occurs in the statistical results based on the data time window. In the actual statistical process, it is easy to appear that the edge of the current data time window will have some data spanning into another data time window, so that the result of the stream calculation is affected.
一旦出現在資料亂序的情況下,只有透過保證當前資料時間窗口的所有資料都到達統計端,才能夠保證串流計算的準確程度。因此現有的實際統計過程只有在統計端接收到資料的資料時間超過資料時間窗口並達到預設值時或者統計端接收到下一資料時間窗口的資料達到預設數量時,才關閉當前資料時間窗口。 Once the data is out of order, only the data that guarantees the current data time window will reach the statistics end, and the accuracy of the stream calculation can be guaranteed. Therefore, the current actual statistical process only closes the current data time window when the data received by the statistical terminal exceeds the data time window and reaches the preset value, or the data received by the statistical terminal reaches the preset data time window. .
在實現本發明的過程中,發明人發現現有技術至少存在如下問題: In the process of implementing the present invention, the inventors have found that the prior art has at least the following problems:
(1)串流計算的資料通常是不穩定的,當某個統計項資料量較少時,下個資料時間窗口資料一直沒有到達統計端,當前資料時間窗口的統計結果沒有及時輸出,導致 資料統計延誤,使得串流計算的結果受到影響。 (1) The data calculated by the stream is usually unstable. When the amount of data in a statistical item is small, the data window of the next data has not reached the statistical end. The statistical result of the current data time window is not output in time, resulting in data. Statistical delays have affected the results of the stream calculation.
(2)串流計算的同級計算單元一般都是併發多個的,當需要同級計算單元的統計結果按順序輸出時,由於不同計算單元的處理進度的不同,則會出現不同計算單元的資料的資料時間不同步的情況,使得串流計算的結果受到影響。 (2) The same-level computing units of the stream computing are generally concurrently multiple. When the statistical results of the same-level computing unit are required to be output in sequence, due to the different processing progress of different computing units, the data of different computing units will appear. When the data time is not synchronized, the result of the stream calculation is affected.
由此可見,現有技術在針對串流計算中進行實時統計資料時,無法在保證資料統計完整的前提下及時輸出統計資料,同時不能對統計資料的進行全域同步,從而影響了資料處理結果的準確性。 It can be seen that the prior art can not output the statistical data in time when the real-time statistical data is carried out for the stream computing, and the statistical data cannot be synchronized globally, thereby affecting the accuracy of the data processing result. Sex.
本發明提供了一種資料處理方法,透過預先為資料統計系統的每一段系統時間窗口設置關閉時間點以及同步時間點的方式,解決了保證統計資料完整和實時進行統計資料處理二者無法兼顧的問題。 The invention provides a data processing method, which solves the problem that both the statistical data integrity and the real-time statistical data processing cannot be balanced by setting the closing time point and the synchronization time point for each system time window of the data statistics system in advance. .
該方法應用於資料統計系統中,其中所述資料統計系統的每一段系統時間窗口均預先設置有關閉時間點以及同步時間點,所述關閉時間點在所述同步時間點之前,並位於所述系統時間窗口的起始時間點之後,所述方法包括:當在當前系統時間窗口接收到待處理資料時,根據所述待處理資料的資料時間確定與所述待處理資料對應的系統時間窗口;若所述資料時間在當前系統時間窗口範圍之內,將系 統接收時間在所述同步時間點之前的待處理資料進行快取,以及將所述系統接收時間在所述同步時間點之後的待處理資料即時進行處理;若所述資料時間在前一系統時間窗口的範圍之內,將所述系統接收時間在所述關閉時間點之前的待處理資料即時進行處理,以及將所述系統接收時間在所述關閉時間點之後的待處理資料丟棄。 The method is applied to a data statistics system, wherein each system time window of the data statistics system is preset with a closing time point and a synchronization time point, the closing time point is before the synchronization time point, and is located in the After the start time point of the system time window, the method includes: when receiving the to-be-processed data in the current system time window, determining a system time window corresponding to the to-be-processed data according to the data time of the to-be-processed data; If the data time is within the current system time window, the system receives the data to be processed before the synchronization time point, and the system receiving time is to be processed after the synchronization time point. The data is processed immediately; if the data time is within the range of the previous system time window, the pending data of the system receiving time before the closing time point is processed immediately, and the system receiving time is The to-be-processed data after the closing time point is discarded.
優選地,根據所述待處理資料的資料時間確定與所述待處理資料對應的系統時間窗口,具體為:根據所述待處理資料在資料時間窗口的時間戳記,獲取所述待處理資料的資料時間;判斷所述資料時間是否在當前的系統時間窗口的起始時間點之後;若是,確認所述待處理資料在當前系統時間窗口範圍之內;若否,確認所述待處理資料在之前系統時間窗口範圍之內。 Preferably, the system time window corresponding to the to-be-processed data is determined according to the data time of the to-be-processed data, specifically: acquiring the data of the to-be-processed data according to the time stamp of the data to be processed in the data time window Time; determining whether the data time is after the start time point of the current system time window; if yes, confirming that the to-be-processed data is within the current system time window; if not, confirming that the pending data is in the previous system Within the time window.
優選地,所述方法還包括:當所述資料統計系統的系統時間到達所述關閉時間點時,將資料時間在所述前一系統時間窗口範圍之內的待處理資料的處理結果進行輸出,並關閉與所述前一系統時間窗口對應的資料時間窗口。 Preferably, the method further includes: when the system time of the data statistics system reaches the closing time point, outputting the processing result of the data to be processed whose data time is within the range of the previous system time window, And closing the data time window corresponding to the previous system time window.
優選地,所述方法還包括:當所述資料統計系統的系統時間到達所述同步時間點 時,處理在所述同步時間點之前所快取的且資料時間在當前系統時間窗口範圍之內的待處理資料,並輸出處理結果。 Preferably, the method further comprises: when the system time of the data statistics system reaches the synchronization time point, processing the cached time before the synchronization time point and the data time is within the current system time window range The data to be processed and the processing result is output.
優選地,所述關閉時間點根據所述系統時間與所述資料時間之間的時間差值設置;所述同步時間點根據關閉時間窗口的耗時以及所述資料統計系統中各設備之間的時鐘差值設置。 Preferably, the closing time point is set according to a time difference between the system time and the data time; the synchronization time point is based on a time consuming time of closing the time window and between the devices in the data statistics system Clock difference setting.
相應地,本發明還提出了一種資料處理設備,該設備應用於資料統計系統中,其中所述資料統計系統的每一段系統時間窗口均預先設置有關閉時間點以及同步時間點,所述關閉時間點在所述同步時間點之前,並位於所述系統時間窗口的起始時間點之後,所述設備包括:確定模組,當在當前系統時間窗口接收到待處理資料時,根據所述待處理資料的資料時間確定與所述待處理資料對應的系統時間窗口;第一處理模組,在所述資料時間在當前系統時間窗口範圍之內,將系統接收時間在所述同步時間點之前的待處理資料進行快取,以及將所述系統接收時間在所述同步時間點之後的待處理資料即時進行處理;第二處理模組,在所述資料時間在前一系統時間窗口的範圍之內,將所述系統接收時間在所述關閉時間點之前的待處理資料即時進行處理,以及將所述系統接收時間在所述關閉時間點之後的待處理資料丟棄。 Correspondingly, the present invention also provides a data processing device, which is applied to a data statistics system, wherein each system time window of the data statistics system is preset with a closing time point and a synchronization time point, the closing time Pointing before the synchronization time point and after the start time point of the system time window, the device includes: a determining module, when the pending data is received in the current system time window, according to the to-be-processed The data time of the data determines a system time window corresponding to the to-be-processed data; the first processing module waits for the system receiving time to be before the synchronization time point within the current system time window within the data time Processing the data for cache, and processing the pending data of the system receiving time after the synchronization time point; the second processing module is within the range of the previous system time window Processing the pending data of the system receiving time before the closing time point, and processing the Receiving the system time data to be processed is discarded after the closing time point.
優選地,所述確定模組具體用於: 根據所述待處理資料在資料時間窗口的時間戳記,獲取所述待處理資料的資料時間;判斷所述資料時間是否在當前的系統時間窗口的起始時間點之後;若是,確認所述待處理資料在當前系統時間窗口範圍之內;若否,確認所述待處理資料在之前系統時間窗口範圍之內。 Preferably, the determining module is specifically configured to: obtain a data time of the to-be-processed data according to a time stamp of the data to be processed in the data time window; and determine whether the data time is in a current system time window. After the start time point; if yes, confirm that the to-be-processed data is within the current system time window; if not, confirm that the pending data is within the previous system time window.
優選地,所述設備還包括:第一輸出模組,在所述資料統計系統的系統時間到達所述關閉時間點時,將資料時間在所述前一系統時間窗口範圍之內的待處理資料的處理結果進行輸出,並關閉與所述前一系統時間窗口對應的資料時間窗口。 Preferably, the device further includes: a first output module, when the system time of the data statistics system reaches the closing time point, the data to be processed within the range of the previous system time window The processing result is outputted, and the data time window corresponding to the previous system time window is closed.
優選地,所述設備還包括:第二輸出模組,在所述資料統計系統的系統時間到達所述同步時間點時,處理在所述同步時間點之前所快取的且資料時間在當前系統時間窗口範圍之內的待處理資料,並輸出處理結果。 Preferably, the device further includes: a second output module, when the system time of the data statistics system reaches the synchronization time point, processing the cached time before the synchronization time point and the data time is in the current system The data to be processed within the time window range and output the processing result.
優選地,所述關閉時間點根據所述系統時間與所述資料時間之間的時間差值設置;所述同步時間點根據關閉時間窗口的耗時以及所述資料統計系統中各設備之間的時鐘差值設置。 Preferably, the closing time point is set according to a time difference between the system time and the data time; the synchronization time point is based on a time consuming time of closing the time window and between the devices in the data statistics system Clock difference setting.
由此可見,透過應用本發明的技術方案,在針對需要實時並完整輸出計算結果的串流計算過程中,透過預先為 資料統計系統的每一段系統時間窗口設置關閉時間點以及同步時間點的方式,對統計資料的進行全域同步,可以在保證資料統計完整的前提下對其及時進行處理,從而提高了資料處理結果的準確性和實時性。 It can be seen that, by applying the technical solution of the present invention, in the process of stream computing for realizing and completely outputting the calculation result, the method of setting the closing time point and the synchronization time point for each system time window of the data statistics system in advance is set. The global synchronization of statistical data can be processed in a timely manner on the premise of ensuring the integrity of the data, thereby improving the accuracy and real-time performance of the data processing results.
S101~S103‧‧‧步驟 S101~S103‧‧‧Steps
301‧‧‧確定模組 301‧‧‧Determining modules
302‧‧‧第一處理模組 302‧‧‧First Processing Module
303‧‧‧第二處理模組 303‧‧‧Second processing module
圖1為本發明提出的一種資料處理方法的流程示意圖;圖2為本發明的具體實施例所提出的一種資料處理方法的流程示意圖;圖3為本發明提出的一種資料處理設備的結構示意圖。 1 is a schematic flowchart of a data processing method according to the present invention; FIG. 2 is a schematic flowchart of a data processing method according to a specific embodiment of the present invention; and FIG. 3 is a schematic structural diagram of a data processing device according to the present invention.
有鑒於現有技術中的問題,本發明提供了一種資料處理方法,透過在各個系統時間窗口預設關閉時間點以及同步時間點,並以關閉時間點和同步時間點為節點對待處理資料進行分段處理與輸出,有效提高了資料處理結果的準確性和實時性。 In view of the problems in the prior art, the present invention provides a data processing method for segmenting a processing time by using a closing time point and a synchronization time point in each system time window, and using the closing time point and the synchronization time point as nodes to process the data. Processing and output effectively improve the accuracy and real-time performance of data processing results.
其中,關閉時間點為所述前一系統時間窗口對應的資料時間窗口的關閉時刻,其同時也作為當前系統時間窗口對應的資料時間窗口的開始同步時刻。同步時間點為當前系統時間窗口對應的資料時間窗口的結束同步時刻。故以關閉時間點和同步時間點作為節點,將系統時間窗口劃分 為不同的處理與輸出階段,可以在保證資料處理的完整性的同時兼顧處理結果輸出的實時性。 The closing time point is the closing time of the data time window corresponding to the previous system time window, and also serves as the starting synchronization time of the data time window corresponding to the current system time window. The synchronization time point is the end synchronization time of the data time window corresponding to the current system time window. Therefore, the system time window is divided into different processing and output stages by using the closing time point and the synchronization time point as nodes, and the real-time performance of the processing result output can be ensured while ensuring the integrity of the data processing.
本發明的優選實施例中,所述關閉時間點根據所述系統時間與所述資料時間之間的時間差值設置,所述同步時間點根據關閉時間窗口的耗時以及所述資料統計系統中各設備之間的時鐘差值設置。 In a preferred embodiment of the present invention, the closing time point is set according to a time difference between the system time and the data time, and the synchronization time point is based on a time consumption of closing the time window and the data statistics system. The clock difference setting between each device.
舉例來說,資料統計系統需要處理0~1min系統時間窗口內由對應設備生成的所有資料,假設其對應資料時間窗口的時間戳記為10:04的所有資料。在實際統計過程中,如果時間戳記為10:04的資料最晚可能在1~2min系統時間窗口的1min10s時到達,則將1min11s處設為關閉時間點,保證0~1min系統時間窗口所對應的所有待處理資料全部到達資料統計系統。如果0~1min系統時間窗口的耗時與資料統計系統中各設備之間時鐘差值之和為2s,則將1min13s處設為同步時間點,接收到的1~2min系統時間窗口所對應的待處理資料在此時間段內全域同步。 For example, the data statistics system needs to process all the data generated by the corresponding device in the system time window of 0~1min, assuming that the timestamp of the corresponding data time window is all data of 10:04. In the actual statistics process, if the data with the timestamp of 10:04 may arrive at 1min10s of the system time window of 1~2min at the latest, set the 1min11s to the closing time point to ensure the corresponding time window of 0~1min. All pending data arrive at the data statistics system. If the sum of the time difference between the 0~1min system time window and the clock difference between the devices in the data statistics system is 2s, the 1min13s is set as the synchronization time point, and the received 1~2min system time window corresponds to Processing data is synchronized globally during this time period.
如圖1所示,為本發明提出的資料處理方法的流程示意圖,該方法應用於資料統計系統中,其中所述資料統計系統的每一段系統時間窗口均預先設置有關閉時間點以及同步時間點,其中,同一系統時間窗口的所述關閉時間點是位於所述同步時間點之前以及起始時間點之後,該方法包括以下步驟: FIG. 1 is a schematic flowchart diagram of a data processing method according to the present invention. The method is applied to a data statistics system, wherein each system time window of the data statistics system is preset with a closing time point and a synchronization time point. Wherein the closing time point of the same system time window is before the synchronization time point and after the starting time point, the method comprising the steps of:
S101,當在當前系統時間窗口接收到待處理資料時, 根據所述待處理資料的資料時間確定與所述待處理資料對應的系統時間窗口。 S101. When receiving the to-be-processed data in the current system time window, determine a system time window corresponding to the to-be-processed data according to the data time of the to-be-processed data.
在實際應用場景中,由於會受到資料產生端的設備的時鐘差值以及資料傳輸過程快慢的影響,所述資料統計系統所接收到不同待處理資料的資料時間未必是嚴格按照資料時間的先後順序遞增的,因此就會容易出現屬於前一系統時間窗口的待處理資料跨越到當前系統時間窗口的情況。也就是說,當前系統時間窗口所接收到待處理資料中,可能會存在部分對應於前一系統時間窗口的待處理資料。這樣不僅會影響到前一系統時間窗口資料的完整性,同時還會對當前系統時間窗口的資料處理造成干擾。 In the actual application scenario, because the clock difference of the device at the data generating end and the speed of the data transmission process are affected, the data time of the data to be processed by the data statistics system may not be strictly increased according to the order of the data time. Therefore, it is easy for the pending data belonging to the previous system time window to cross the current system time window. That is to say, in the data to be processed received by the current system time window, there may be some pending data corresponding to the previous system time window. This will not only affect the integrity of the data of the previous system time window, but also interfere with the data processing of the current system time window.
舉例來說,資料統計系統需要處理0~1min系統時間窗口內由對應設備生成的所有資料,假設其對應資料時間窗口的時間戳記為10:04的所有資料。在實際統計過程中,時間戳記為10:04的資料種可能會存在部分在1~2min系統時間窗口才到達。這樣不僅會影響到0~1min系統時間窗口資料處理的完整性,同時還會對1~2min系統時間窗口的資料處理造成干擾。 For example, the data statistics system needs to process all the data generated by the corresponding device in the system time window of 0~1min, assuming that the timestamp of the corresponding data time window is all data of 10:04. In the actual statistical process, the data type with a time stamp of 10:04 may arrive in the system time window of 1~2min. This will not only affect the integrity of the data processing of the system time window of 0~1min, but also interfere with the data processing of the system time window of 1~2min.
因此,為了防止由於待處理資料的亂序對計算結果造成影響,在本發明的優選實施例中,當在當前系統時間窗口接收到待處理資料時,需要根據所述待處理資料的資料時間確定與所述待處理資料對應的系統時間窗口,具體確定過程如下:a)根據所述待處理資料在資料時間窗口的時間戳 記,獲取所述待處理資料的資料時間;b)判斷所述資料時間是否在當前的系統時間窗口的起始時間點之後;c)若是,確認所述待處理資料在當前系統時間窗口範圍之內;d)若否,確認所述待處理資料在之前系統時間窗口範圍之內。 Therefore, in order to prevent the result of the disorder of the data to be processed from affecting the calculation result, in the preferred embodiment of the present invention, when the data to be processed is received in the current system time window, it is determined according to the data time of the data to be processed. The system time window corresponding to the to-be-processed data is specifically determined as follows: a) obtaining the data time of the to-be-processed data according to the time stamp of the data to be processed in the data time window; b) determining the data time Whether it is after the start time point of the current system time window; c) if yes, confirm that the pending data is within the current system time window; d) If not, confirm that the pending data is in the previous system time window range within.
需要說明的是,以上待處理資料的時間戳記僅為本發明優選實施例提出的示例,在此基礎上還可以透過選擇其他形式來表示待處理資料的資料時間,以使本發明適用於更多的應用領域,這些改進都屬於本發明的保護範圍。 It should be noted that the timestamp of the above-mentioned data to be processed is only an example provided by the preferred embodiment of the present invention. On the basis of this, the data time of the data to be processed can also be represented by selecting other forms, so that the present invention is applicable to more. The field of application, these improvements are within the scope of the invention.
S102,若所述資料時間在當前系統時間窗口範圍之內,將系統接收時間在所述同步時間點之前的待處理資料進行快取,以及將所述系統接收時間在所述同步時間點之後的待處理資料即時進行處理。 S102. If the data time is within the current system time window, the system receiving time is cached by the data to be processed before the synchronization time point, and the system receiving time is after the synchronization time point. The pending data is processed immediately.
具體的,接收到的如果是當前系統時間窗口所對應的待處理資料,以同步時間點為分界點對待處理資料進行分別處理。 Specifically, if the received data corresponding to the current system time window is received, the processed data is processed separately by using the synchronization time point as a demarcation point.
在同步時間點之前,資料統計系統不僅需要處理並輸出資料時間在前一系統時間窗口範圍之內的待處理資料,還需要對資料時間在當前系統時間窗口範圍之內的待處理資料進行全域同步,考慮到統計與計算效率以及防止統計與計算過程的混亂,故只將在系統接收時間在所述同步時間點之前的待處理資料進行快取。 Before the synchronization time point, the data statistics system not only needs to process and output the data to be processed whose data time is within the range of the previous system time window, but also needs to synchronize the data to be processed within the current system time window within the current system time window. Considering the statistical and computational efficiency and preventing the confusion of the statistics and calculation process, only the pending data before the synchronization time point in the system reception time is cached.
在同步時間點之後,由於前一系統時間窗口對應的資料時間窗口已關閉,且當前系統時間窗口範圍之內的待處理資料也已完成全域同步,故將所述系統接收時間在所述同步時間點之後的待處理資料即時進行處理。 After the synchronization time point, since the data time window corresponding to the previous system time window is closed, and the data to be processed within the current system time window range has also completed the global synchronization, the system receiving time is at the synchronization time. The pending data after the point is processed immediately.
S103,若所述資料時間在前一系統時間窗口的範圍之內,將所述系統接收時間在所述關閉時間點之前的待處理資料即時進行處理,以及將所述系統接收時間在所述關閉時間點之後的待處理資料丟棄。 S103. If the data time is within the range of the previous system time window, the pending data of the system receiving time before the closing time point is processed immediately, and the system receiving time is turned off. The pending data after the time point is discarded.
具體的,接收到的如果是前一系統時間窗口所對應的待處理資料,以關閉時間點為分界點對待處理資料進行分別處理。 Specifically, if the received data corresponding to the previous system time window is received, the processed data is processed separately by using the closing time point as the demarcation point.
在關閉時間點之前,由於會受到資料產生端的設備的時鐘差值以及資料傳輸過程快慢的影響,可能會存在部分對應於前一系統時間窗口的待處理資料在當前系統時間窗口才到達資料統計系統,這些資料對保證前一系統時間窗口資料處理的完整性與準確性起著重要的作用。故將所述系統接收時間在所述關閉時間點之前的待處理資料即時進行處理。 Before the closing time point, due to the clock difference of the device at the data generating end and the speed of the data transmission process, there may be some pending data corresponding to the previous system time window reaching the data statistics system in the current system time window. These materials play an important role in ensuring the integrity and accuracy of data processing in the previous system time window. Therefore, the to-be-processed data of the system receiving time before the closing time point is processed immediately.
在關閉時間點之後,由於前一系統時間窗口對應的資料時間窗口已關閉,此刻到達資料統計系統的待處理資料則會影響到資料時間在當前系統時間窗口範圍之內的待處理資料的全域同步,甚至還會造成資料時間在當前系統時間窗口範圍之內的待處理資料統計與計算過程的混亂。故將所述系統接收時間在所述關閉時間點之後的待處理資料 丟棄。 After the closing time point, since the data time window corresponding to the previous system time window has been closed, the pending data arriving at the data statistics system at this moment will affect the global synchronization of the data to be processed within the current system time window. It may even cause confusion in the statistics and calculation process of the data to be processed within the current system time window. Therefore, the pending data of the system receiving time after the closing time point is discarded.
本發明的優選實施例中,分別以關閉時間點和同步時間點作為處理結果的輸出節點。 In a preferred embodiment of the present invention, the output node is the result of the processing by the closing time point and the synchronization time point, respectively.
具體的,當所述資料統計系統的系統時間到達所述關閉時間點時,將資料時間在所述前一系統時間窗口範圍之內的待處理資料的處理結果進行輸出,並關閉與所述前一系統時間窗口對應的資料時間窗口。 Specifically, when the system time of the data statistics system reaches the closing time point, the processing result of the to-be-processed data whose data time is within the range of the previous system time window is output, and is closed with the previous A data time window corresponding to a system time window.
本發明的優選實施例中,當所述資料統計系統的系統時間到達所述同步時間點時,處理在所述同步時間點之前所快取的且資料時間在當前系統時間窗口範圍之內的待處理資料,並輸出處理結果。 In a preferred embodiment of the present invention, when the system time of the data statistics system reaches the synchronization time point, the process that is cached before the synchronization time point and the data time is within the current system time window range is processed. Process the data and output the processing results.
由此可見,透過應用本發明的技術方案,透過預先為資料統計系統的每一段系統時間窗口設置關閉時間點以及同步時間點的方式,分別以關閉時間點和同步時間點為節點,並以關閉時間點和同步時間點為節點對待處理資料進行分段處理與輸出,同時還增加了待處理資料的全域同步過程,可以在保證資料統計完整的前提下對其及時進行處理,從而提高了資料處理結果的準確性和實時性。 It can be seen that, by applying the technical solution of the present invention, by setting the closing time point and the synchronization time point in advance for each system time window of the data statistics system, the closing time point and the synchronization time point are respectively used as nodes, and are closed. The time point and the synchronization time point are segmented processing and output for the node to process the data, and the global synchronization process of the data to be processed is also added, and the data can be processed in time under the premise of ensuring the completeness of the data statistics, thereby improving the data processing. The accuracy and real-time of the results.
為了進一步闡述本發明的技術思想,現結合圖2所示的具體的應用場景,對本發明的技術方案進行說明。 In order to further illustrate the technical idea of the present invention, the technical solution of the present invention will be described in conjunction with the specific application scenario shown in FIG.
在對串流計算的資料進行統計時,由於資料的系統時間和資料時間之間存在差異,故很容易出現當前資料時間窗口的邊緣會有部分資料跨越到另一資料時間窗口中的情況,從而使得串流計算的結果受到影響。由於資料的系統 時間和資料時間之間的差異往往不是很大,故當前資料時間窗口的邊緣的部分資料一般僅跨越到相鄰的資料時間窗口(即下一資料時間窗口)中,即使出現極少量當前資料時間窗口對應的資料跨越到非相鄰的資料時間窗口時(即下一資料時間窗口之後的資料時間窗口),則到對應的資料時間窗口資料處理節點時,將該跨越的資料丟棄即可。在串流計算中,出現上述情況的可能性極低,再者即使出現上述情況,選擇將極少量的資料丟棄,也不會影響到對當前資料時間窗口資料的統計處理結果。 When the data calculated by the stream is counted, because there is a difference between the system time and the data time of the data, it is easy to see that some data at the edge of the current data time window crosses into another data time window, thereby The result of the stream calculation is affected. Since the difference between the system time and the data time of the data is often not very large, part of the data at the edge of the current data time window generally only spans to the adjacent data time window (ie, the next data time window), even if there is a pole When a small amount of data corresponding to the current data time window spans the non-adjacent data time window (ie, the data time window after the next data time window), the data that is skipped is discarded when the corresponding data time window data processing node is reached. Just fine. In the stream calculation, the possibility of the above situation is extremely low. In addition, even if the above situation occurs, the selection of a very small amount of data is discarded, and the statistical processing result of the current data time window data is not affected.
故此具體的應用場景中,在針對需要實時並完整輸出計算結果的串流計算過程中,提出了一種資料處理方法,以資料時間窗口作為資料統計的時間依據,系統時間窗口作為資料時間窗口關閉和全域同步的依據。 Therefore, in the specific application scenario, in the process of stream computing for real-time and complete output calculation results, a data processing method is proposed. The data time window is used as the time basis for data statistics, and the system time window is closed as the data time window. The basis for global synchronization.
該方法透過在每一段系統時間窗口均預先設置關閉時間點以及同步時間點。其中關閉時間點根據系統時間與所述資料時間之間的時間差值設置,同步時間點根據關閉時間窗口的耗時以及資料統計系統中各設備之間的時鐘差值設置。 The method pre-sets the closing time point and the synchronization time point in each system time window. The closing time point is set according to the time difference between the system time and the data time, and the synchronization time point is set according to the time consumption of the closing time window and the clock difference between the devices in the data statistics system.
具體的,本應用場景以0-1為上一系統時間窗口,1-2為當前系統時間窗口。其中,上一系統時間窗口預先設置關閉時間點ct0和同步時間點st0,當前系統時間窗口預先設置關閉時間點ct1和同步時間點st1。 Specifically, the application scenario takes 0-1 as the previous system time window, and 1-2 is the current system time window. The previous system time window presets the closing time point ct0 and the synchronization time point st0, and the current system time window presets the closing time point ct1 and the synchronization time point st1.
該方法具體包括如下步驟: The method specifically includes the following steps:
步驟1 1~ct1,資料統計系統接收待處理資料,根 據所述待處理資料的資料時間進行判斷,若所述資料時間在當前系統時間窗口1-2的範圍之內,將待處理資料進行快取;若所述資料時間在前一系統時間窗口0-1的範圍之內,將待處理資料進行即時處理並輸出。 Step 1 1~ct1, the data statistics system receives the data to be processed, and judges according to the data time of the data to be processed. If the data time is within the range of the current system time window 1-2, the data to be processed is fast. If the data time is within the range of 0-1 of the previous system time window, the data to be processed is processed and output immediately.
具體地,此時間段內接收的待處理資料,既有對應前一系統時間窗口0-1的,也有對應當前系統時間窗口1-2的。若待處理資料對應前一系統時間窗口0-1,則對資料進行即時處理並輸出;若待處理資料對應當前系統時間窗口1-2,則僅對資料進行快取而不處理。 Specifically, the to-be-processed data received in this time period corresponds to the previous system time window 0-1 and the current system time window 1-2. If the data to be processed corresponds to the previous system time window 0-1, the data is processed and output immediately; if the data to be processed corresponds to the current system time window 1-2, only the data is cached and not processed.
步驟2 ct1時刻,輸出資料時間在所述前一系統時間窗口0-1的範圍之內的所有待處理資料的處理結果,並關閉與所述前一系統時間窗口0-1對應的資料時間窗口。 Step 2: At time ct1, outputting the processing result of all the data to be processed within the range of the previous system time window 0-1, and closing the data time window corresponding to the previous system time window 0-1 .
具體地,到達ct1時刻時,則不再快取對應於前一系統時間窗口0-1的待處理資料。 Specifically, when the time ct1 is reached, the to-be-processed data corresponding to the previous system time window 0-1 is no longer cached.
步驟3 ct1~st1,接收待處理資料,根據所述待處理資料的資料時間進行判斷,若所述資料時間在當前系統時間窗口1-2的範圍之內,將待處理資料進行快取,並對快取的將待處理資料進行全域同步;若所述資料時間在前一系統時間窗口0-1的範圍之內,丟棄待處理資料。 Step 3 ct1~st1, receiving the data to be processed, and determining according to the data time of the data to be processed. If the data time is within the range of the current system time window 1-2, the data to be processed is cached, and The data to be processed is globally synchronized for the cache; if the data time is within the range of 0-1 of the previous system time window, the pending data is discarded.
具體地,此時間段內接收的待處理資料,既有對應前一系統時間窗口0-1的,也有對應當前系統時間窗口1-2的。若待處理資料對應前一系統時間窗口0-1,則丟棄待處理資料;若待處理資料對應當前系統時間窗口1-2,則對待處理資料進行快取以進行全域同步而不處理。 Specifically, the to-be-processed data received in this time period corresponds to the previous system time window 0-1 and the current system time window 1-2. If the data to be processed corresponds to the previous system time window 0-1, the data to be processed is discarded; if the data to be processed corresponds to the current system time window 1-2, the data to be processed is cached for global synchronization without processing.
步驟4 st1~2,接收待處理資料並處理,同時還對1~ct1快取的,實時輸出上述處理結果。 Step 4 st1~2, receiving the data to be processed and processing, and also outputting the above processing result in real time for 1~ct1 cache.
具體地,此時間段內接收的待處理資料有對應當前系統時間窗口1-2的。將所接收的待處理資料連同之前在1~st1時間段內快取的資料時間在當前系統時間窗口1-2的範圍之內的待處理資料進行處理,並實時輸出上述處理結果。上述具體應用場景的技術方案中,在針對需要實時並完整輸出計算結果的串流計算過程中,透過預先為資料統計系統的每一段系統時間窗口設置關閉時間點以及同步時間點的方式,對統計資料的進行全域同步,可以在保證資料統計完整的前提下對其及時進行處理,從而提高了資料處理結果的準確性和實時性。 Specifically, the to-be-processed data received during this time period corresponds to the current system time window 1-2. The received data to be processed is processed along with the data to be processed within the range of the current system time window 1-2, together with the data time cached in the previous time period of 1~st1, and the processing result is output in real time. In the technical solution of the specific application scenario described above, in the process of calculating the streaming stream that requires real-time and complete outputting of the calculation result, the statistics are set by closing the time point and the synchronization time point in advance for each system time window of the data statistics system. The global synchronization of the data can be processed in a timely manner on the premise of ensuring the integrity of the data, thereby improving the accuracy and real-time performance of the data processing results.
在此需要說明的是,以上具體的應用場景的內容僅為本發明優選實施例提出的示例,在此基礎上還可以包括更多的應用領域,從而使得本技術方案具有更廣泛的應用,這些改進都屬於本發明的保護範圍。 It should be noted that the content of the above specific application scenario is only an example provided by the preferred embodiment of the present invention, and more application fields may be included on the basis of the above, so that the technical solution has a wider application. Improvements are within the scope of the invention.
為達到以上技術目的,本發明還提出了一種資料處理設備,如圖3所示,該設備應用於資料統計系統中,其中所述資料統計系統的每一段系統時間窗口均預先設置有關閉時間點以及同步時間點,其中,同一系統時間窗口的所述關閉時間點位於所述同步時間點之前以及起始時間點之後。該設備包括:確定模組,當在當前系統時間窗口接收到待處理資料時,根據所述待處理資料的資料時間確定與所述待處理資 料對應的系統時間窗口;第一處理模組,在所述資料時間在當前系統時間窗口範圍之內,將系統接收時間在所述同步時間點之前的待處理資料進行快取,以及將所述系統接收時間在所述同步時間點之後的待處理資料即時進行處理;第二處理模組,在所述資料時間在前一系統時間窗口的範圍之內,將所述系統接收時間在所述關閉時間點之前的待處理資料即時進行處理,以及將所述系統接收時間在所述關閉時間點之後的待處理資料丟棄。 In order to achieve the above technical purpose, the present invention also provides a data processing device. As shown in FIG. 3, the device is applied to a data statistics system, wherein each system time window of the data statistics system is preset with a closing time point. And a synchronization time point, wherein the closing time point of the same system time window is before the synchronization time point and after the starting time point. The device includes: a determining module, when receiving the to-be-processed data in the current system time window, determining a system time window corresponding to the to-be-processed data according to the data time of the to-be-processed data; the first processing module is The data time is within a range of the current system time window, the system receiving time is cached by the data to be processed before the synchronization time point, and the data to be processed after the system receiving time is after the synchronization time point Instant processing; the second processing module, in the range of the data time of the previous system time window, the processing data of the system receiving time before the closing time point is processed immediately, and The pending data of the system receiving time after the closing time point is discarded.
在具體的應用場景中,所述確定模組具體用於:根據所述待處理資料在資料時間窗口的時間戳記,獲取所述待處理資料的資料時間;判斷所述資料時間是否在當前的系統時間窗口的起始時間點之後;若是,確認所述待處理資料在當前系統時間窗口範圍之內;若否,確認所述待處理資料在之前系統時間窗口範圍之內。 In a specific application scenario, the determining module is specifically configured to: obtain a data time of the to-be-processed data according to a time stamp of the data to be processed in the data time window; and determine whether the data time is in a current system. After the start time point of the time window; if yes, confirm that the to-be-processed data is within the current system time window; if not, confirm that the pending data is within the previous system time window.
在具體的應用場景中,所述設備還包括:第一輸出模組,在所述資料統計系統的系統時間到達所述關閉時間點時,將資料時間在所述前一系統時間窗口範圍之內的待處理資料的處理結果進行輸出,並關閉與所述前一系統時間窗口對應的資料時間窗口。 In a specific application scenario, the device further includes: a first output module, when the system time of the data statistics system reaches the closing time point, the data time is within the range of the previous system time window The processing result of the to-be-processed data is output, and the data time window corresponding to the previous system time window is closed.
在具體的應用場景中,所述設備還包括: 第二輸出模組,在所述資料統計系統的系統時間到達所述同步時間點時,處理在所述同步時間點之前所快取的且資料時間在當前系統時間窗口範圍之內的待處理資料,並輸出處理結果。 In a specific application scenario, the device further includes: a second output module, when the system time of the data statistics system reaches the synchronization time point, processing the data cached before the synchronization time point The data to be processed whose time is within the current system time window and outputs the processing result.
在具體的應用場景中,所述關閉時間點根據所述系統時間與所述資料時間之間的時間差值設置;所述同步時間點根據關閉時間窗口的耗時以及所述資料統計系統中各設備之間的時鐘差值設置。 In a specific application scenario, the closing time point is set according to a time difference between the system time and the data time; the synchronization time point is based on time consumption of closing the time window and each of the data statistics systems The clock difference setting between devices.
透過以上的實施方式的描述,本領域的技術人員可以清楚地瞭解到本發明可以透過硬體實現,也可以借助軟體加必要的通用硬體平臺的方式來實現。基於這樣的理解,本發明的技術方案可以以軟體產品的形式體現出來,該軟體產品可以儲存在一個非易失性儲存媒體(可以是CD-ROM,USB,行動硬碟等)中,包括若干指令用以使得一台電腦設備(可以是個人電腦,伺服器,或者網路設備等)執行本發明各個實施場景所述的方法。 Through the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be implemented by hardware, or by means of a software plus a necessary universal hardware platform. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a USB, a mobile hard disk, etc.), including several The instructions are for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various implementation scenarios of the present invention.
本領域技術人員可以理解圖式只是一個優選實施場景的示意圖,圖式中的模組或流程並不一定是實施本發明所必須的。 Those skilled in the art can understand that the drawings are only a schematic diagram of a preferred implementation scenario, and the modules or processes in the drawings are not necessarily required to implement the present invention.
本領域技術人員可以理解實施場景中的裝置中的模組可以按照實施場景描述進行分佈於實施場景的裝置中,也可以進行相應變化位於不同於本實施場景的一個或多個裝置中。上述實施場景的模組可以合併為一個模組,也可以進一步拆分成多個子模組。 A person skilled in the art may understand that the modules in the device in the implementation scenario may be distributed in the device that implements the scenario according to the implementation scenario description, or may be correspondingly changed in one or more devices different from the implementation scenario. The modules of the above implementation scenarios may be combined into one module, or may be further split into multiple sub-modules.
上述本發明序號僅僅為了描述,不代表實施場景的優劣。 The above-mentioned serial numbers of the present invention are merely for description, and do not represent the advantages and disadvantages of the implementation scenario.
以上公開的僅為本發明的幾個具體實施場景,但是,本發明並非局限於此,任何本領域的技術人員能思之的變化都應落入本發明的保護範圍。 The above disclosure is only a few specific implementation scenarios of the present invention, but the present invention is not limited thereto, and any changes that can be made by those skilled in the art should fall within the protection scope of the present invention.
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW106102697A TW201828097A (en) | 2017-01-24 | 2017-01-24 | Data processing method and apparatus capable of performing a real-time processing under the premise of ensuring the integrity of data statistics |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW106102697A TW201828097A (en) | 2017-01-24 | 2017-01-24 | Data processing method and apparatus capable of performing a real-time processing under the premise of ensuring the integrity of data statistics |
Publications (1)
Publication Number | Publication Date |
---|---|
TW201828097A true TW201828097A (en) | 2018-08-01 |
Family
ID=63960231
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
TW106102697A TW201828097A (en) | 2017-01-24 | 2017-01-24 | Data processing method and apparatus capable of performing a real-time processing under the premise of ensuring the integrity of data statistics |
Country Status (1)
Country | Link |
---|---|
TW (1) | TW201828097A (en) |
-
2017
- 2017-01-24 TW TW106102697A patent/TW201828097A/en unknown
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2017107793A1 (en) | Data processing method and device | |
US20220377019A1 (en) | In-order processing of network packets | |
US7865760B2 (en) | Use of T4 timestamps to calculate clock offset and skew | |
US10798199B2 (en) | Network traffic accelerator | |
US8484269B2 (en) | Computing time-decayed aggregates under smooth decay functions | |
WO2015096692A1 (en) | Method and system for controlling data reception traffic and computer storage medium | |
WO2020135436A1 (en) | Block delayed broadcasting method, device and storage medium | |
CN110932814B (en) | Software-defined network time service safety protection method, device and system | |
US20080049635A1 (en) | Method and system for determining one-way packet travel time using RTCP | |
US9535450B2 (en) | Synchronization of data streams with associated metadata streams using smallest sum of absolute differences between time indices of data events and metadata events | |
US20090190604A1 (en) | Method and System for Dynamically Adjusting Acknowledgement Filtering for High-Latency Environments | |
US20120327954A1 (en) | Packet Meta-Tagging Using Excess Payload | |
US8898223B2 (en) | System for high reliability and high performance application message delivery | |
EP3602974B1 (en) | Apparatus and method for maintaining message databases in eventual consistency distributed database systems | |
WO2015196637A1 (en) | Method and apparatus for service clock passthrough | |
CN113965488A (en) | Method and device for delayed acquisition of data packets, electronic device and storage medium | |
WO2017097034A1 (en) | Time stamp processing method and apparatus for time message in optical transport network | |
US20200374337A1 (en) | Transmitting data over a network in representational state transfer (rest) applications | |
TW201828097A (en) | Data processing method and apparatus capable of performing a real-time processing under the premise of ensuring the integrity of data statistics | |
US20170337316A1 (en) | Data processing system | |
Wu et al. | Accurate traffic replay based on interactive sequence and timestamp | |
US20130283293A1 (en) | System and method for Intelligently distributing a plurality of transactions for parallel processing | |
CN112866178A (en) | Method and device for audio data transmission | |
US20180091434A1 (en) | Managing a data stream in a multicore system | |
Vinogradov et al. | Measurement of one-way delays in IP networks |