WO2015067115A1

WO2015067115A1 - Deduplication method and apparatus of real-time system data

Info

Publication number: WO2015067115A1
Application number: PCT/CN2014/088312
Authority: WO
Inventors: 杨基彬
Original assignee: 北京国双科技有限公司
Priority date: 2013-11-07
Filing date: 2014-10-10
Publication date: 2015-05-14
Also published as: CN103559282A; CN103559282B

Abstract

Disclosed are a deduplication method and apparatus of real-time system data. The deduplication method of real-time system data comprises: receiving the real-time system data; determining whether a data source of first data is same as a data source of the real-time system data, the first data being data stored in a data buffer; and if it is determined that the data source of the first data is same as the data source of the real-time system data, deleting data that is same in the data source of the first data and the data source of the real-time system data; and temporarily storing the real-time system data to the data buffer. The present invention solves the problem of low deduplication efficiency of a real-time system, thereby improving the efficiency of the real-time system.

Description

Deduplication method and device for real-time system data

Technical field

The present invention relates to the field of computers, and in particular to a method and apparatus for deduplicating real-time system data.

Background technique

StreamInsight is a real-time data stream processing framework provided by Microsoft Corporation of the United States for efficient real-time computing. However, StreamInsight does not have the function of deduplicating statistics. In actual applications, it is sometimes necessary to use deduplication statistics.

Real-time processing systems usually acquire the latest data at regular intervals, which facilitates real-time management of the system, but at the same time the amount of data is large. At the same time, the data of the real-time system generally has a data cycle, and repeated statistics often occur when statistics are performed on the data. For example, the video real-time processing system needs to receive the broadcast data in real time. Each play Id sends a snapshot data every interval, indicating the latest playback information.

The prior art solution is to use the stream clip method to count the number of times of play. When the number of times of play is counted by the stream clip method, the play data is given a certain life cycle first, and if the number of play times is calculated once every minute, the life cycle of the play data is played. Set to 1 minute, when the new snapshot with the same Id is played, the life cycle of the old snapshot is truncated to the start time of the new snapshot. When you want to calculate the time interval from 00:00:00 to 00:00:59 For the number of playbacks, you only need to count the total number of snapshots at 59 seconds. Although the stream editing scheme seems to be very convenient to count the number of playbacks, it is not easy to implement, because it needs to use the same stream self-joining. If there are 10w playback snapshots in 1 minute, it is equivalent to 10w playback snapshots. The entry and 10w play snapshot entries are subjected to a Cartesian product of the set, and then the records that match the filter criteria are filtered out. This calculation is very CPU and memory.

In view of the low efficiency of deduplication of real-time system data in the related art, an effective solution has not yet been proposed.

Summary of the invention

The main object of the present invention is to provide a method and apparatus for deduplicating real-time system data to solve the problem of low data de-duplication efficiency of real-time systems.

According to an aspect of the present invention, a method for de-emphasizing real-time system data includes: receiving real-time system data; determining whether a data source of the first data is the same as a data source of real-time system data, wherein the first number According to the data stored in the data buffer; when it is determined that the data source of the first data is the same as the data source of the real-time system data, deleting the data in the first data and the data source of the real-time system data; and Temporarily store real-time system data to the data buffer.

Optionally, the number of real-time system data is multiple, and determining whether the data source of the first data is the same as the data source of the real-time system data includes: determining a data source and real-time system data of the first data each time a real-time system data is received. Is the data source the same?

Optionally, it is determined whether the data source of the first data is the same as the data source of the real-time system data by determining whether the identifier ID of each first data is the same as the identifier ID of the real-time system data.

Optionally, after temporarily storing the real-time system data to the data buffer, the de-duplication method further includes: storing the data in the data buffer to the target storage area; and clearing the data in the data buffer.

Optionally, the data in the data buffer is stored to the target storage area every predetermined time interval.

Optionally, the real-time system data is data from a video real-time processing system or a webpage real-time processing system.

According to another aspect of the present invention, there is provided a deduplication device for real-time system data, the deduplication device being mainly for performing a de-duplication method for any real-time system data provided by the above-mentioned contents of the present invention.

In order to achieve the above object, according to another aspect of the present invention, a deduplication device for real-time system data is provided, comprising: a receiving unit, configured to receive real-time system data; and a determining unit, configured to determine a data source of the first data Whether the data source of the real-time system data is the same, wherein the first data is data stored in the data buffer; and the deleting unit is configured to determine, when the data source of the first data is the same as the data source of the real-time system data, The data in the first data is deleted from the data source of the real-time system data source; and the temporary storage unit is configured to temporarily store the real-time system data to the data buffer.

Optionally, the number of real-time system data is multiple, and the determining unit includes: a determining sub-unit, configured to determine whether the data source of the first data and the data source of the real-time system data are the same each time a real-time system data is received.

Optionally, the determining subunit includes: a determining module, configured to determine whether the identifier ID of each first data is the same as the identifier ID of the real-time system data, to determine whether the data source of the first data is the same as the data source of the real-time system data. .

Optionally, the deduplication device further includes: a storage unit configured to store data in the data buffer to the target storage area; and a clearing unit to clear data in the data buffer.

Optionally, the storage unit is configured to store data in the data buffer to the target storage area every predetermined time interval.

The invention adopts receiving real-time system data; determining whether the data source of the first data is the same as the data source of the real-time system data, wherein the first data is data stored in the data buffer; and the data source of the first data is determined When the data source of the real-time system data is the same, the data source in the first data is deleted and the data of the real-time system data source is the same; and the real-time system data is temporarily stored in the data buffer. Before receiving real-time system data, a data buffer is created, and the real-time system data is temporarily stored in the buffer, and the data is judged before the data is temporarily buffered to the buffer, if there is a data source from the same data source. The data first deletes the data of the data source in the data buffer, and deduplicates the data source before the next step of processing, avoiding a large amount of data of the same data source directly entering the real-time processing system for deduplication. The problem of low efficiency of real-time system is solved, and the effect of real-time system is improved.

DRAWINGS

The accompanying drawings, which are incorporated in the claims In the drawing:

1 is a flow chart of a real-time system data deduplication method according to an embodiment of the present invention;

2 is a schematic diagram of a data life cycle before stream editing according to the prior art;

3 is a schematic diagram of a data life cycle after stream editing according to the prior art;

4 is a schematic structural diagram of a real-time system data deduplication apparatus according to an embodiment of the present invention.

detailed description

It should be noted that the embodiments in the present application and the features in the embodiments may be combined with each other without conflict. The invention will be described in detail below with reference to the drawings in conjunction with the embodiments.

The invention provides a deduplication method for real-time system data. The following describes the de-duplication method of the real-time system data of the present invention:

1 is a flow chart of a real-time system data deduplication method according to an embodiment of the present invention. As shown in FIG. 1, the method includes the following steps S102 to S108:

Step S102, receiving real-time system data. Specifically, a data buffer is opened in the memory before receiving the data.

Step S104, determining whether the data source of the first data is the same as the data source of the real-time system data, wherein the first data is data stored in the data buffer.

Step S106, in the case that it is determined that the data source of the first data is the same as the data source of the real-time system data, deleting the same data of the data source in the first data and the real-time system data data source,

Step S108, temporarily storing the real-time system data to the data buffer.

The deduplication method of the real-time system data in the embodiment of the present invention first establishes a data buffer before receiving the real-time system data, and temporarily stores the real-time system data into the buffer, and before the data is temporarily stored in the buffer, the data is first Judging, if there is data from the same data source in the buffer, the data of the data source in the data buffer is deleted first, and the duplicate data of the data source is deduplicated before proceeding to the next step, avoiding the same A large amount of data of the data source directly enters the real-time processing system for deduplication, and solves the problem that the real-time system has low de-emphasis efficiency, thereby achieving the effect of improving the efficiency of the real-time system processing data.

Optionally, the real-time system data is multiple, and determining whether the data source of the first data is the same as the data source of the real-time system data comprises: determining the data source of the first data and the received real-time system each time a real-time system data is received. Whether the data source of the data is the same. Each data source of the real-time system will send an up-to-date data every interval to monitor the running status of the system. Whenever a new data is received, a judgment is made, that is, the data source of the data in the data buffer is received and received. The data source of the real-time system data is compared. If the data source of a certain data in the data buffer is the same as the data source of the received real-time system data, the old data in the data buffer is deleted, and the data source is temporarily stored. The latest data sent.

Optionally, in the embodiment of the present invention, whether the data source of the first data is the same as the data source of the real-time system data is determined by determining whether the identifier ID of each first data is the same as the identifier ID of the real-time system data. Wherein, when it is determined that the identifier ID of the first data is the same as the identifier ID of the real-time system data, the data source of the first data is determined to be the same as the data source of the real-time system data, and vice versa. Such a method of judging by the ID is also called a self-joining judging method, and the real-time system data is taken as an example for the data from the video real-time processing system, and it is assumed that a plurality of "playing records" are currently being processed, and each of the pieces is played. The record has a unique play Id. For judging whether the data source of the newly received video play data is the same as the data source of the already recorded video play data, the newly received video play data set and the already recorded video play are played. The data set is connected. When the two sets are connected, the playback ID is determined according to the condition of the connection, that is, the play record of any one play Id is selected from the newly received video data set, and the set of the video play data has been recorded. Go to find if there is a play record with the same play Id.

Through one-to-one self-connection, it is more efficient to judge whether the data in the data buffer and the real-time system data are from the same source than the many-to-many data self-joining.

Optionally, after the real-time system data is temporarily buffered to the data buffer, the de-duplication method of the real-time system data in the embodiment of the present invention further includes: storing the data in the data buffer to the target storage area every predetermined time interval, and clearing The data in the data buffer.

In order to make the latest data of the real-time system output in time, in order to obtain the running state of the real-time system in real time, it is necessary to store the deduplicated data at intervals to facilitate the next processing. After the data is stored, the buffer is emptied to buffer the data received after the buffer. The length of the predetermined time can be determined according to the requirements of the subsequent processing system for real-time data.

Optionally, the de-duplication method of the real-time data in the embodiment of the present invention may be used to count the number of times of video playback in a period of time, or to count the number of times of web browsing in a period of time, that is, in real time provided by the embodiment of the present invention. In the deduplication method of the system data, the real-time system data may be data from a video real-time processing system, or may be data from a webpage real-time processing system. For a better understanding of the embodiments of the present invention, the deduplication method of the real-time data in the embodiment of the present invention is described by taking the statistics of the number of times of video playback in a period of time as an example.

The snapshots received during the time between 00:00:00 and 00:00:59 are shown in the following table:

时间time	00:00:0000:00:00	00:00:0500:00:05	00:00:1000:00:10	00:00:1500:00:15	00:00:2000:00:20
时间time	00:00:0000:00:00	00:00:0500:00:05	00:00:1000:00:10	00:00:1500:00:15	00:00:2000:00:20	播放IdPlay Id	播放1快照1Play 1 snapshot 1	播放1快照2Play 1 snapshot 2	播放1快照3Play 1 snapshot 3	播放2快照1Play 2 snapshot 1	播放2快照2Play 2 snapshot 2

Although there are 5 snapshot data in the table, only Play 1 and Play 2 play twice. Assuming that the lifetime of each data is 1 minute, the number of playbacks will be 5 times during the time period from 00:00:00 to 00:00:59 (as shown in Figure 2), but only Play 2 times. In order to get an accurate number of plays, the stream clip method is used before the number of play times. When a new snapshot of the same ID is encountered, the life cycle of the old snapshot is truncated to the start time of the new snapshot. The snapshot life cycle after stream editing is shown in Figure 3. After the stream clip, statistics are counted from 00:00:00 to 00:00:59.

If the number of snapshots is large, such as 00:00:00 to 00:00:59, there are 100,000 playback snapshots. Then, when streaming clips in StreamInsight, 10000*10000 self-connections will be performed. In order to de-duplicate all snapshots, the de-duplication method of the real-time data of the present invention establishes a data buffer before data is input into StreamInsight, and receives "play 1 snapshot 1" at 00:00:00. Since there is no data in the data buffer at this time, it is directly placed in the data buffer, and 00:00:05 receives "Play 1 Snapshot 2". At this time, the judgment is made first, because "Play 1 Snapshot 1" and "Play 1 Snapshot" 2" are all from the data source "Play 1". So delete "Play 1 Snapshot 1" and save "Play 1 Snapshot 2" to the data buffer. According to this method, data is sequentially received. By 00:00:59, only the data of "Play 1 Snapshot 3" and "Play 2 Snapshot 2" are in the data buffer, that is, 00:00:00 to 00: 00:59 has played twice during this time period.

The present invention also provides a real-time system data deduplication device, which is mainly used to implement the de-duplication method of real-time system data provided by the above content in the embodiment of the present invention, and the following describes the de-duplication method of the real-time system data of the present invention. Introduction:

4 is a schematic structural diagram of a real-time system data deduplication method according to an embodiment of the present invention. As shown in FIG. 4, the device mainly includes a receiving unit 10, a determining unit 20, a deleting unit 30, and a temporary storage unit 40, wherein:

The receiving unit 10 is configured to receive real-time system data. Specifically, a data buffer is opened in the memory before receiving the data.

The determining unit 20 is configured to determine whether the data source of the first data is the same as the data source of the real-time system data, wherein the first data is data stored in the data buffer.

The deleting unit 30 is configured to delete the data in the first data and the data in the real-time system data source when the data source of the first data is determined to be the same as the data source of the real-time system data.

The temporary storage unit 40 is configured to temporarily store real-time system data to the data buffer.

The deduplication device of the real-time system data in the embodiment of the present invention first establishes a data buffer before receiving the real-time system data, and temporarily stores the real-time system data into the data buffer, and before the data is temporarily stored in the data buffer, Judging the data, if there is data from the same data source in the buffer, the data of the data source in the data buffer is deleted first, and the duplicate data of the data source is deduplicated before proceeding to the next step, thereby avoiding A large amount of data of the same data source directly enters the real-time processing system for de-duplication, and solves the problem that the real-time system has low de-emphasis efficiency, thereby achieving the effect of improving the efficiency of real-time system processing data.

Optionally, the real-time system data is multiple, and the determining unit 20 includes a determining sub-unit, configured to determine whether the data source of the first data and the data source of the real-time system data are the same each time a real-time system data is received. Each data source of the real-time system will send an up-to-date data every interval to monitor the running status of the system. Whenever a new data is received, a judgment is made, that is, the data source of the data in the data buffer is received and received. The data source of the real-time system data is compared. If the data source of the data buffer has the same data source as the data of the real-time system received, the old data in the data buffer is deleted, and the data source is sent. The latest data coming.

Optionally, the determining subunit includes a determining module, configured to determine whether the identifier ID of each first data is the same as the identifier ID of the real-time system data, to determine whether the data source of the first data is the same as the data source of the real-time system data, Wherein, when it is determined that the identifier ID of the first data is the same as the identifier ID of the real-time system data, the data source of the first data is determined to be the same as the data source of the real-time system data, and vice versa. Such a method of judging by the identification ID is also called a self-connection judging method, and the real-time system data is used as an example for the data from the video real-time processing system, and it is assumed that a plurality of “play recording records” are currently being processed, each of which is The play record has a unique play Id. For determining whether the data source of the newly received video play data is the same as the data source of the already recorded video play data, the newly received video play data set and the already recorded video are The play data set is connected. When the two sets are connected, the play ID is determined according to the condition of the connection, that is, the play record of any one play Id is selected from the newly received video data set, and the video play data of the already recorded video is recorded. Find the playlist with the same play Id in the collection. .

Optionally, the deduplication device further comprises a storage unit for storing data in the data buffer to the target storage area; and a clearing unit for clearing data in the data buffer. The storage unit mainly stores the data in the data buffer to the target storage area every predetermined time interval. In order to output the latest data of the real-time system in time, so as to obtain the running state of the real-time system in real time, it is necessary to go every other time. The heavy data is stored for easy processing in the next step. After the data is stored, the buffer is emptied to buffer the data received after the buffer. The length of the predetermined time can be determined according to the requirements of the subsequent processing system for real-time data.

Optionally, the deduplication device of the real-time data in the embodiment of the present invention may be used to count the number of times of video playback in a period of time, or to count the number of times of web browsing in a period of time, that is, in real time provided by the embodiment of the present invention. In the deduplication method of the system data, the real-time system data may be data from a video real-time processing system, or may be data from a webpage real-time processing system. The de-duty device is configured to collect a specific statistical manner of the number of video playbacks within a period of time, and the de-duplication method of the real-time system data provided by the foregoing content in the embodiment of the present invention counts the same number of times of video playback in a period of time, where No longer.

From the above description, it can be seen that the present invention solves the problem of low de-emphasis efficiency of the real-time system, thereby achieving the effect of improving the data processing efficiency of the real-time system.

It will be apparent to those skilled in the art that the various modules or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device, such that they may be stored in a storage device for execution by the computing device, or they may be separately fabricated into individual integrated circuit modules. Blocks, or a plurality of modules or steps in them, are implemented as a single integrated circuit module. Thus, the invention is not limited to any specific combination of hardware and software.

The above description is only the preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes can be made to the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and scope of the present invention are intended to be included within the scope of the present invention.

Claims

A method for deduplicating real-time system data, including:

Receiving the real-time system data;

Determining whether the data source of the first data is the same as the data source of the real-time system data, wherein the first data is data stored in a data buffer;

Determining, in a case where the data source of the first data is the same as the data source of the real-time system data, deleting data in the first data that is the same as the data source of the real-time system data;

Temporarily storing the real-time system data to the data buffer.
The deduplication method according to claim 1, wherein the number of the real-time system data is plural, and determining whether the data source of the first data is the same as the data source of the real-time system data comprises: receiving each of the real-time The system data determines whether the data source of the first data is the same as the data source of the real-time system data.
The deduplication method according to claim 2, wherein the data source of the first data is determined by determining whether the identification ID of each of the first data and the identification ID of the real-time system data are the same Whether the data source of real-time system data is the same.
The deduplication method according to claim 1, wherein after the storing the real-time system data to the data buffer, the de-duplication method further comprises:

Storing data in the data buffer to a target storage area;

Empty the data in the data buffer.
The deduplication method according to claim 4, wherein the data in the data buffer is stored to the target storage area every predetermined time interval.
The deduplication method according to claim 1, wherein the real-time system data is data from a video real-time processing system or a webpage real-time processing system.
A deduplication device for real-time system data, comprising:

a receiving unit, configured to receive the real-time system data;

a determining unit, configured to determine whether a data source of the first data is the same as a data source of the real-time system data, where the first data is data stored in a data buffer;

a deleting unit, configured to delete, in a case where the data source of the first data is the same as the data source of the real-time system data, deleting the data in the first data and the data source in the real-time system data source ;as well as

a temporary storage unit, configured to temporarily store the real-time system data to the data buffer.
The deduplication device according to claim 7, wherein the number of the real-time system data is plural, and the judging unit comprises:

The determining subunit is configured to determine whether the data source of the first data and the data source of the real-time system data are the same each time the real-time system data is received.
The deduplication device of claim 8, wherein the determining subunit comprises:

The determining module is configured to determine whether the identifier ID of each of the first data is the same as the identifier ID of the real-time system data, to determine whether the data source of the first data is the same as the data source of the real-time system data.
The deduplication device of claim 7, wherein the deduplication device further comprises:

a storage unit, configured to store data in the data buffer to a target storage area;

Emptying the unit for clearing data in the data buffer.
The deduplication device according to claim 10, wherein said storage unit is configured to store data in said data buffer to said target storage area every predetermined time interval.
The deduplication device of claim 7, wherein the real-time system data is data from a video real-time processing system or a web page real-time processing system.