CN107145529B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN107145529B CN107145529B CN201710250035.4A CN201710250035A CN107145529B CN 107145529 B CN107145529 B CN 107145529B CN 201710250035 A CN201710250035 A CN 201710250035A CN 107145529 B CN107145529 B CN 107145529B
- Authority
- CN
- China
- Prior art keywords
- data
- window
- filter processing
- processing unit
- window data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24539—Query rewriting; Transformation using cached or materialised query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2428—Query predicate definition using graphical user interfaces, including menus and forms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/335—Filtering based on additional data, e.g. user or group profiles
Abstract
The disclosure relates to a data processing method and device. The method comprises the following steps: the current filter processing unit receives incremental data and a control signal sent by an upstream filter processing unit, wherein the control signal comprises starting time corresponding to the incremental data; the current filter processing unit judges whether window data need to be constructed or not; if window data need to be constructed, reading historical window data from a local cache according to the size of a preset window and the starting time; the current filter processing unit cleans the data with the longest time in the historical window data according to the time step corresponding to the incremental data to obtain the retained data in the historical window data; and the current filter processing unit constructs new window data based on the reserved data and the incremental data, and replaces the historical window data in the local cache with the new window data, wherein the window size of the new window data is the same as the size of a preset window. By the scheme, the incremental query scheme can be better suitable for the pipeline-filter mode.
Description
Technical Field
The present disclosure relates to the field of computer processing technologies, and in particular, to a data processing method and apparatus.
Background
At present, when tracking and querying window data, the following two methods are mostly adopted:
in the first mode, the window data is queried in full quantity. That is to say, a single query needs to load all data in a window, the method is simple and easy to implement, but in a large data scene, due to a large data base number, a large amount of repeated data can be loaded by repeated queries, so that resource waste is caused, and a performance bottleneck is easy to appear in practical application.
And the mode two is incremental query of window data. In order to solve the problem of resource waste in the first mode, the prior art provides an incremental query scheme, that is, only the latest incremental data in a window needs to be loaded in a single query, so that resource waste caused by repeatedly loading repeated data can be effectively avoided.
However, in practical applications, when the incremental query scheme is applied to a Pipe-And-Filter (Pipe-And-Filter) mode, for a Filter processing unit that needs to perform global processing, if only the latest incremental data is provided, it cannot be ensured that the final window data is obtained, And thus the query result of the window data cannot be obtained. That is, current incremental query schemes are not well suited for the pipe-filter model.
Disclosure of Invention
The purpose of the present disclosure is to provide a data processing method and apparatus, so that the incremental query scheme can be better applied to the pipeline-filter model.
In order to achieve the above object, in a first aspect, the present disclosure provides a data processing method, including:
the method comprises the steps that a current filter processing unit receives incremental data and control signals sent by an upstream filter processing unit, wherein the control signals comprise starting time corresponding to the incremental data;
the current filter processing unit judges whether window data need to be constructed or not;
if window data need to be constructed, the current filter processing unit reads historical window data from a local cache according to the size of a preset window and the starting time;
the current filter processing unit clears the data with the longest time in the historical window data according to the time step corresponding to the incremental data to obtain the reserved data in the historical window data;
and the current filter processing unit constructs new window data based on the retention data and the incremental data, and replaces historical window data in the local cache with the new window data, wherein the window size of the new window data is the same as the size of the preset window.
Optionally, the preset window size is transmitted to the current filter processing unit through the control signal; or, the preset window size is pre-configured in the current filter processing unit.
Optionally, the determining, by the current filter processing unit, whether window data needs to be constructed includes:
the current filter processing unit is configured with a preset zone bit, and whether window data need to be constructed is judged according to the state of the preset zone bit.
Optionally, the determining, by the current filter processing unit, whether window data needs to be constructed includes:
the current filter processing unit judges whether the preset window size is configured or not;
and if the preset window size is configured, the current filter processing unit judges that window data needs to be constructed.
Optionally, if the current filter processing unit is provided with a corresponding index file, then
The reading of the historical window data from the local cache comprises: the current filter processing unit reads the historical window data from the index file;
the replacing the historical window data in the local cache with the new window data includes: and the current filter processing unit writes the new window data into the index file to replace the historical window data.
Optionally, if the current filter processing unit determines that window data does not need to be constructed, the method further comprises:
and the current filter processing unit transmits the control signal and the processing result of the incremental data to a downstream link.
In a second aspect, the present disclosure provides a data processing apparatus belonging to a current filter processing unit, the apparatus comprising:
the receiving module is used for receiving incremental data and control signals sent by an upstream filter processing unit, wherein the control signals comprise starting time corresponding to the incremental data;
the judging module is used for judging whether window data need to be constructed or not;
the reading module is used for reading historical window data from a local cache according to the size of a preset window and the starting time when the judging module judges that the window data need to be constructed;
the cleaning module is used for cleaning the data with the longest time in the historical window data according to the time step corresponding to the incremental data to obtain the reserved data in the historical window data;
the construction module is used for constructing new window data based on the reserved data and the incremental data, and the window size of the new window data is the same as the size of the preset window;
and the replacing module is used for replacing the historical window data in the local cache by using the new window data.
Optionally, the preset window size is transmitted to the current filter processing unit through the control signal; or, the preset window size is pre-configured in the current filter processing unit.
Optionally, the determining module is configured to determine whether window data needs to be constructed according to a state of a preset flag bit when the preset flag bit is configured.
Optionally, the determining module is configured to determine whether the preset window size is configured; and if the preset window size is configured, judging that window data needs to be constructed.
Optionally, if the current filter processing unit is provided with a corresponding index file, then
The reading module is used for reading the historical window data from the index file;
and the replacing module is used for writing the new window data into the index file to replace the historical window data.
Optionally, the apparatus further comprises:
and the transmission module is used for transmitting the control signal and the processing result of the incremental data to a downstream link when the judgment module judges that the window data does not need to be constructed.
In a third aspect, the present disclosure provides a data processing apparatus belonging to a current filter processing unit, the apparatus comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to:
receiving incremental data and a control signal sent by an upstream filter processing unit, wherein the control signal comprises a starting time corresponding to the incremental data;
judging whether window data need to be constructed or not;
if window data need to be constructed, reading historical window data from a local cache according to the size of a preset window and the starting time;
according to the time step corresponding to the incremental data, cleaning the data with the longest time in the historical window data to obtain the retained data in the historical window data;
and constructing new window data based on the reserved data and the incremental data, and replacing historical window data in the local cache with the new window data, wherein the window size of the new window data is the same as the size of the preset window.
The data processing scheme can improve the existing pipeline-filter mode, adds a control pipeline for transmitting control signals between adjacent filter processing units, and enables the filter processing unit needing to construct window data to have local cache read-write capacity. Therefore, when the upstream link transmits the incremental data through the data pipeline, the starting time corresponding to the incremental data can be transmitted through the control pipeline, so that the current filter processing unit can conveniently read the historical window data from the local cache according to the starting time and the size of the preset window, and further, the incremental data and the historical window data are combined to construct a complete new window data, so that the incremental query scheme can be better suitable for a pipeline-filter mode.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure without limiting the disclosure. In the drawings:
FIG. 1 is a schematic illustration of a prior art tube-filter model;
FIG. 2 is a schematic view of a tube-filter pattern of the present disclosure;
FIG. 3 is a schematic flow diagram of a data processing method of the present disclosure;
FIG. 4 is a schematic diagram of an architecture of a data processing apparatus according to the present disclosure;
fig. 5 is a schematic view of another structure of the data processing apparatus of the present disclosure.
Detailed Description
The following detailed description of specific embodiments of the present disclosure is provided in connection with the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present disclosure, are given by way of illustration and explanation only, not limitation.
Before introducing the present disclosure, the pipe-filter model is explained as follows.
Referring to the prior art tube-filter model shown in fig. 1, it may include: the device comprises a plurality of filter processing units for data processing, and a plurality of data pipelines for data transmission between adjacent filter processing units. Among them, the filter processing units can be classified into two types according to the implemented functions: the device comprises a filter processing unit for carrying out global processing and a filter processing unit for carrying out non-global processing. For example, if the filter processing unit is configured to perform data classification, average calculation, and the like on the window data, it may be considered that the filter processing unit needs to perform global processing on the complete window data.
For example, the following tasks are created at 8: 00: and collecting the transaction records through a pipeline-filter mode, and calculating the average transaction amount of the single transaction in the last hour every 10 minutes according to the transaction amount in the transaction records. The transaction amount in the transaction record collected in the last hour can be regarded as a complete window data, and the average transaction amount of a single transaction in the last hour can be regarded as a query result of the window data.
If the full query scheme is adopted to realize the tasks: firstly, the filter processing unit 1 can load 7: 00-8: 00 original data from a data source at 8:00, and calculates an average transaction amount by matching with other filter processing units in a link. Secondly, the filter processing unit 1 can load 7: 10-8: 10 original data from a data source at 8:10, and calculates another average transaction amount by matching with other filter processing units in the link. The above steps are repeated, and data loading and average transaction amount calculation are carried out every 10 minutes. According to the example, a large amount of repeated data can be loaded by repeated query, so that resource waste is caused.
If the incremental query scheme is adopted to realize the tasks: firstly, the filter processing unit 1 can load 7: 00-8: 00 original data from a data source at 8:00, and calculates an average transaction amount by matching with other filter processing units in a link. Secondly, the filter processing unit 1 can load 8: 00-8: 10 original data from a data source at 8:10, that is, only incremental data is transmitted in a link, if the filter processing unit 3 is configured to calculate an average transaction amount, in the current round of data processing, the filter processing unit 3 can only receive the incremental data of the time period of 8: 00-8: 10 transmitted by an upstream link, that is, the filter processing unit 3 cannot obtain 7: 10-8: 10 complete window data, that is, cannot calculate the average transaction amount corresponding to 7: 10-8: 10.
In view of the above, the present disclosure provides a data processing method to enable an incremental query scheme to be better adapted to a pipe-filter model. In response, the present disclosure requires two improvements to the tube-filter model shown in fig. 1:
in the first aspect, a complete window data is constructed, and the time corresponding to the window needs to be determined, so that a control pipeline for transmitting a control signal can be additionally arranged between adjacent filter processing units, and the start time of incremental data transmitted by the data pipeline is identified through the control signal.
In the second aspect, considering that the data received by the filter processing unit through the data pipeline are all processed by the upstream link, in order to make incremental query feasible, the filter processing unit that needs to construct the window data may be configured to have a local cache read-write capability. Therefore, data processed by the upstream link can be locally cached as historical data, and the filter processing unit can construct complete window data in a mode of reading the historical data. As an example, local caching of data may be achieved by configuring the filter processing unit with an index file.
Through the two improvements, the pipeline-filter mode shown in fig. 2 can be obtained, and then incremental query of window data is realized according to the data processing scheme disclosed by the invention. The following explains the present disclosure with reference to examples.
Fig. 3 is a schematic flow chart of the data processing method of the present disclosure. The method may comprise the steps of:
When increment inquiry is carried out, the current filter processing unit can receive increment data transmitted by the upstream filter processing unit through the data pipeline and receive control signals transmitted by the upstream filter processing unit through the control pipeline.
In connection with the above example, the filter processing unit 1 loads 8: 00-8: 10 raw data from the data source at 8:10, that is, incremental data of the time period of 8: 00-8: 10 transmitted to the filter processing unit 3 through the data pipeline, and the processing of the filter processing units 1 and 2 is performed; the control signal transmitted to the filter processing unit 3 via the control conduit may include at least a start time 8:00 corresponding to the incremental data.
It is to be understood that the disclosed solution may refer to the filter processing unit in the link that is performing data processing as the current filter processing unit. In the above example, the filter processing unit 3 is the current filter processing unit.
In step 302, the current filter processing unit determines whether window data needs to be constructed.
In the solution of the present disclosure, the filter processing unit that needs to construct window data generally has the following characteristics: used for carrying out global processing; there is an upstream filter processing unit, i.e. the window data required by the present filter processing unit cannot be constructed directly from the raw data in the data source.
As an example, whether window data needs to be constructed may be determined by configuring a preset flag bit. Specifically, the state of the preset flag bit can be obtained, and accordingly, whether window data needs to be constructed is judged. For example, a default flag bit state of "1" indicates that window data needs to be constructed, and a state of "0" indicates that window data does not need to be constructed.
As an example, whether window data needs to be constructed may be determined by whether a preset window size is configured. Specifically, if the current filter processing unit is configured with a preset window size, it is indicated that window data needs to be constructed.
And 304, the current filter processing unit cleans the longest data in the historical window data according to the time step corresponding to the incremental data to obtain the reserved data in the historical window data.
If the current filter processing unit needs to construct window data, historical window data can be read first, outdated data can be cleared on the basis, incremental data are added, and new window data are formed. The method can be embodied as the following steps:
first, a local cache can be accessed, and the data according with the size of a preset window is determined as historical window data by pushing forward on the basis of the starting time. In the above example, the starting time is 8:00, and the preset window size is 1 hour, so that the data between 7:00 and 8:00 can be determined as the historical window data.
In the scheme of the disclosure, the size of a preset window can be configured in the current filter processing unit in advance; alternatively, the preset window size may be transmitted to the current filter processing unit through a control signal, that is, the control signal at least includes: and the initial time and the preset window size corresponding to the incremental data. The present disclosure may not specifically limit the manner in which the current filter processing unit obtains the preset window size.
And secondly, determining overdue data in the historical window data according to the time step corresponding to the incremental data, and cleaning the overdue data. In the example mentioned above, the time step corresponding to the incremental data is 10 minutes (8: 00-8: 10), so that the oldest data in the history window data, that is, the data between 7:00 and 7:10, can be determined as the expired data, data cleaning is performed, and the data between 7:10 and 8:00 is determined as the reserved data in the history window data.
And finally, constructing new window data by using the incremental data and the reserved data in the historical window data transmitted by the upstream filter processing unit. Corresponding to the above example, 7: 10-8: 10 complete window data can be constructed, and then average transaction amount corresponding to 7: 10-8: 10 is calculated.
It can be understood that, in order to ensure the normal operation of the next round of data processing, the new window data may be cached locally, and in order to reduce the space occupied by the local cache as much as possible, the data may be saved in a manner that the historical window data is replaced by the new window data. Therefore, when the average transaction amount corresponding to the next round of 7: 20-8: 20 is calculated, the data in the time period of 7: 10-8: 10 can be used as historical window data, new window data corresponding to the next round of data processing process is constructed, and the steps are repeated in a circulating mode, and incremental query is achieved in a pipeline-filter mode.
The following explains the implementation process of the scheme of the present disclosure with reference to the above-mentioned task of calculating the average transaction amount.
It is assumed that the filter processing units 1, 2 are configured to perform non-global processing, the filter processing unit 3 is configured to perform global processing, and the filter processing unit 1 is configured with the following information: the preset window size is 1 hour, and the update step length is 10 minutes. The data processing procedure of the present disclosure can be briefly described as follows:
firstly, the filter processing unit 1 loads 7: 00-8: 00 original data from a data source at 8:00, after progressive transmission processing, the filter processing unit 3 calculates to obtain 7: 00-8: 00 average transaction amount, and outputs the result. Meanwhile, the filter processing unit 3 can write the data of 7: 00-8: 00 transmitted by the filter processing unit 2 through the data pipeline into the index file.
Secondly, the filter processing unit 1 loads 8: 00-8: 10 original data from a data source at a ratio of 8:10, transmits a data processing result to the filter processing unit 2 through a data pipeline after completing data processing, and transmits a control signal to the filter processing unit 2 through a control pipeline, wherein the control signal may include: the size of the preset window is 1 hour, and the starting time is 8: 00.
Then, the filter processing unit 2 receives the data and the control signal transmitted by the filter processing unit 1, and determines that the filter processing unit 2 does not need to construct the window data, and the filter processing unit 2 can perform the following processing:
(1) the data transmitted by the filter processing unit 1 is processed, and the data processing result is transmitted to the filter processing unit 3 through a data pipeline;
(2) the control signal is transmitted to the filter processing unit 3 through the control pipeline, and the control signal may include: the size of the preset window is 1 hour, and the starting time is 8: 00.
Finally, the filter processing unit 3 receives the data and the control signal transmitted by the filter processing unit 2, and determines that the filter processing unit 3 needs to construct window data, and the filter processing unit 3 can perform the following processing:
(1) reading 7: 00-8: 00 data from the index file as historical window data, and combining 8: 00-8: 10 incremental data transmitted by the filter processing unit 2 at this time to obtain 7: 10-8: 10 complete window data as new window data of the current data processing;
(2) carrying out average value calculation by using the new window data to obtain an average transaction amount of 7: 10-8: 10, and outputting a result;
(3) and writing new window data (7: 10-8: 10 data) into the index file, and replacing historical window data (7: 00-8: 00 data). Therefore, the data processing process can be completed in the current round.
Fig. 4 is a schematic structural diagram of the data processing apparatus according to the present disclosure. The data processing apparatus belongs to a current filter processing unit, the apparatus comprising:
a receiving module 401, configured to receive incremental data and a control signal sent by an upstream filter processing unit, where the control signal includes a start time corresponding to the incremental data;
a judging module 402, configured to judge whether window data needs to be constructed;
a reading module 403, configured to read historical window data from a local cache according to a preset window size and the start time when the determining module determines that window data needs to be constructed;
a cleaning module 404, configured to clean, according to a time step corresponding to the incremental data, data with the longest time in the historical window data to obtain retained data in the historical window data;
a building module 405, configured to build new window data based on the retained data and the incremental data, where a window size of the new window data is the same as the preset window size;
a replacement module 406, configured to replace the historical window data in the local cache with the new window data.
Optionally, the preset window size is transmitted to the current filter processing unit through the control signal; or, the preset window size is pre-configured in the current filter processing unit.
Optionally, the determining module is configured to determine whether window data needs to be constructed according to a state of a preset flag bit when the preset flag bit is configured.
Optionally, the determining module is configured to determine whether the preset window size is configured; and if the preset window size is configured, judging that window data needs to be constructed.
Optionally, if the current filter processing unit is provided with a corresponding index file, then
The reading module is used for reading the historical window data from the index file;
and the replacing module is used for writing the new window data into the index file to replace the historical window data.
Optionally, the apparatus further comprises:
and the transmission module is used for transmitting the control signal and the processing result of the incremental data to a downstream link when the judgment module judges that the window data does not need to be constructed.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Fig. 5 is a schematic structural diagram of a data processing apparatus 500 according to the present disclosure. For example, the apparatus 500 may be provided as a server. Referring to fig. 5, apparatus 500 includes a processing component 501 that further includes one or more processors and memory resources, represented by memory 502, for storing instructions, such as applications, that are executable by processing component 501. The application programs stored in memory 502 may include one or more modules that each correspond to a set of instructions. Further, the processing component 501 is configured to execute instructions to perform the above-described data processing method.
The apparatus 500 may also include a power component 503 configured to perform power management of the apparatus 500, a wired or wireless network interface 504 configured to connect the apparatus 500 to a network, and an input/output (I/O) interface 505. The apparatus 500 may operate based on an operating system, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like, stored in the memory 502.
The preferred embodiments of the present disclosure are described in detail with reference to the accompanying drawings, however, the present disclosure is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solution of the present disclosure within the technical idea of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.
It should be noted that, in the foregoing embodiments, various features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various combinations that are possible in the present disclosure are not described again.
In addition, any combination of various embodiments of the present disclosure may be made, and the same should be considered as the disclosure of the present disclosure, as long as it does not depart from the spirit of the present disclosure.
Claims (10)
1. A data processing method, comprising:
the method comprises the following steps that a current filter processing unit receives incremental data and control signals sent by an upstream filter processing unit, the control signals comprise starting time corresponding to the incremental data, and a control pipeline used for transmitting the control signals is arranged between adjacent filter processing units;
the current filter processing unit judges whether window data need to be constructed or not;
if window data need to be constructed, the current filter processing unit reads historical window data from a local cache according to the size of a preset window and the starting time;
the current filter processing unit clears the data with the longest time in the historical window data according to the time step corresponding to the incremental data to obtain the reserved data in the historical window data;
and the current filter processing unit constructs new window data based on the retention data and the incremental data, and replaces historical window data in the local cache with the new window data, wherein the window size of the new window data is the same as the size of the preset window.
2. The method of claim 1, wherein the preset window size is communicated to the current filter processing unit via the control signal; or, the preset window size is pre-configured in the current filter processing unit.
3. The method of claim 1, wherein the determining whether window data needs to be constructed by the current filter processing unit comprises:
the current filter processing unit is configured with a preset flag bit, and whether window data need to be constructed is judged according to the state of the preset flag bit;
alternatively, the first and second electrodes may be,
the current filter processing unit judges whether the preset window size is configured or not; and if the preset window size is configured, judging that window data needs to be constructed.
4. The method of claim 1, wherein if the current filter processing unit is provided with a corresponding index file, then
The reading of the historical window data from the local cache comprises: the current filter processing unit reads the historical window data from the index file;
the replacing the historical window data in the local cache with the new window data includes: and the current filter processing unit writes the new window data into the index file to replace the historical window data.
5. The method of any of claims 1 to 4, wherein if the current filter processing unit determines that window data does not need to be constructed, the method further comprises:
and the current filter processing unit transmits the control signal and the processing result of the incremental data to a downstream link.
6. A data processing apparatus, characterized in that the data processing apparatus belongs to a current filter processing unit, the apparatus comprising:
the receiving module is used for receiving incremental data and control signals sent by upstream filter processing units, the control signals comprise starting time corresponding to the incremental data, and a control pipeline used for transmitting the control signals is arranged between adjacent filter processing units;
the judging module is used for judging whether window data need to be constructed or not;
the reading module is used for reading historical window data from a local cache according to the size of a preset window and the starting time when the judging module judges that the window data need to be constructed;
the cleaning module is used for cleaning the data with the longest time in the historical window data according to the time step corresponding to the incremental data to obtain the reserved data in the historical window data;
the construction module is used for constructing new window data based on the reserved data and the incremental data, and the window size of the new window data is the same as the size of the preset window;
and the replacing module is used for replacing the historical window data in the local cache by using the new window data.
7. The apparatus of claim 6,
the judging module is used for judging whether window data need to be constructed or not according to the state of a preset zone bit when the preset zone bit is configured;
alternatively, the first and second electrodes may be,
the judging module is used for judging whether the preset window size is configured or not; and if the preset window size is configured, judging that window data needs to be constructed.
8. The apparatus of claim 6, wherein if the current filter processing unit is provided with a corresponding index file, then
The reading module is used for reading the historical window data from the index file;
and the replacing module is used for writing the new window data into the index file to replace the historical window data.
9. The apparatus of any one of claims 6 to 8, further comprising:
and the transmission module is used for transmitting the control signal and the processing result of the incremental data to a downstream link when the judgment module judges that the window data does not need to be constructed.
10. A data processing apparatus, characterized in that the data processing apparatus belongs to a current filter processing unit, the apparatus comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to:
receiving incremental data and control signals sent by an upstream filter processing unit, wherein the control signals comprise starting time corresponding to the incremental data, and a control pipeline for transmitting the control signals is arranged between adjacent filter processing units;
judging whether window data need to be constructed or not;
if window data need to be constructed, reading historical window data from a local cache according to the size of a preset window and the starting time;
according to the time step corresponding to the incremental data, cleaning the data with the longest time in the historical window data to obtain the retained data in the historical window data;
and constructing new window data based on the reserved data and the incremental data, and replacing historical window data in the local cache with the new window data, wherein the window size of the new window data is the same as the size of the preset window.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710250035.4A CN107145529B (en) | 2017-04-17 | 2017-04-17 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710250035.4A CN107145529B (en) | 2017-04-17 | 2017-04-17 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107145529A CN107145529A (en) | 2017-09-08 |
| CN107145529B true CN107145529B (en) | 2020-04-07 |
Family
ID=59774264
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710250035.4A Active CN107145529B (en) | 2017-04-17 | 2017-04-17 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107145529B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108197162B (en) * | 2017-12-12 | 2020-10-16 | 上海天旦网络科技发展有限公司 | System and method for presenting business transaction situation |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103049519A (en) * | 2012-12-18 | 2013-04-17 | 曙光信息产业(北京)有限公司 | Data uploading method and data uploading device |
| CN103514235A (en) * | 2012-06-30 | 2014-01-15 | 北京百度网讯科技有限公司 | Method and device for establishing incremental code library |
| CN105608204A (en) * | 2015-12-25 | 2016-05-25 | 北京奇虎科技有限公司 | Generation method and device for incremental index list |
-
2017
- 2017-04-17 CN CN201710250035.4A patent/CN107145529B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103514235A (en) * | 2012-06-30 | 2014-01-15 | 北京百度网讯科技有限公司 | Method and device for establishing incremental code library |
| CN103049519A (en) * | 2012-12-18 | 2013-04-17 | 曙光信息产业(北京)有限公司 | Data uploading method and data uploading device |
| CN105608204A (en) * | 2015-12-25 | 2016-05-25 | 北京奇虎科技有限公司 | Generation method and device for incremental index list |
Non-Patent Citations (2)
| Title |
|---|
| 张玉琢、余英."基于软件体系结构的软件重用技术".《云南师范大学学报》.1999,第19卷(第6期),论文第45页-47页. * |
| 赵鹏、孔德华."软件体系结构描述中连接子实现机制的研究".《太远师范专科学校学报》.2002,(第1期),论文第5页-6页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107145529A (en) | 2017-09-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113391973B (en) | Internet of things cloud container log collection method and device | |
| CN111581563A (en) | Page response method and device, storage medium and electronic equipment | |
| CN111966289B (en) | Partition optimization method and system based on Kafka cluster | |
| CN104969222A (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| CN101821717A (en) | Circuit and method with cache coherence stress control | |
| CN113485962B (en) | Log file storage method, device, equipment and storage medium | |
| CN104636286A (en) | Data access method and equipment | |
| CN103336792A (en) | Method and device for data partition | |
| CN114979158A (en) | Resource monitoring method, system, equipment and computer readable storage medium | |
| CN107145529B (en) | Data processing method and device | |
| CN111782473A (en) | Distributed log data processing method, device and system | |
| CN113168314A (en) | Update management device, update management system, and update management method | |
| CN112241474B (en) | Information processing method, apparatus and storage medium | |
| CN109067649B (en) | Node processing method and device, storage medium and electronic equipment | |
| CN113127472B (en) | Method and system for real-time deduplication statistics of number of drivers with large reporting amount | |
| CN112507676B (en) | Method and device for generating energy report, electronic equipment and computer readable medium | |
| CN112486979B (en) | Data processing method, device and system, electronic equipment and computer readable storage medium | |
| CN114090252A (en) | Resource management method and device, storage medium and computer system | |
| CN112749166A (en) | Service data processing method, device, equipment and storage medium | |
| CN103617052A (en) | Method and device for processing cache of application program | |
| CN110908886A (en) | Data sending method and device, electronic equipment and storage medium | |
| CN103856359A (en) | Method and system for obtaining information | |
| CN104536800A (en) | Virtual machine placement method and device | |
| CN111143057B (en) | Heterogeneous cluster data processing method and system based on multiple data centers and electronic equipment | |
| CN117520399A (en) | Data storage method, apparatus, electronic device, and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |