CN115757508A - Streaming data processing method, device, computer equipment, storage medium and product - Google Patents
Streaming data processing method, device, computer equipment, storage medium and product Download PDFInfo
- Publication number
- CN115757508A CN115757508A CN202211639226.7A CN202211639226A CN115757508A CN 115757508 A CN115757508 A CN 115757508A CN 202211639226 A CN202211639226 A CN 202211639226A CN 115757508 A CN115757508 A CN 115757508A
- Authority
- CN
- China
- Prior art keywords
- time window
- resource transfer
- data
- looping
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The application relates to a streaming data processing method, a streaming data processing device, a computer device, a storage medium and a product, wherein streaming data to be processed is obtained, the streaming data is divided according to preset time windows to obtain resource transfer records in each time window, each resource transfer record is analyzed according to node granularity respectively to obtain standard processing data corresponding to each resource transfer record respectively, the standard processing data in each time window is traversed, an out-degree node in a previous time window and an in-degree node in a subsequent time window in continuous time windows are associated in the traversing process until a looping condition is met, a plurality of associated loops are obtained, looping processing can be carried out on the streaming data in real time, and the streaming data association analysis efficiency is improved.
Description
Technical Field
The present application relates to the field of streaming data weaving technologies, and in particular, to a streaming data processing method, apparatus, computer device, storage medium, and product.
Background
Streaming data is a set of sequential, large, fast, continuous arriving data sequences, and in general, streaming data can be viewed as a dynamic collection of data that grows indefinitely over time. The method for weaving the streaming data is generally performed by a way of associating circles, and the associating circles can represent the association relationship between the streaming data.
In a traditional mode, a batch query mode is mostly adopted as a forming mode of the association ring, and the queried data is associated layer by layer and whether the data is looped or not is judged.
However, this approach often requires a large amount of data to be queried, and the association efficiency is low.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a streaming data processing method, apparatus, computer device, storage medium, and product capable of improving streaming data association efficiency.
In a first aspect, the present application provides a streaming data processing method, including:
acquiring streaming data to be processed; the streaming data comprises resource transfer records in a preset time period;
dividing the streaming data according to a preset time window to obtain resource transfer records in each time window;
analyzing each resource transfer record according to the node granularity to obtain standard processing data corresponding to each resource transfer record; the standard processing data comprises an out-degree node and an in-degree node;
and traversing the standard processing data in each time window, associating the out-degree node in the front time window with the in-degree node in the rear time window in the continuous time window in the traversing process, and stopping until the looping condition is met to obtain a plurality of associated loops.
In one embodiment, the step of dividing the streaming data by a preset time window comprises:
acquiring a time label of streaming data; the time labels are used for representing time windows;
the streaming data is partitioned based on the time stamp.
In one embodiment, the resource transfer records include a resource transfer party, a resource receiving party, and a resource transfer value, and the step of analyzing each resource transfer record according to the node granularity to obtain the standard processing data corresponding to each resource transfer record includes:
respectively analyzing the resource transmitting party and the resource receiving party of each resource transfer record according to the node granularity to obtain initial processing data of each resource transfer record; the resource transfer party is a degree output node; the resource receiver is an in-degree node;
processing the initial processing data based on the resource transfer numerical value, and adding a time label to the initial processing data to obtain a resource transfer transverse table in a corresponding time window;
and obtaining standard processing data respectively corresponding to each resource transfer record based on the resource transfer transverse table.
In one embodiment, traversing the standard processing data in each time window, and associating the out-degree node in the previous time window with the in-degree node in the subsequent time window in the consecutive time windows in the traversing process until the looping condition is met, includes the steps of:
traversing the standard processing data in each time window, and associating the out-degree node in the front time window and the in-degree node in the back time window in the continuous time windows in the traversing process;
and if the output node in the last time window is the output node in the foremost time window, determining that the looping condition is met and stopping association.
In one embodiment, after satisfying the looping condition and stopping the association, the method further comprises:
and marking the standard processing data corresponding to each resource transfer record in each time window meeting the looping condition as looping data, and recording the number of looping nodes of each looping data.
In one embodiment, the method further comprises:
receiving a request for querying looping data; the looping data query request carries the number of target looping nodes;
screening target looping data corresponding to the number of looping nodes which is the same as the number of the target looping nodes from the looping data based on the number of the target looping nodes;
and carrying out data analysis on the target looping data to obtain a streaming data processing result.
In a second aspect, the present application further provides a streaming data processing apparatus, including:
the acquisition module is used for acquiring streaming data to be processed; the streaming data comprises resource transfer records in a preset time period;
the dividing module is used for dividing the streaming data according to a preset time window to obtain resource transfer records in each time window;
the analysis module is used for respectively analyzing each resource transfer record according to the node granularity to obtain standard processing data respectively corresponding to each resource transfer record; the standard processing data comprises an out-degree node and an in-degree node;
and the association module is used for traversing the standard processing data in each time window, associating the out-degree node in the previous time window with the in-degree node in the subsequent time window in the continuous time window in the traversing process until the looping condition is met, and stopping to obtain a plurality of associated loops.
In a third aspect, the present application further provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the method steps of any one of the first aspect when executing the computer program.
In a fourth aspect, the present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the method steps of any of the first aspects.
In a fifth aspect, the present application also provides a computer program product comprising a computer program that, when executed by a processor, performs the method steps of any one of the first aspect.
According to the streaming data processing method, the streaming data processing device, the computer equipment, the storage medium and the product, the streaming data to be processed is obtained, the streaming data is divided according to the preset time windows to obtain the resource transfer records in each time window, each resource transfer record is analyzed according to the node granularity to obtain the standard processing data corresponding to each resource transfer record, the standard processing data in each time window is traversed, the out-degree nodes in the previous time window and the in-degree nodes in the next time window in the continuous time windows are associated in the traversing process until the looping condition is met, a plurality of associated loops are obtained, the streaming data can be looped in real time, and the streaming data association analysis efficiency is improved.
Drawings
FIG. 1 is a diagram of an application environment of a streaming data processing method in one embodiment;
FIG. 2 is a flow diagram illustrating a method for processing streaming data in one embodiment;
FIG. 3 is a schematic flow chart of the step of obtaining standard process data in one embodiment;
FIG. 4 is a flow diagram illustrating a method for processing streaming data in one embodiment;
FIG. 5 is a flow diagram illustrating a method for streaming data according to one embodiment;
FIG. 6 is a block diagram showing the structure of a streaming data processing apparatus according to one embodiment;
FIG. 7 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
The streaming data processing method provided by the embodiment of the application can be applied to the application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be placed on the cloud or other network server. The server 104 stores streaming data obtained in real time, and the terminal 102 obtains streaming data to be processed from the server 104 and divides the streaming data according to a preset time window. The terminal 102 is further configured to analyze each resource transfer record according to the node granularity, obtain standard processing data corresponding to each resource transfer record, traverse the standard processing data in each time window, associate an out-degree node in a previous time window with an in-degree node in a subsequent time window in consecutive time windows in the traversal process, and stop until a looping condition is met, so as to obtain a plurality of associated loops. The terminal 102 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and the like. The server 104 may be implemented as a stand-alone server or as a server cluster comprised of multiple servers.
In one embodiment, as shown in fig. 2, a streaming data processing method is provided, which is described by taking the method as an example applied to the terminal 102 in fig. 1, and includes the following steps:
s202: acquiring streaming data to be processed; the streaming data includes resource transfer records for a preset period of time.
The streaming data to be processed is streaming data acquired by a terminal in real time, and includes resource transfer records in a preset time period, where the preset time period may be set according to actual application requirements, and the resource transfer records include resource transfer records in the preset time period, where the resource transfer records include a resource transfer party, a resource receiving party, and a resource transfer value, for example, "a resource transfer party a transfers a certain amount of resources to a resource receiving party B," and the resource transfer records are one resource transfer record, and in the preset time period, the resource receiving party B may transfer a certain amount of resources to a resource receiving party C as the resource transfer party, and a plurality of resource transfer records may be obtained, and all the resource transfer records in the preset time period are acquired by the terminal.
S204: and dividing the streaming data according to a preset time window to obtain resource transfer records in each time window.
The preset time window can select different time periods according to actual application requirements, the terminal divides streaming data according to the preset time window, a time label corresponding to the preset time window is added in the resource transfer record, taking "the resource transfer party A transfers a certain amount of resources to the resource receiver B" as an example, assuming that the preset time window is T, the resource transfer record is "the resource transfer party A transfers a certain amount of resources to the resource receiver B within the T time window", and so on, the terminal divides each resource transfer record to obtain the resource transfer record within each time window.
S206: analyzing each resource transfer record according to the node granularity to obtain standard processing data corresponding to each resource transfer record; the standard processing data comprises an out-degree node and an in-degree node.
Wherein, the node refers to a resource forwarding party and a resource receiving party in the resource transfer record, and the granularity refers to a drill-down level of the node, that is, a level of increasing dimension, for example, a transfers a certain amount of resources to B, and then B transfers a certain amount of resources to C, and the granularity of the node of a is drill-down 2 levels. And simultaneously, the terminal collects all resources transferred between each resource transfer party and the corresponding resource receiving party in a preset time window, analyzes the output information of the output node and the input information of the input node and obtains the final standard processing data. The out-degree node is a resource roll-out party, the in-degree node is a resource receiving party, and taking "the resource roll-out party a transfers a certain amount of resources to the resource receiving party B within a T time window" as an example, the standard processing data is expressed as "the resource roll-out party a transfers a certain amount of resources to the resource receiving party B within a T time window," the out-degree node a drills down with a degree of 1, the out-degree is 1, and the in-degree node B inputs with a degree of 1".
S208: and traversing the standard processing data in each time window, associating the out-degree node in the front time window with the in-degree node in the rear time window in the continuous time window in the traversing process, and stopping until the looping condition is met to obtain a plurality of associated loops.
The terminal traverses standard processing data in each time window, associates a degree-out node in a previous time window with a degree-in node in a subsequent time window in consecutive time windows, specifically, the terminal acquires a degree-in node corresponding to the degree-out node in the previous time window, then the terminal acquires a degree-out node in the subsequent time window, which is the same as the degree-in node, and associates the degree-in node corresponding to the degree-out node with the degree-out node in the previous time window. Streaming data in two continuous time windows comprises that a certain amount of resources are transferred from a resource transmitting party A to a resource receiving party B in a T time window, the drilling depth of a node A at the output degree is 1, and the input degree of a node B at the input degree is 1; transferring a certain amount of resources to a resource receiver C by a resource transfer party B in a T1 time window, taking the drilling degree of a departure node B as 1, the departure as 1 and the entrance degree of an entrance node C as 1 as examples, acquiring an entrance node B corresponding to the departure node A in the T time window by a terminal, then acquiring a corresponding departure node B in the T1 time window, associating the entrance node C corresponding to the departure node B with the departure node A in the T time window to obtain associated data A-B-C, and stopping until a looping condition is met to obtain a plurality of associated loops.
According to the streaming data processing method, streaming data to be processed are obtained, the streaming data are divided according to preset time windows to obtain resource transfer records in each time window, each resource transfer record is analyzed according to node granularity, standard processing data corresponding to each resource transfer record are obtained, the standard processing data in each time window are traversed, an out-degree node in a previous time window and an in-degree node in a subsequent time window in continuous time windows are associated in the traversing process until a looping condition is met, a plurality of associated loops are obtained, looping processing can be performed on the streaming data in real time, and the streaming data association analysis efficiency is improved.
In one embodiment, the step of partitioning the streaming data by a preset time window comprises: acquiring a time label of streaming data; the time labels are used for representing time windows; the streaming data is partitioned based on the time stamp.
The time labels are transfer time of each resource transfer record, the terminal obtains the time labels of the corresponding time windows according to the time period of the preset time windows, and the streaming data are divided according to the time labels.
In this embodiment, the accuracy of data partitioning can be ensured by obtaining the time tag of the streaming data and partitioning the streaming data based on the time tag.
In one embodiment, as shown in fig. 3, the resource transfer records include a resource transfer party, a resource receiving party, and a resource transfer value, and the step of analyzing each resource transfer record according to the node granularity respectively to obtain the standard processing data corresponding to each resource transfer record includes:
s302: respectively analyzing the resource transmitting party and the resource receiving party of each resource transfer record according to the node granularity to obtain initial processing data of each resource transfer record; the resource transfer party is an out-degree node; the resource receiver is an in-degree node.
The node granularity represents the drilling-down level of the node, and the initial processing data comprises a time window, the drilling-down level, the output information, the input information and the total resource transfer amount. And simultaneously, the terminal collects all resources transferred between each resource transfer party and the corresponding resource receiving party in a preset time window, analyzes the output information of the output node and the input information of the input node and obtains the final standard processing data.
S304: and processing the initial processing data based on the resource transfer numerical value, and adding a time label to the initial processing data to obtain a resource transfer transverse table in the corresponding time window.
The resource transfer transverse table is a transverse resource information table, and takes the example that standard processing data indicates that "resource forwarding party a transfers a certain amount of resources to resource receiving party B within T time window, the out-degree node a has a drilling depth of 1, and the in-degree node B has an in-degree of 1", and the corresponding resource transfer transverse table can be represented as follows: (out-degree node: A, drill down 1 layer, out-degree information: 1, time stamp: time window T, in-degree node: B, in-degree information: 1, total amount of resource transfer: 10).
S306: and obtaining standard processing data respectively corresponding to each resource transfer record based on the resource transfer transverse table.
The terminal collects all resource transfer record tables in the time window to obtain corresponding resource transfer transverse tables, and obtains standard processing data corresponding to all resource transfer records respectively based on the resource transfer transverse tables.
In this embodiment, the resource forwarding part and the resource receiving part of each resource transfer record are respectively analyzed according to the node granularity to obtain the initial processing data of each resource transfer record, then the initial processing data is processed based on the resource transfer numerical value, a time tag is added to the initial processing data to obtain a resource transfer transverse table in a corresponding time window, and finally, the standard processing data respectively corresponding to each resource transfer record is obtained based on the resource transfer transverse table, so that each standard processing data can be accurately obtained.
In one embodiment, traversing the standard processing data in each time window, and associating the out-degree node in the previous time window with the in-degree node in the subsequent time window in the consecutive time windows in the traversing process until the looping condition is met, comprises the following steps: traversing the standard processing data in each time window, and associating the out-degree node in the front time window and the in-degree node in the back time window in the continuous time windows in the traversing process; and if the out-degree node in the last time window is the out-degree node in the foremost time window, determining that the looping condition is met and stopping association.
The terminal traverses the standard processing data in each time window, associates the out-degree node in the front time window with the in-degree node in the rear time window in the continuous time window in the traversing process, the in-degree node in the rear time window is actually the out-degree node in the rear time window, and if the final out-degree node in the last time window is the initial in-degree node of the front window, the looping condition is met, and the association is stopped. datse:Sub>A were processed with the standard "(out-degree node: A, drilling down se:Sub>A layer 1, out-degree information 1, se:Sub>A time label, se:Sub>A time window T, an in-degree node B, in-degree information 1, se:Sub>A resource transfer total amount 10)," (out-degree node B, drilling down se:Sub>A layer 1, out-degree information 1, se:Sub>A time label, se:Sub>A time window T1, an in-degree node C, in-degree information 1, se:Sub>A resource transfer total amount 20), "(out-degree node C, drilling down se:Sub>A layer 1, out-degree information 1, se:Sub>A time label, se:Sub>A time window T2, an in-degree node A, in-degree information 1, and se:Sub>A resource transfer total amount 30)" for example, the terminal associates the out-degree node A in the time window T with the in-degree node C in the time window T1 to obtain se:Sub>A piece of associated datse:Sub>A A-B-C, then associates the associated datse:Sub>A with the in-degree node C in the time window T2 to obtain associated datse:Sub>A A-B-C-A, and at this time, the final in-degree node in the time window T2 is the node A, and the initial out-degree node in the time window T meets the association condition and then the association is determined.
In this embodiment, standard processing data in each time window is traversed, and a degree-out node in a previous time window and a degree-in node in a subsequent time window in consecutive time windows are associated in the traversing process, and if the degree-out node in the last time window is the degree-out node in the foremost time window, it is determined that a looping condition is met and association is stopped, looping processing can be performed on streaming data, and streaming data association analysis efficiency is improved.
In one embodiment, after satisfying the looping condition and stopping the association, the method further comprises: and marking the standard processing data corresponding to each resource transfer record in each time window meeting the looping condition as looping data, and recording the number of looping nodes of each looping data.
The terminal marks the associated datse:Sub>A meeting the looping condition as looping datse:Sub>A, the number of looping nodes indicates how many nodes the looping datse:Sub>A passes through to loop, and taking the looping datse:Sub>A A-B-C-A as an example, the number of the looping nodes of the looping datse:Sub>A is 3.
In this embodiment, the standard processing data corresponding to each resource transfer record in each time window that meets the looping condition is marked as looping data, and the number of looping nodes of each looping data is recorded, so that the number of looping nodes of each looping data can be accurately obtained.
In one embodiment, as shown in fig. 4, the method further comprises:
s402: receiving a request for querying looping data; the looping data query request carries the number of target looping nodes.
Wherein the target number of looping nodes represents looping data to be queried for the target number of looping nodes to be looped.
S404: and screening out target looping data corresponding to the number of looping nodes which is the same as the number of the target looping nodes from the looping data based on the number of the target looping nodes.
In practical application, the number of the looping nodes can be obtained by inquiring se:Sub>A drill-down level in the looping datse:Sub>A, and if the looping datse:Sub>A with 3 points to be looped is to be inquired, the looping datse:Sub>A with se:Sub>A drill-down level of 2, such as the looping datse:Sub>A A-B-C-A, can be directly inquired.
S406: and carrying out data analysis on the target looping data to obtain a streaming data processing result.
The terminal analyzes the inquired target looping data, acquires the associated information in the looping data and obtains a corresponding streaming data processing result.
In this embodiment, by receiving the request for query of the looping data, and screening the number of the looping data from the number of the target looping nodes to obtain the target looping data corresponding to the number of the looping nodes that is the same as the number of the target looping nodes, the data analysis is performed on the target looping data to obtain a streaming data processing result, so that the streaming data analysis efficiency can be improved.
In one embodiment, as shown in fig. 5, there is provided a streaming data weaving method, including the steps of:
s1: analyzing the collected real-time streaming data: and the terminal divides the streaming data according to a preset time window, and analyzes each resource transfer record in the streaming data according to the node granularity to obtain standard processing data corresponding to each resource transfer record. And repeating the data analysis in the next time window until the data analysis of a plurality of time windows of the actual demand is completed.
S2: and the terminal disassociates the out-degree node in the time window 1 with the in-degree node in the time window 2, if the out-degree node in the time window 2 can not be associated with the out-degree node in the time window 1, the out-degree node is kept as it is, and if the final out-degree node is the initial in-degree node, the out-degree node is marked to be a circle.
S3: and the terminal associates the out-degree node of the time window 2 with the in-degree node in the window 3, adds 1 to the drill-down hierarchy if the out-degree node can be associated, keeps the original state if the out-degree node cannot be associated, marks the ring if the final drill-down node is the initial in-degree node, and repeats the association process until all the time windows are associated.
S4: and (3) looping data query: and the terminal receives the request for inquiring the looping data, and if the looping data with the number of the target looping nodes needs to be inquired, the data with the drill-down level of one less than the number of the target looping nodes in the looping data only needs to be inquired.
In the embodiment, in the judgment of the association circle, aggregation is performed in the window according to the granularity of the circle-forming node, and the horizontal table retention is performed on the recursive data, so that the real-time recursive judgment can be performed on the streaming data of the latest time window, and the timeliness of the association of the streaming data is improved.
It should be understood that, although the steps in the flowcharts related to the above embodiments are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders. Moreover, at least a part of the steps in the flowcharts related to the above embodiments may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.
Based on the same inventive concept, the embodiment of the present application further provides a streaming data processing apparatus for implementing the above-mentioned streaming data processing method. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the method, so specific limitations in one or more embodiments of the streaming data processing apparatus provided below may refer to the limitations on the streaming data processing method in the foregoing, and details are not described here again.
In one embodiment, as shown in fig. 6, there is provided a streaming data processing apparatus including: an obtaining module 10, a dividing module 20, an analyzing module 30 and an associating module 40, wherein:
an obtaining module 10, configured to obtain streaming data to be processed; the streaming data includes resource transfer records for a preset period of time.
The dividing module 20 is configured to divide the streaming data according to preset time windows to obtain resource transfer records in each time window.
The analysis module 30 is configured to analyze each resource transfer record according to the node granularity, and obtain standard processing data corresponding to each resource transfer record; the standard processing data comprises an out-degree node and an in-degree node.
And the association module 40 is configured to traverse the standard processing data in each time window, associate a degree-out node in a previous time window with a degree-in node in a subsequent time window in consecutive time windows in the traversal process, and stop until a looping condition is met to obtain a plurality of associated loops.
In one embodiment, the partitioning module 20 includes: a tag acquisition unit and a data dividing unit, wherein:
a tag acquisition unit configured to acquire a time tag of streaming data; time labels are used to indicate the time windows.
And the data dividing unit is used for dividing the streaming data based on the time labels.
In one embodiment, the resource transfer record includes a resource transfer party, a resource receiving party, and a resource transfer value, and the analysis module 30 includes: node analysis unit, data processing unit and data acquisition unit, wherein:
the node analysis unit is used for respectively analyzing the resource transmitting party and the resource receiving party of each resource transfer record according to the node granularity to obtain initial processing data of each resource transfer record; the resource transfer party is a degree output node; the resource receiver is an in-degree node.
And the data processing unit is used for processing the initial processing data based on the resource transfer numerical value and adding a time label to the initial processing data to obtain a resource transfer transverse table in the corresponding time window.
And the data acquisition unit is used for acquiring standard processing data respectively corresponding to each resource transfer record based on the resource transfer transverse table.
In one embodiment, the association module 40 is further configured to associate an out-degree node in a previous time window with an in-degree node in a subsequent time window; and if the output node in the last time window is the output node in the foremost time window, determining that the looping condition is met and stopping association.
In one embodiment, after the looping condition is satisfied and the association is stopped, the association module 40 is further configured to mark the standard processing data corresponding to each resource transfer record in each time window that satisfies the looping condition as looping data, and record the number of looping nodes of each looping data.
In one embodiment, association module 40 is further configured to: receiving a turn data query request; the looping data query request carries the number of target looping nodes; screening target looping data corresponding to the number of looping nodes which is the same as the number of the target looping nodes from the looping data based on the number of the target looping nodes; and carrying out data analysis on the target looping data to obtain a streaming data processing result.
The various modules in the streaming data processing apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as in fig. 7. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program to run on the non-volatile storage medium. The communication interface of the computer device is used for communicating with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a streaming data processing method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on a shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the configuration shown in fig. 7 is a block diagram of only a portion of the configuration associated with the present application, and does not constitute a limitation on the computing devices to which the present application may be applied, and that a particular computing device may include more or fewer components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program: acquiring streaming data to be processed; the streaming data comprises resource transfer records in a preset time period; dividing the streaming data according to a preset time window to obtain resource transfer records in each time window; analyzing each resource transfer record according to the node granularity to obtain standard processing data corresponding to each resource transfer record; the standard processing data comprises an out-degree node and an in-degree node; and traversing the standard processing data in each time window, and associating the out-degree node in the previous time window with the in-degree node in the subsequent time window in the continuous time window in the traversing process until a looping condition is met, so as to obtain a plurality of associated loops.
In one embodiment, the partitioning of streaming data by preset time windows involved in the execution of the computer program by the processor comprises: acquiring a time label of streaming data; the time labels are used for representing time windows; the streaming data is partitioned based on the time tags.
In one embodiment, the resource transfer records include a resource transfer party, a resource receiving party, and a resource transfer value, and the analyzing of each resource transfer record according to the node granularity, which is involved when the processor executes the computer program, obtains standard processing data corresponding to each resource transfer record, respectively, includes: respectively analyzing the resource transmitting party and the resource receiving party of each resource transfer record according to the node granularity to obtain initial processing data of each resource transfer record; the resource transfer party is an out-degree node; the resource receiver is an access node; processing the initial processing data based on the resource transfer value, and adding a time label to the initial processing data to obtain a resource transfer transverse table in a corresponding time window; and obtaining standard processing data respectively corresponding to each resource transfer record based on the resource transfer transverse table.
In one embodiment, the traversing the standard machining data in each time window involved in the processor executing the computer program, the associating an out-degree node in a preceding time window with an in-degree node in a succeeding time window in successive time windows during the traversing until a looping condition is satisfied, includes: traversing the standard processing data in each time window, and associating the out-degree node in the previous time window with the in-degree node in the subsequent time window in the continuous time window in the traversing process; and if the out-degree node in the last time window is the out-degree node in the foremost time window, determining that the looping condition is met and stopping association.
In one embodiment, after satisfying the looping condition and stopping the association, the processor when executing the computer program further performs the steps of: and marking the standard processing data corresponding to each resource transfer record in each time window meeting the looping condition as looping data, and recording the number of looping nodes of each looping data.
In one embodiment, the processor when executing the computer program further performs the steps of: receiving a request for querying looping data; the looping data query request carries the number of target looping nodes; screening target looping data corresponding to the number of looping nodes which is the same as the number of the target looping nodes from the looping data based on the number of the target looping nodes; and carrying out data analysis on the target looping data to obtain a streaming data processing result.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, performs the steps of: acquiring streaming data to be processed; the streaming data comprises resource transfer records in a preset time period; the streaming data are divided according to preset time windows to obtain resource transfer records in each time window; analyzing each resource transfer record according to the node granularity to obtain standard processing data corresponding to each resource transfer record; the standard processing data comprises an out-degree node and an in-degree node; and traversing the standard processing data in each time window, and associating the out-degree node in the previous time window with the in-degree node in the subsequent time window in the continuous time window in the traversing process until a looping condition is met, so as to obtain a plurality of associated loops.
In one embodiment, the computer program, when executed by the processor, relates to partitioning streaming data by a preset time window, comprising: acquiring a time label of streaming data; the time labels are used for representing time windows; the streaming data is partitioned based on the time tags.
In one embodiment, the resource transfer records include a resource transfer party, a resource receiving party, and a resource transfer value, and when the computer program is executed by the processor, the computer program analyzes each resource transfer record according to the node granularity, and obtains standard processing data corresponding to each resource transfer record, respectively, where the method includes: respectively analyzing the resource transmitting party and the resource receiving party of each resource transfer record according to the node granularity to obtain initial processing data of each resource transfer record; the resource transfer party is an out-degree node; the resource receiver is an in-degree node; processing the initial processing data based on the resource transfer numerical value, and adding a time label to the initial processing data to obtain a resource transfer transverse table in a corresponding time window; and obtaining standard processing data respectively corresponding to each resource transfer record based on the resource transfer transverse table.
In one embodiment, the computer program, when executed by the processor, involves traversing the standard machining data within each time window, associating a departure node within a preceding time window with an arrival node within a succeeding time window in successive time windows during the traversing, until a looping condition is satisfied, comprising: traversing the standard processing data in each time window, and associating the out-degree node in the previous time window with the in-degree node in the subsequent time window in the continuous time window in the traversing process; and if the output node in the last time window is the output node in the foremost time window, determining that the looping condition is met and stopping association.
In one embodiment, the computer program when executed by the processor further realizes the following steps after the looping condition is fulfilled and the association is stopped: and marking the standard processing data corresponding to each resource transfer record in each time window meeting the looping condition as looping data, and recording the number of looping nodes of each looping data.
In one embodiment, the computer program when executed by the processor further performs the steps of: receiving a request for querying looping data; the looping data query request carries the number of target looping nodes; screening target looping data corresponding to the number of looping nodes which is the same as the number of the target looping nodes from the looping data based on the number of the target looping nodes; and carrying out data analysis on the target looping data to obtain a streaming data processing result.
In one embodiment, a computer program product is provided, comprising a computer program which when executed by a processor performs the steps of: acquiring streaming data to be processed; the streaming data comprises resource transfer records in a preset time period; dividing the streaming data according to a preset time window to obtain resource transfer records in each time window; analyzing each resource transfer record according to the node granularity to obtain standard processing data corresponding to each resource transfer record; the standard processing data comprises an out-degree node and an in-degree node; and traversing the standard processing data in each time window, and associating the out-degree node in the previous time window with the in-degree node in the subsequent time window in the continuous time window in the traversing process until a looping condition is met, so as to obtain a plurality of associated loops.
In one embodiment, the computer program, when executed by the processor, relates to partitioning streaming data by a preset time window, comprising: acquiring a time label of streaming data; the time labels are used for representing time windows; the streaming data is partitioned based on the time stamp.
In one embodiment, the resource transfer records include a resource transfer party, a resource receiving party, and a resource transfer value, and the analysis of each resource transfer record according to the node granularity when the computer program is executed by the processor is performed to obtain the standard processing data corresponding to each resource transfer record, respectively includes: respectively analyzing the resource transmitting party and the resource receiving party of each resource transfer record according to the node granularity to obtain initial processing data of each resource transfer record; the resource transfer party is an out-degree node; the resource receiver is an access node; processing the initial processing data based on the resource transfer value, and adding a time label to the initial processing data to obtain a resource transfer transverse table in a corresponding time window; and obtaining standard processing data respectively corresponding to each resource transfer record based on the resource transfer transverse table.
In one embodiment, the computer program, when executed by the processor, involves traversing the standard machining data within each time window, associating a departure node within a preceding time window with an arrival node within a succeeding time window in successive time windows during the traversing, until a looping condition is satisfied, comprising: traversing the standard processing data in each time window, and associating the out-degree node in the previous time window with the in-degree node in the subsequent time window in the continuous time window in the traversing process; and if the out-degree node in the last time window is the out-degree node in the foremost time window, determining that the looping condition is met and stopping association.
In one embodiment, the computer program when executed by the processor further realizes the following steps after the looping condition is fulfilled and the association is stopped: and marking the standard processing data corresponding to each resource transfer record in each time window meeting the looping condition as looping data, and recording the number of looping nodes of each looping data.
In one embodiment, the computer program when executed by the processor further performs the steps of: receiving a request for querying looping data; the looping data query request carries the number of target looping nodes; screening target looping data corresponding to the number of looping nodes which is the same as the number of the target looping nodes from the looping data based on the number of the target looping nodes; and carrying out data analysis on the target looping data to obtain a streaming data processing result.
It should be noted that the resource transfer records (including but not limited to the resource transferring party, the resource receiving party, the resource transfer value, etc.) and the data (including but not limited to the data for analysis, the stored data, the displayed data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), magnetic Random Access Memory (MRAM), ferroelectric Random Access Memory (FRAM), phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.
Claims (10)
1. A method of streaming data processing, the method comprising:
acquiring streaming data to be processed; the streaming data comprises resource transfer records in a preset time period;
dividing the streaming data according to preset time windows to obtain resource transfer records in each time window;
analyzing each resource transfer record according to the node granularity to obtain standard processing data corresponding to each resource transfer record; the standard processing data comprises an out-degree node and an in-degree node;
and traversing the standard processing data in each time window, associating the out-degree node in the front time window with the in-degree node in the rear time window in the continuous time window in the traversing process, and stopping until the looping condition is met to obtain a plurality of associated loops.
2. The method of claim 1, wherein the partitioning the streaming data by a preset time window comprises:
acquiring a time label of the streaming data; the time labels are used for representing time windows;
the streaming data is partitioned based on the time tags.
3. The method of claim 2, wherein the resource transfer records include a resource transfer party, a resource receiving party, and a resource transfer value, and the analyzing each resource transfer record according to the node granularity to obtain the standard processing data corresponding to each resource transfer record respectively includes:
respectively analyzing a resource transmitting party and a resource receiving party of each resource transfer record according to the node granularity to obtain initial processing data of each resource transfer record; the resource transfer party is a degree output node; the resource receiver is an in-degree node;
processing the initial processing data based on the resource transfer numerical value, and adding the time label to the initial processing data to obtain a resource transfer transverse table in a corresponding time window;
and obtaining standard processing data respectively corresponding to each resource transfer record based on the resource transfer transverse table.
4. The method of claim 1, wherein traversing the standard machining data in each time window, associating an out-degree node in a preceding time window with an in-degree node in a succeeding time window in successive time windows during the traversing until a looping condition is met, comprises:
traversing the standard processing data in each time window, and associating the out-degree node in the front time window and the in-degree node in the back time window in the continuous time windows in the traversing process;
and if the in-degree node in the last time window is the out-degree node in the foremost time window, determining that the looping condition is met and stopping association.
5. The method of claim 4, wherein after the looping condition is satisfied and association is stopped, the method further comprises:
and marking the standard processing data corresponding to each resource transfer record in each time window meeting the looping condition as looping data, and recording the number of looping nodes of each looping data.
6. The method of claim 5, further comprising:
receiving a request for querying looping data; the looping data query request carries the number of target looping nodes;
screening out target looping data corresponding to the number of looping nodes which is the same as the number of the target looping nodes from the looping data based on the number of the target looping nodes;
and carrying out data analysis on the target looping data to obtain a streaming data processing result.
7. A streaming data processing apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring streaming data to be processed; the streaming data comprises resource transfer records in a preset time period;
the dividing module is used for dividing the streaming data according to a preset time window to obtain resource transfer records in each time window;
the analysis module is used for respectively analyzing each resource transfer record according to the node granularity to obtain standard processing data respectively corresponding to each resource transfer record; the standard processing data comprises an out-degree node and an in-degree node;
and the association module is used for traversing the standard processing data in each time window, associating the out-degree node in the previous time window with the in-degree node in the subsequent time window in the continuous time window in the traversing process, and stopping until the looping condition is met to obtain a plurality of associated loops.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 6.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 6 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211639226.7A CN115757508A (en) | 2022-12-20 | 2022-12-20 | Streaming data processing method, device, computer equipment, storage medium and product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211639226.7A CN115757508A (en) | 2022-12-20 | 2022-12-20 | Streaming data processing method, device, computer equipment, storage medium and product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115757508A true CN115757508A (en) | 2023-03-07 |
Family
ID=85346819
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211639226.7A Pending CN115757508A (en) | 2022-12-20 | 2022-12-20 | Streaming data processing method, device, computer equipment, storage medium and product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115757508A (en) |
-
2022
- 2022-12-20 CN CN202211639226.7A patent/CN115757508A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106980669B (en) | A kind of storage of data, acquisition methods and device | |
| CN102915347B (en) | A kind of distributed traffic clustering method and system | |
| CN107590250A (en) | A kind of space-time orbit generation method and device | |
| CN106909372B (en) | Method and system for calculating purchase path of mobile terminal user | |
| CN111061758B (en) | Data storage method, device and storage medium | |
| CN112818013B (en) | Time sequence database query optimization method, device, equipment and storage medium | |
| CN108062384A (en) | The method and apparatus of data retrieval | |
| CN111950857A (en) | Index system management method and device based on service indexes and electronic equipment | |
| CN110851663B (en) | Method and device for managing metadata | |
| CN114138907A (en) | Data processing method, computer device, storage medium, and computer program product | |
| CN106776810B (en) | Big data processing system and method | |
| CN115757508A (en) | Streaming data processing method, device, computer equipment, storage medium and product | |
| CN107798009A (en) | Data aggregation method, apparatus and system | |
| CN105975634B (en) | The storage method of multidimensional ordered data in distributed data-storage system | |
| CN114745436A (en) | Data acquisition method and device, computer equipment and storage medium | |
| CN115495452A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN112685557A (en) | Visualized information resource management method and device | |
| CN112685329A (en) | Method for processing data and related device | |
| CN114676675A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN116243855A (en) | Index establishment and data access method and device and electronic equipment | |
| CN116882648A (en) | Account resource allocation method, device, computer equipment and storage medium | |
| CN117648336A (en) | Data query method, device, computer equipment and storage medium | |
| CN117453957A (en) | Resource processing method, device, computer equipment and storage medium | |
| CN116401292A (en) | Data ordering method, device, computer equipment and storage medium | |
| CN116880927A (en) | Rule management method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |