CN110007967A - A kind of data processing method based on streaming frame, device and equipment - Google Patents

A kind of data processing method based on streaming frame, device and equipment Download PDF

Info

Publication number
CN110007967A
CN110007967A CN201711470644.7A CN201711470644A CN110007967A CN 110007967 A CN110007967 A CN 110007967A CN 201711470644 A CN201711470644 A CN 201711470644A CN 110007967 A CN110007967 A CN 110007967A
Authority
CN
China
Prior art keywords
analytical unit
unit
output
analysis
analytical
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201711470644.7A
Other languages
Chinese (zh)
Other versions
CN110007967B (en
Inventor
周胜凯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Hikvision Digital Technology Co Ltd
Original Assignee
Hangzhou Hikvision Digital Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Hikvision Digital Technology Co Ltd filed Critical Hangzhou Hikvision Digital Technology Co Ltd
Priority to CN201711470644.7A priority Critical patent/CN110007967B/en
Publication of CN110007967A publication Critical patent/CN110007967A/en
Application granted granted Critical
Publication of CN110007967B publication Critical patent/CN110007967B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Abstract

The embodiment of the invention provides a kind of data processing method based on streaming frame, device and equipment, method includes: to obtain the corresponding task parameters of pending data；According to task parameters, the analysis sequence of the analytical unit and analytical unit that are handled pending data is determined；Identified analytical unit is loaded；According to analysis sequence, each analysis unit loaded is attached, obtains analytical unit combination；By pending data input analytical unit combination, the thread for including using each analysis unit analyzes pending data；According to the output content that analytical unit combines, processing result is obtained.It can be seen that, in this programme, after getting task parameters, reload analytical unit, analytical unit is not the continuous service in a manner of process, and after load analytical unit, is analyzed using the thread that analytical unit includes data, compared to the scheme of analytical unit continuous service in a manner of process, reduce resources occupation rate.

Description

A kind of data processing method based on streaming frame, device and equipment
Technical field
The present invention relates to technical field of data processing, more particularly to a kind of data processing method based on streaming frame, Device and equipment.
Background technique
In the biggish scene of some data processing amounts, commonly used streaming frame carries out data processing.Streaming frame is such as Storm can be applied in the various scenes such as real-time analysis, online machine learning, lasting calculating, distributed remote calling Storm carries out data processing.
But Storm is applied to carry out in data handling procedure, each analytical unit continuous service in a manner of process occupies Vast resources.
Summary of the invention
The embodiment of the present invention is designed to provide a kind of data processing method based on streaming frame, device and equipment, To reduce resources occupation rate.
In order to achieve the above objectives, the embodiment of the invention provides a kind of data processing methods based on streaming frame, comprising:
Obtain the corresponding task parameters of pending data；
According to the task parameters, the analytical unit handled the pending data and the analysis are determined The analysis sequence of unit；Wherein, each analysis unit includes an independent thread；
Identified analytical unit is loaded；And according to the analysis sequence, each analysis unit that will be loaded It is attached, obtains analytical unit combination；
The pending data is obtained, and the pending data is inputted into the analytical unit and is combined, utilizes described point The thread that each analysis unit includes in analysis unit combination, analyzes the pending data；
According to the output content that the analytical unit combines, processing result is obtained.
Optionally, the corresponding task parameters of the acquisition pending data may include:
Obtain corresponding first configuration file of pending data；
It is described according to the task parameters, determine the analytical unit handled the pending data and described The analysis sequence of analytical unit may include:
According to the mark for the analytical unit for including in first configuration file and mark sequence, determine to described to be processed The analysis sequence of analytical unit and the analytical unit that data are handled.
It optionally, can also include: to be generated pre- for the pending data after obtaining the pending data If the structural body object of structure；
It is described the pending data is inputted into the analytical unit to combine, using each dividing in analytical unit combination The thread that analysis unit includes, analyzes the pending data, comprising:
The structural body object is inputted into first analytical unit in the analytical unit combination, utilizes described first The thread that analytical unit includes analyzes the structural body object, is analyzed as a result, analysis result is added to described Structural body object obtains the output object of first analytical unit；
According to the analysis sequence, successively each analysis unit after first analytical unit is determined as currently Analytical unit；The output object of a upper analytical unit for present analysis unit is inputted into present analysis unit, utilizes current point The thread that analysis unit includes, analyzes the output object of a upper analytical unit, is analyzed as a result, analysis is tied Fruit is added to the output object of a upper analytical unit, obtains the output object of present analysis unit；
The output content combined according to the analytical unit, obtains processing result, may include:
According to the output object of the last one analytical unit in analytical unit combination, processing result is obtained.
Optionally, after the output object for obtaining first analytical unit, can also include:
The output object of first analytical unit is added in the output queue of first analytical unit；
The output object of a upper analytical unit by present analysis unit inputs present analysis unit, comprising:
From the input rank of present analysis unit, according to default queue sequence, it is single to be successively read a upper analysis The output object of member；
After the output object for obtaining present analysis unit, further includes:
The output object of active cell is added in the output queue of present analysis unit.
Optionally, the output object according to the last one analytical unit in analytical unit combination, acquisition processing As a result, may include:
Judge whether the output object of the last one analytical unit in the analytical unit combination meets output condition；
If it is satisfied, obtaining processing result from the output object of the last one analytical unit；
If conditions are not met, returning to first analysis for executing and inputting the structural body object in the analytical unit combination The step of unit.
It optionally, can also include: to obtain pending data corresponding second to match before obtaining the pending data Set file；
The mark for reading the interaction accessory library for including in second configuration file, interaction corresponding to read mark Accessory inventory loads, and includes input interface, output interface, assembling fittings and detection accessory in the interactive accessory library；
After obtaining the pending data, can also include:
The input interface is called, the pending data is input to the interactive accessory library；
It is described to be directed to the pending data, the structural body object of preset structure is generated, may include:
Using the assembling fittings, the pending data is assembled, generates the structural body object of preset structure；
Whether the output object for judging the last one analytical unit in the analytical unit combination meets output condition, May include:
The detection accessory is called, the output object of the last one analytical unit in analytical unit combination is examined It surveys, judges whether testing result meets output condition；
After obtaining processing result in the output object of the last one analytical unit, can also include:
The output interface is called, the processing result is exported.
Optionally, further include deleting accessory in the interactive accessory library, calling the output interface, exporting the processing As a result after, can also include:
The deletion accessory is called, the output object of the last one analytical unit is deleted.
Optionally, can also include: after obtaining processing result in the output content combined according to the analytical unit Unload the analytical unit in the analytical unit combination.
In order to achieve the above objectives, the embodiment of the invention also provides a kind of data processing equipment based on streaming frame, packets It includes:
First obtains module, for obtaining the corresponding task parameters of pending data；
Determining module is used for according to the task parameters, the determining analytical unit that the pending data is handled, And the analysis sequence of the analytical unit；Wherein, each analysis unit includes an independent thread；
First loading module, for being loaded to identified analytical unit；
Link block, for each analysis unit loaded being attached, is analyzed according to the analysis sequence Unit combination；
Second obtains module, for obtaining the pending data；
Analysis module combines for the pending data to be inputted the analytical unit, utilizes the analytical unit group The thread that each analysis unit includes in conjunction analyzes the pending data；
Third obtains module, and the output content for being combined according to the analytical unit obtains processing result.
Optionally, described first module is obtained, specifically can be used for: obtaining the corresponding first configuration text of pending data Part；
The determining module, specifically can be used for:
According to the mark for the analytical unit for including in first configuration file and mark sequence, determine to described to be processed The analysis sequence of analytical unit and the analytical unit that data are handled.
Optionally, described device can also include:
Generation module generates the structural body object of preset structure for being directed to the pending data；
The analysis module, specifically can be used for:
The structural body object is inputted into first analytical unit in the analytical unit combination, utilizes described first The thread that analytical unit includes analyzes the structural body object, is analyzed as a result, analysis result is added to described Structural body object obtains the output object of first analytical unit；
According to the analysis sequence, successively each analysis unit after first analytical unit is determined as currently Analytical unit；The output object of a upper analytical unit for present analysis unit is inputted into present analysis unit, utilizes current point The thread that analysis unit includes, analyzes the output object of a upper analytical unit, is analyzed as a result, analysis is tied Fruit is added to the output object of a upper analytical unit, obtains the output object of present analysis unit；
The third obtains module, specifically can be used for:
According to the output object of the last one analytical unit in analytical unit combination, processing result is obtained.
Optionally, described device can also include: the first adding module and the second adding module, wherein
First adding module, it is single for the output object of first analytical unit to be added to first analysis In the output queue of member；
The analysis module is also used to from the input rank of present analysis unit, according to default queue sequence, is successively read Take the output object of a upper analytical unit；
Second adding module, for after the output object for obtaining present analysis unit, by the defeated of active cell Object is added in the output queue of present analysis unit out.
Optionally, described device can also include:
Judgment module, for judging whether the output object of the last one analytical unit in the analytical unit combination meets Output condition；If it is satisfied, triggering the third obtains module；If conditions are not met, triggering the analysis module；
The third obtains module, specifically can be used for: obtaining from the output object of the last one analytical unit Processing result.
Optionally, described device further include: the 4th obtains module, the second loading module, input module and output module, In,
4th obtains module, for obtaining corresponding second configuration file of pending data；
Second loading module, for reading the mark in the interaction accessory library for including in second configuration file, to being read The corresponding interactive accessory inventory of the mark taken loads, and includes input interface, output interface, assembling in the interactive accessory library Accessory and detection accessory；
The pending data is input to the interactive accessory library for calling the input interface by input module；
The generation module, specifically can be used for: the assembling fittings utilized, the pending data is assembled, Generate the structural body object of preset structure；
The judgment module, specifically can be used for:
The detection accessory is called, the output object of the last one analytical unit in analytical unit combination is examined It surveys, judges whether testing result meets output condition；
Output module, for calling after obtaining processing result in the output object of the last one analytical unit The output interface exports the processing result.
It optionally, further include deleting accessory in the interactive accessory library, described device can also include:
Removing module deletes the output object of the last one analytical unit for calling the deletion accessory.
Optionally, described device can also include:
Unload module, for after obtaining processing result, unloading institute in the output content combined according to the analytical unit State the analytical unit in analytical unit combination.
In order to achieve the above objectives, the embodiment of the invention also provides a kind of electronic equipment, including processor and memory；
Memory, for storing computer program；
Processor when for executing the program stored on memory, realizes any of the above-described kind of number based on streaming frame According to processing method.
Using the embodiment of the present invention, after getting task parameters, analytical unit is reloaded, analytical unit is not with process Mode continuous service, and load analytical unit after, data are analyzed using the thread that analytical unit includes, compared to The scheme of analytical unit continuous service in a manner of process, reduces resources occupation rate.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with It obtains other drawings based on these drawings.
Fig. 1 is the first flow diagram of the data processing method provided in an embodiment of the present invention based on streaming frame；
Fig. 2 is second of flow diagram of the data processing method provided in an embodiment of the present invention based on streaming frame；
Fig. 3 is a kind of Stream Processing block schematic illustration provided in an embodiment of the present invention；
Fig. 4 is a kind of analytical unit provided in an embodiment of the present invention and pipeline connection schematic diagram；
Fig. 5 is another Stream Processing block schematic illustration provided in an embodiment of the present invention；
Fig. 6 is a kind of structural schematic diagram of the data processing equipment based on streaming frame provided in an embodiment of the present invention；
Fig. 7 is the structural schematic diagram of a kind of electronic equipment provided in an embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other Embodiment shall fall within the protection scope of the present invention.
In order to solve the above-mentioned technical problem, the embodiment of the invention provides a kind of data processing sides based on streaming frame Method, device and electronic equipment.This method and device can be applied to various electronic equipments, specifically without limitation.
A kind of data processing method based on streaming frame provided in an embodiment of the present invention is carried out specifically first below It is bright.
Fig. 1 is the first flow diagram of the data processing method provided in an embodiment of the present invention based on streaming frame, Include:
S101: the corresponding task parameters of pending data are obtained.
For example, pending data can be one section of video, an image etc., specifically without limitation.It needs to illustrate , pending data is not obtained in S101, but obtains the corresponding task parameters of pending data.
As an implementation, S101 may include: to obtain corresponding first configuration file of pending data.In order to The configuration file in content is mutually distinguished below, and the configuration file obtained in S101 is known as the first configuration file.First configuration text It may include the mark and mark sequence of analytical unit in part, mark and the mark sequence of the analytical unit belong to task ginseng Number.Alternatively, task parameters also may include a numerical value, the corresponding of numerical value and analytical unit mark and mark sequence is preset Relationship.The particular content of task parameters is without limitation.
S102: according to the task parameters, the analytical unit and the analysis list handled the pending data is determined The analysis sequence of member.Wherein, each analysis unit includes an independent thread.
As set forth above, it is possible to obtain analytical unit mark and mark sequence, which is to carry out to the pending data The mark of the analytical unit of processing, the mark of analytical unit can be serial number, ID etc., specifically without limitation, mark sequence The processing sequence of each analytical unit as handled for this part of pending data.
For by taking pending data is video as an example, video can be decoded, personnel targets detection, extract face spy Multiple processing steps, each processing steps such as sign, extraction personnel's clothing feature can correspond to an analytical unit.These processing steps There are sequencings between rapid, correspondingly, there is also analysis sequences between the analytical unit handled pending data.And For different pending datas, the analytical unit handled it can be different, the analysis list handled it The analysis sequence of member can also be different.
For example, including human face data in one section of video, needs to be decoded this section of video, human face target detection, extract Multiple processing steps such as face characteristic, each processing step can correspond to an analytical unit.Assuming that determining to pending data Handled there are three analytical unit: analytical unit 1, analytical unit 2 and analytical unit 3, wherein analytical unit 1 for pair Pending data is decoded, and analytical unit 2 is used to detect the human face target in pending data, and analytical unit 3 is for extracting Face characteristic in pending data.
For another example, in another section of video include license plate data, need to be decoded this section of video, vehicle target detection, An analytical unit can be corresponded to by extracting multiple processing steps, each processing steps such as vehicle license plate characteristic.Assuming that determining to be processed Data handled there are three analytical unit: analytical unit 1, analytical unit 4 and analytical unit 5, wherein analytical unit 1 is used It is decoded in pending data, analytical unit 4 is used to detect the vehicle target in pending data, and analytical unit 5 is used for Extract the vehicle license plate characteristic in pending data.
Each analysis unit includes an independent thread, and each analysis unit is using self-contained thread to be processed Data are handled.
S103: identified analytical unit is loaded；And according to the analysis sequence, each analysis list that will be loaded Member is attached, and obtains analytical unit combination.
In this equipment (executing subject) or the other equipment communicated to connect with this equipment, deposited in advance in the form of plug-in unit Multiple analytical units are stored up, the determining analytical unit handled the pending data is loaded onto memory.
Connection relationship can be not present between above-mentioned pre-stored multiple analytical units in advance.By identified analysis After unit is loaded onto the analysis sequence of memory and the determining each analytical unit handled pending data, according to the analysis Sequentially, each analytical unit after load is attached, obtains analytical unit combination.In this way, the connection between analytical unit is closed System be it is adjustable, can be directed to different pending datas, establish different connection relationships, obtain different analytical unit groups It closes.
Continue above-mentioned example, pending data be one section include human face data video, determine to pending data into Row processing is analytical unit 1-3, and analysis sequence is from front to back are as follows: analytical unit 1, analytical unit 2, analytical unit 3.According to this These three analytical units are attached by analysis sequence, specifically, by the defeated of the output interface of analytical unit 1 and analytical unit 2 Incoming interface is connected, and the output interface of analytical unit 2 is connected with the input interface of analytical unit 3, obtains analytical unit combination.
S104: the pending data is obtained, and the pending data is inputted into analytical unit combination, utilizes the analysis list The thread that each analysis unit includes in member combination, analyzes the pending data.
For example, the task parameters got in S101 can also include the address information of pending data, the address Information can be URL (Uniform Resource Locator, uniform resource locator), or, or other, specifically Without limitation.It is available to arrive pending data according to the address information.
It is still illustrated so that pending data is video as an example, first by the video input analytical unit 1, utilizes analytical unit Thread in 1 is decoded the video, obtains decoded data；By decoded data input analytical unit 2, using point Thread in analysis unit 2 detects the human face target in decoded data, obtains human face target data；Human face target data are defeated Enter analytical unit 3, extracts the face characteristic in pending data using the thread in analytical unit 3, obtain face characteristic data.
S105: the output content combined according to the analytical unit obtains processing result.
It as an implementation, can be directly using the output content as processing result；Alternatively, can will be in the output Appearance formats, using the data after conversion as processing result；Alternatively, significant figure can be extracted from the output content According to as processing result, etc., specifically without limitation.
For example, analytical unit 3 is the last one unit in analytical unit combination, the output of analytical unit 3 in above-mentioned example Content i.e. the output content of analytical unit combination, the output content are face characteristic, can be directly by face spy Data are levied as processing result.
Using embodiment illustrated in fig. 1 of the present invention, after getting task parameters, analytical unit is reloaded, analytical unit is not It is the continuous service in a manner of process, and after load analytical unit, data is divided using the thread that analytical unit includes Analysis, compared to the scheme of analytical unit continuous service in a manner of process, reduces resources occupation rate, and thread is compared to process It is more easily controlled.
As an implementation, after getting pending data, it can be directed to pending data, generate default knot The structural body object of structure.For example, which may include three parts, and first part is properly termed as head construction, head It can store the processing progress information handled to pending data, error code information in treatment process etc. in portion's structure Deng.Second part can store the pending data, and Part III can store point of each analysis unit in analysis combination Analyse result.
It is combined in this way, the pending data is inputted the analytical unit, using each in analytical unit combination The thread that analytical unit includes analyzes the pending data, may include:
One, the structural body object is inputted into first analytical unit in analytical unit combination, utilizes described the The thread that one analytical unit includes analyzes the structural body object, is analyzed as a result, analysis result is added to The structural body object obtains the output object of first analytical unit；
Two, according to the analysis sequence, successively each analysis unit after first analytical unit is determined as Present analysis unit；The output object of a upper analytical unit for present analysis unit is inputted into present analysis unit, using working as The thread that preceding analytical unit includes analyzes the output object of a upper analytical unit, is analyzed as a result, will divide Analysis result is added to the output object of a upper analytical unit, obtains the output object of present analysis unit.
S105 may include: according to the output object of the last one analytical unit in analytical unit combination, at acquisition Manage result.
Continue above-mentioned example, the analysis result that each analysis unit all obtains itself is added in above-mentioned preset structure Part III, that is to say, that comprising the full content in structural body object in the output object of the last one analytical unit, also wrap Analysis result containing analytical units whole in analytical unit combination.It for example, can be in the last one analytical unit It exports in object, extracts the analysis result of the last one analytical unit as processing result；Alternatively, can also at this last one In the output object of analytical unit, the analysis result of each analysis unit is extracted as processing result；Alternatively, can also be by this most The output object of the latter analytical unit is as processing result.
It as an implementation, may include an input rank, an output queue, analysis in each analysis unit Unit obtains the output object of a upper analytical unit from input rank, or obtains the structural body object, and analytical unit will Itself output object is added in output queue.
Specifically, after the output object for obtaining first analytical unit, described first is analyzed The output object of unit is added in the output queue of first analytical unit；
The output object of a upper analytical unit by present analysis unit inputs present analysis unit, comprising: from In the input rank of present analysis unit, according to default queue sequence, it is successively read the output pair of a upper analytical unit As；
After the output object for obtaining present analysis unit, the output object of active cell is added to current point In the output queue for analysing unit.
For example, if pending data be video, each analytical unit can respectively to every image in video into Row processing.Assuming that being handled using analytical unit 1-2 video, first by the corresponding structural body object input point of first image Unit 1 is analysed, analytical unit 1 analyzes the structural body object using self-contained thread, obtains an output object, will The output object is added in the output queue of analytical unit 1；It is similar, analytical unit 1 to each image in video all Carry out above-mentioned processing.
The input rank of analytical unit 2 can be connected with the output queue of analytical unit 1 by pipeline, in this way, analysis Unit 2 can get the output object of analytical unit 1 from the input rank of itself.Input team of the analytical unit 2 from itself It is successively read the output object of analytical unit 1 in column, an output object can be read every time.Analytical unit 2 is wrapped using itself The thread contained analyzes the output object read, obtains an output object, and the output object is added to In the output queue of analytical unit 2；Similar, analytical unit 2 carries out above-mentioned processing to each output object read.
In the present embodiment, input/output queue is set for each analysis unit, each analysis unit is come Say may exist multiple output objects in the input rank of the analytical unit, in this way, even if a upper analysis for the analytical unit is single First processing speed is lower, which improves number compared to the scheme for being not provided with input/output queue without waiting According to treatment effeciency.
As embodiment at one, in the output object according to the last one analytical unit in analytical unit combination, After obtaining processing result, it can also judge whether the output object of the last one analytical unit in the analytical unit combination meets Output condition；If it is satisfied, obtaining processing result from the output object of the last one analytical unit；If conditions are not met, It returns and executes the step of structural body object is inputted into first analytical unit in the analytical unit combination.
By it is above-mentioned comprising the video of license plate data for for, in the output object of the last one analytical unit include license plate Characteristic, such as license plate number；Output condition can be set for the license plate number, for example whether there is unidentified character out, If it is present repeating first analytical unit and subsequent step in the input analytical unit combination of structural body object. Until there is no unidentified characters out in the license plate number for including in the output object of the last one analytical unit, alternatively, until The number repeated reaches threshold value.In other words, whether which can also reach threshold for the number repeated Value.
In the present embodiment, circulative metabolism is realized, that is, when the output object of the last one analytical unit is discontented In the case where sufficient output condition, recycles each analytical unit and analyzed, until output object meets output condition, or Cycle-index reaches threshold value, in this way, the accuracy of processing result can be improved.It can not achieve this in existing Storm frame Kind circulative metabolism, therefore present embodiment improves the accuracy of processing result compared to Storm frame.
As an implementation, before obtaining pending data, pending data corresponding second can also be obtained Configuration file；The mark for reading the interaction accessory library for including in second configuration file, friendship corresponding to read mark Mutual accessory inventory loads, and includes input interface, output interface, assembling fittings and detection accessory in the interactive accessory library. " input interface " mentioned here refers to that the input interface in interactive accessory library, " output interface " mentioned here refer to interactive accessory The output interface in library.The input/output interface in interaction accessory library can be to connect with what upper layer application or user equipment were connected Mouthful, it can be understood as external interface, and in the input/output interface of the analytical unit in above content and following the description The input/output interface of pipeline is not connect with upper layer application, user equipment etc., can be understood as internal interface.
After obtaining pending data, the input interface in interactive accessory library may call upon, by the pending data It is input to the interaction accessory library.
In the present embodiment, there are one in this equipment (executing subject) or the other equipment communicated to connect with this equipment A or multiple interactive accessory libraries include input interface, output interface, assembling fittings and detection accessory in each interactive accessory library. Every part of pending data can correspond to a second configuration file.It is described to be distinguished with the configuration file in content above, it will The configuration file of mark comprising interaction accessory library is known as the second configuration file.
Alternatively, the first configuration file and the second configuration file also may belong to same configuration file, in the configuration file both Mark comprising interaction accessory library, and include the mark and mark sequence of analytical unit.
Every part of pending data can correspond to a second configuration file, in second configuration file comprising to the part to The mark in the interaction accessory library that processing data are handled.The mark in interaction accessory library can be serial number, ID etc., not do specifically It limits, the corresponding interactive accessory library of the mark is loaded onto memory.
It is described to be directed to pending data, the structural body object of preset structure is generated, may include: to utilize described group of assembly Part assembles the pending data, generates the structural body object of preset structure.
In the present embodiment, the input interface in interaction accessory library is called, pending data is input to the interaction and is matched Pending data is assembled using the assembling fittings, generates the structural body object of preset structure by part library.
In addition, determine the analytical unit handled pending data and each analytical unit sequence, it, will according to the sequence Identified each analytical unit connects into analytical unit combination；Structural body object is inputted into analytical unit combination, analytical unit Combination one output object of output.
Whether the output object for judging the last one analytical unit in the analytical unit combination meets output condition, It may include: to call the detection accessory, the output object of the last one analytical unit in analytical unit combination carried out Detection, judges whether testing result meets output condition.
If conditions are not met, the structural body object is inputted analytical unit combination again；If it is satisfied, from the output object Middle acquisition processing result, and the output interface in interactive accessory library is called, the processing result is exported, " output " here can be The processing result is sent into user equipment or upper layer application.
Can also include deleting accessory in interaction accessory library, call the output interface, export the processing result it Afterwards, it may call upon the deletion accessory, delete the output object of the last one analytical unit, in this way, in can saving Resource is deposited, guarantees that memory does not leak.
As an implementation, it after pending data is disposed, can unload in the analytical unit combination Analytical unit, in this way, it is possible to reduce the occupancy of memory.
Fig. 2 is second of flow diagram of the data processing method provided in an embodiment of the present invention based on streaming frame,
S201: corresponding second configuration file of pending data is obtained.
S202: reading the mark in the interaction accessory library for including in second configuration file, corresponding to read mark Interaction accessory inventory loads, and includes input interface, output interface, assembling fittings and detection accessory in the interaction accessory library.
In the present embodiment, there may be two-stages to load, and in order to distinguish description, the first order is loaded to utilized configuration file The second level is loaded utilized configuration file and is known as the first configuration file by referred to as the second configuration file.The first order is loaded as Interaction accessory library is carried, the second level is loaded as load analytical unit.
Specific processing frame can be as shown in figure 3, include interaction accessory library, plugin manager and analytical unit.Plug-in unit Manager loads interaction accessory library, by reading the first configuration file, loads analytical unit by reading the second configuration file. Interaction accessory library and analytical unit can have multiple, for different pending datas, usually load different interaction accessory libraries And analytical unit.
In first order load, plugin manager obtains handling pending data by reading the second configuration file Interaction accessory library mark；The corresponding interactive accessory library of the mark is loaded onto memory by plugin manager, and maps the interaction The interface IP address in accessory library, interface IP address include input interface address, output interface address.
S203: corresponding first configuration file of pending data is obtained；According to the analysis for including in first configuration file Mark and the mark sequence of unit determine point of the analytical unit and the analytical units that handled the pending data Analysis sequence；Wherein, each analysis unit includes an independent thread.
S204: identified analytical unit is loaded；And according to the analysis sequence, each analysis list that will be loaded Member is attached, and obtains analytical unit combination.
S204 is second level load, and the plugin manager in Fig. 3 obtains analytical unit by reading the first configuration file Mark and mark sequence, plugin manager the corresponding analytical unit of the mark is loaded onto memory, and according to mark sequence, Each analytical unit after load is connected, analytical unit combination is obtained.
In the present embodiment, as shown in figure 3, can be attached by pipeline between each analytical unit, each analytical unit Between the data transmitted another analytical unit is flowed to from an analytical unit by pipeline.First analytical unit and last A analytical unit is connected by pipeline with accessory library is interacted.As shown in figure 4, pipeline internal interface includes an input interface Pin and output interface pout, wherein pin is connected with the output interface of analytical unit 1, pout and analytical unit 2 it is defeated Incoming interface is connected, in this way, data can flow to analytical unit 2 from analytical unit 1.
It for example, can be without data buffer storage in pipeline；Furthermore it is also possible to which setting repeats transmission time in the duct Number, for example in Fig. 4, if the data transmission fails between analytical unit 1 and analytical unit 2, repeat to transmit, until repeating to pass Defeated number reaches the number of pipeline setting.
In the present embodiment, encapsulated layer can be set between interaction accessory library and analytical unit, the encapsulated layer is for shielding Logical relation inside each analytical unit.Than as shown in Figure 5, plugin manager and each analytical unit with interact between accessory library Encapsulated layer is set, and interaction accessory library need to only call the interface of encapsulated layer, can response connecing to each analytical unit as streaming Mouthful.In this way, not needing traversal for called side and calling all analytical units, do not need to be concerned about between analytical unit yet Calling logic, it is easy to operate, experience more preferably.
Moreover, including an independent thread in each analysis unit, that is to say, that each analysis list in the present embodiment Operation in member is executed by different threads, is independent of each other between each thread, even if the operand in an analytical unit is larger, The operation in other analytical units will not be influenced, improves operation efficiency, and hold in a manner of process compared to analytical unit The scheme of reforwarding row, reduces resources occupation rate.
S205: pending data is obtained.
S206: calling the input interface, which is input to the interaction accessory library.
S207: the assembling fittings are utilized, which is assembled, the structural body object of preset structure is generated.
Interaction accessory library is loaded onto memory in S202, includes assembling fittings in the interaction accessory library, which can Pending data to be assembled into the structural body object of preset structure.For example, which may include three parts, the A part is properly termed as head construction, can store in head construction the processing progress information that pending data is handled, Error code information in treatment process etc..Second part can store the pending data, and Part III can store this point The analysis result of each analysis unit in analysis combination.
S208: the structural body object is inputted into first analytical unit in analytical unit combination, utilizes this first The thread that analytical unit includes analyzes the structural body object, is analyzed as a result, analysis result is added to the structure Body object obtains the output object of first analytical unit.
S209: according to the analysis sequence, successively each analysis unit after first analytical unit is determined as working as Preceding analytical unit；The output object of a upper analytical unit for present analysis unit is inputted into present analysis unit, using current The thread that analytical unit includes analyzes the output object of an analytical unit on this, is analyzed as a result, analysis is tied Fruit is added to the output object of an analytical unit on this, obtains the output object of present analysis unit.
It as an implementation, may include an input rank, an output queue, analysis in each analysis unit Unit obtains the output object of a upper analytical unit from input rank, or obtains the structural body object, and analytical unit will Itself output object is added in output queue.
Specifically, after the output object for obtaining first analytical unit, by the defeated of first analytical unit Object is added in the output queue of first analytical unit out；From the input rank of present analysis unit, according to pre- If queue sequence, it is successively read the output object of a upper analytical unit；In the output object for obtaining present analysis unit Later, the output object of active cell is added in the output queue of present analysis unit.
In the present embodiment, input/output queue is set for each analysis unit, each analysis unit is come Say may exist multiple output objects in the input rank of the analytical unit, in this way, even if a upper analysis for the analytical unit is single First processing speed is lower, which improves number compared to the scheme for being not provided with input/output queue without waiting According to treatment effeciency.
S210: calling the detection accessory, carries out to the output object of the last one analytical unit in analytical unit combination Detection, judges whether testing result meets output condition, if so, executing S211, executes S208 if not, returning.
As described above, being directed to different pending datas, different interaction accessory libraries can be loaded, that is to say, that interaction Each accessory in accessory library is suitable for the pending data, and therefore, the detection accessory in interaction accessory library can be directed to should The output object of pending data is detected.
For example, pending data is the video comprising license plate data, include in the output object of the last one analytical unit The recognition result of license plate number, the detection accessory can detecte in the recognition result with the presence or absence of unidentified character out；If deposited Then testing result is unsatisfactory for output condition, repeats S208-S210；If it does not exist, then testing result meets output bars Part executes S211.
In addition, can also be for the number given threshold repeated, if testing result is discontented with always in detection accessory Sufficient output condition, the number for repeating S208-S210 reach the threshold value, also think that testing result meets output condition.
When returning to execution S208, it can be directed to the output object of the last one analytical unit, by the third of the output object The analysis result of each analytical unit stored in part is deleted, and by the information stored in the first part of the output object into Row updates, for example the error code information etc. in processing progress information, treatment process is updated to initial value, etc., does not do specifically It limits.It is single that first analysis is inputted using the output object of the last one analytical unit Jing Guo above-mentioned processing as structural body object Member.
Alternatively, can re-call assembling fittings when returning to execution S208, regenerate the structural body pair of preset structure As the newly-generated structural body object is inputted first analytical unit.
In the present embodiment, circulative metabolism is realized, that is, when the output object of the last one analytical unit is discontented In the case where sufficient output condition, recycles each analytical unit and analyzed, until output object meets output condition, or Cycle-index reaches threshold value, in this way, the accuracy of processing result can be improved.It can not achieve this in existing Storm frame Kind circulative metabolism, therefore present embodiment improves the accuracy of processing result compared to Storm frame.
S211: obtaining processing result from the output object of the last one analytical unit, and call the output interface, defeated The processing result out.
Continue above-mentioned example, the analysis result that each analysis unit all obtains itself is added in above-mentioned preset structure Part III, in this way, comprising the full content in structural body object in the output object of the last one analytical unit, also comprising being somebody's turn to do The analysis result of whole analytical units in analytical unit combination.
For example, the last one analytical unit can be extracted in the output object of the last one analytical unit Result is analyzed as processing result；Alternatively, each analysis can also be extracted in the output object of the last one analytical unit The analysis result of unit is as processing result；Alternatively, can also be using the output object of the last one analytical unit as processing As a result.
S212: calling the deletion accessory, deletes the output object of the last one analytical unit.
S213: the analytical unit in the interaction accessory library and analytical unit combination is unloaded.
As an implementation, after pending data is disposed, it can be unloaded and be interacted by plugin manager Accessory library and analytical unit.It is corresponding with load interaction accessory library and analytical unit, it can unload and hand over by the second configuration file Mutual accessory library, passes through the first configuration file, Unloading Analysis unit.
Specifically, can be according to the address handle in the interaction accessory library for including in the second configuration file, to discharge in S202 The memory headroom applied when loading interaction accessory library；According to the address handle for the analytical unit for including in the first configuration file, come The memory headroom applied when loading analytical unit in release S204.
In present embodiment, after the completion of data processing to be processed, the memory of release treatment process occupancy, it is possible to reduce interior Deposit occupancy.
A specific embodiment is introduced below with reference to Fig. 3-5:
1, this equipment receives a processing task: the license plate number in one section of video of identification.This equipment obtains the processing and appoints It is engaged in relevant two parts of configuration files: configuration file A and configuration file B.Pending data is this section of video, and configuration file A is For the second configuration file in above content, configuration file B is the first configuration file in above content.
Read the mark in the interaction accessory library for including in configuration file A, it is assumed that it is serial number 3, then it, will by plugin manager The interaction accessory library of serial number 3 is loaded onto memory, in the interaction accessory library include input interface, output interface, assembling fittings and Detect accessory.This section of video is obtained, the input interface in the interaction accessory library is called, by this section of video input interaction accessory Library.Using the assembling fittings in the interaction accessory library, this section of video is assembled into the structural body object of preset structure.
For example, which may include three parts, and first part is properly termed as head construction, in head construction It can store the processing progress information handled to pending data, error code information in treatment process etc..Second Divide and can store the pending data, Part III can store the analysis result of each analysis unit in analysis combination.
In addition, reading the analytical unit mark and mark sequence for including in configuration file B, it is assumed that mark and mark sequence Are as follows: these three analytical units are loaded onto interior by analytical unit 1, analytical unit 4 and analytical unit 5 then by plugin manager It deposits, and according to the sequence, these three analytical units is attached, obtain analytical unit combination.Specifically, can be such as Fig. 3 institute Show, by pipeline, these three analytical units are connected, wherein analytical unit 1 and analytical unit 5 are matched by pipeline with interacting Part library is connected.It include an independent thread in each analysis unit.
As shown in figure 5, encapsulated layer is arranged with interacting in plugin manager and each analytical unit between accessory library, interaction accessory library The interface of encapsulated layer need to only be called, can response as streaming to each analytical unit interface.In this way, coming for called side It says, does not need traversal and call all analytical units, also do not need the calling logic being concerned about between analytical unit, it is easy to operate, Experience is more preferably.
The structural body object that assembling fittings are assembled inputs the defeated of first analytical unit in analytical unit combination Enqueue, that is, the input rank of analytical unit 1.Analytical unit 1 can obtain a knot periodically from itself input rank Structure body object, and read the pending data that second part stores in the structural body object, that is, this section of video；Analytical unit 1 is decoded processing to this section of video using self-contained thread, obtains decoding data, ties the decoding data as analysis Fruit is added to the Part III of structural body object, and the structural body object being added to after the decoding data is analytical unit 1 Export object；The output object is output in the output queue of itself by analytical unit 1.
The output queue of analytical unit 1 is connected with the input rank of analytical unit 4 by pipeline, in this way, analytical unit 4 Input rank in there is the output objects of one or more analytical units 1.Analytical unit 4 can be periodically from itself input rank In, obtain the output object an of analytical unit 1.
Analytical unit 4 is read first in the decoding data that Part III stores in the output object and the output object Partial processing progress information；Analytical unit 4 utilizes self-contained thread, in conjunction with the processing progress information, detects the decoding Vehicle target in data, the vehicle target data that will test are added to the third portion of the output object as analysis result Point, it is added to the output object that the output object after the vehicle target data is analytical unit 4；Analytical unit 4 is defeated by this Object is output in the output queue of itself out.
The output queue of analytical unit 4 is connected with the input rank of analytical unit 5 by pipeline, in this way, analytical unit 5 Input rank in there is the output objects of one or more analytical units 4.Analytical unit 5 can be periodically from itself input rank In, obtain the output object an of analytical unit 4.
Analytical unit 5 is read in the vehicle target data that Part III stores in the output object and the output object The processing progress information of first part；Analytical unit 5 utilizes self-contained thread, in conjunction with the processing progress information, extracts vehicle License plate number in target data is added to the Part III of the output object using the license plate number as analysis result, is added to Output object after the license plate number is the output object of analytical unit 5；The output object is output to itself by analytical unit 5 Output queue in.
Interaction accessory library in detection accessory can the output object to analytical unit 5 detect, judge the license plate number In there is no it is unidentified go out character；If it does not exist, then testing result meets output condition, from the output pair of analytical unit 5 As middle acquisition license plate number is as processing result, and the output interface is called, exports the processing result.
If in the license plate number there are it is unidentified go out character, testing result is unsatisfactory for output condition, return execute general Structural body object inputs the step of analytical unit 1, until testing result meets output condition, or until returns to the number executed Reach threshold value.
When returning to step of the execution by structural body object input analytical unit 1, the output pair of analytical unit 5 can be directed to As the analysis result of each analytical unit stored in the Part III of the output object being deleted, and by the output object The information stored in first part is updated, for example the error code information etc. in processing progress information, treatment process is updated For initial value, etc., specifically without limitation.Using the output object of the analytical unit 5 Jing Guo above-mentioned processing as structural body object Input analytical unit 1.
Alternatively, assembling fittings can be re-called when returning to step of the execution by structural body object input analytical unit 1, The newly-generated structural body object is inputted analytical unit 1 by the structural body object for regenerating preset structure.
After exporting processing result, the deletion accessory in interaction accessory library is called, the output object of analytical unit 5 is deleted.Separately Outside, interaction accessory library and analytical unit can also be unloaded by plugin manager.Specifically, can be wrapped according in configuration file A The address handle in the interaction accessory library contained, the memory headroom applied when loading interaction accessory library to discharge；According in configuration file B The address handle for the analytical unit for including, to discharge the memory headroom applied when load analytical unit.
Using present embodiment, in a first aspect, reloading analytical unit, analytical unit is not after getting task parameters It is the continuous service in a manner of process, and after load analytical unit, data is divided using the thread that analytical unit includes Analysis, compared to the scheme of analytical unit continuous service in a manner of process, reduces resources occupation rate, and thread is compared to process It is more easily controlled；Second aspect, realizes circulative metabolism, that is, when the output object of the last one analytical unit be unsatisfactory for it is defeated It out in the case where condition, recycles each analytical unit and is analyzed, until going out output object meets output condition, or follow Ring number reaches threshold value, in this way, the accuracy of processing result can be improved；The third aspect is loaded by two-stage, and load is handed over respectively Mutual accessory library and analytical unit load different interaction accessory library and analytical unit, load spirit for different pending datas It is living adjustable.
Corresponding with above method embodiment, the embodiment of the present invention also provides a kind of data processing dress based on streaming frame It sets, as shown in Figure 6, comprising:
First obtains module 601, for obtaining the corresponding task parameters of pending data；
Determining module 602, for determining the analysis list handled the pending data according to the task parameters The analysis sequence of member and the analytical unit；Wherein, each analysis unit includes an independent thread；
First loading module 603, for being loaded to identified analytical unit；
Link block 604, for each analysis unit loaded being attached, is obtained according to the analysis sequence Analytical unit combination；
Second obtains module 605, for obtaining the pending data；
Analysis module 606 combines for the pending data to be inputted the analytical unit, utilizes the analytical unit The thread that each analysis unit includes in combination analyzes the pending data；
Third obtains module 607, and the output content for being combined according to the analytical unit obtains processing result.
As an implementation, first module 601 is obtained, specifically can be used for: obtaining pending data corresponding the One configuration file；
Determining module 602, specifically can be used for:
According to the mark for the analytical unit for including in first configuration file and mark sequence, determine to described to be processed The analysis sequence of analytical unit and the analytical unit that data are handled.
As an implementation, described device can also include:
Generation module (not shown) generates the structural body object of preset structure for being directed to the pending data；
Analysis module 606, specifically can be used for:
The structural body object is inputted into first analytical unit in the analytical unit combination, utilizes described first The thread that analytical unit includes analyzes the structural body object, is analyzed as a result, analysis result is added to described Structural body object obtains the output object of first analytical unit；
According to the analysis sequence, successively each analysis unit after first analytical unit is determined as currently Analytical unit；The output object of a upper analytical unit for present analysis unit is inputted into present analysis unit, utilizes current point The thread that analysis unit includes, analyzes the output object of a upper analytical unit, is analyzed as a result, analysis is tied Fruit is added to the output object of a upper analytical unit, obtains the output object of present analysis unit；
Third obtains module 607, specifically can be used for:
According to the output object of the last one analytical unit in analytical unit combination, processing result is obtained.
As an implementation, described device can also include: the first adding module and the second adding module (in figure not Show), wherein
First adding module, it is single for the output object of first analytical unit to be added to first analysis In the output queue of member；
Analysis module 606 can be also used for from the input rank of present analysis unit, according to default queue sequence, according to The secondary output object for reading a upper analytical unit；
Second adding module, for after the output object for obtaining present analysis unit, by the defeated of active cell Object is added in the output queue of present analysis unit out.
As an implementation, described device can also include:
Judgment module (not shown), for judging the output of the last one analytical unit in the analytical unit combination Whether object meets output condition；If it is satisfied, triggering third obtains module 607；If conditions are not met, triggering analysis module 606；
Third obtains module 607, specifically can be used for: acquisition place from the output object of the last one analytical unit Manage result.
As an implementation, described device can also include: the 4th acquisition module, the second loading module, input mould Block and output module (not shown), wherein
4th obtains module, for obtaining corresponding second configuration file of pending data；
Second loading module, for reading the mark in the interaction accessory library for including in second configuration file, to being read The corresponding interactive accessory inventory of the mark taken loads, and includes input interface, output interface, assembling in the interactive accessory library Accessory and detection accessory；
The pending data is input to the interactive accessory library for calling the input interface by input module；
The generation module, specifically can be used for: the assembling fittings utilized, the pending data is assembled, Generate the structural body object of preset structure；
The judgment module, specifically can be used for:
The detection accessory is called, the output object of the last one analytical unit in analytical unit combination is examined It surveys, judges whether testing result meets output condition；
Output module, for calling after obtaining processing result in the output object of the last one analytical unit The output interface exports the processing result.
It as an implementation, further include deleting accessory, described device in the interactive accessory library further include:
Removing module (not shown) deletes the last one analytical unit for calling the deletion accessory Export object.
As an implementation, described device can also include:
Unload module (not shown), for obtaining processing knot in the output content combined according to the analytical unit After fruit, the analytical unit in the analytical unit combination is unloaded.
Using embodiment illustrated in fig. 6 of the present invention, after getting task parameters, analytical unit is reloaded, analytical unit is not It is the continuous service in a manner of process, and after load analytical unit, data is divided using the thread that analytical unit includes Analysis, compared to the scheme of analytical unit continuous service in a manner of process, reduces resources occupation rate.
The embodiment of the invention also provides a kind of electronic equipment, as shown in fig. 7, comprises processor 701 and memory 702,
Memory 702, for storing computer program；
Processor 701 when for executing the program stored on memory 702, realizes that any of the above-described kind is based on streaming frame The data processing method of frame.
The memory that above-mentioned electronic equipment is mentioned may include random access memory (Random Access Memory, It RAM), also may include nonvolatile memory (Non-Volatile Memory, NVM), for example, at least a disk storage Device.Optionally, memory can also be that at least one is located remotely from the storage device of aforementioned processor.
Above-mentioned processor can be general processor, including central processing unit (Central Processing Unit, CPU), network processing unit (Network Processor, NP) etc.；It can also be digital signal processor (Digital Signal Processing, DSP), it is specific integrated circuit (Application Specific Integrated Circuit, ASIC), existing It is field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete Door or transistor logic, discrete hardware components.
It should be noted that, in this document, relational terms such as first and second and the like are used merely to a reality Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation In any actual relationship or order or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to Non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only wanted including those Element, but also including other elements that are not explicitly listed, or further include for this process, method, article or equipment Intrinsic element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that There is also other identical elements in process, method, article or equipment including the element.
Each embodiment in this specification is all made of relevant mode and describes, same and similar portion between each embodiment Dividing may refer to each other, and each embodiment focuses on the differences from other embodiments.Especially for Fig. 6 institute The data processing equipment embodiment based on streaming frame shown, for electronic equipment embodiment shown in Fig. 7, due to its basic phase It is similar to the data processing method embodiment shown in Fig. 1-5 based on streaming frame, so being described relatively simple, related place ginseng See the part explanation of the data processing method embodiment shown in Fig. 1-5 based on streaming frame.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all Any modification, equivalent replacement, improvement and so within the spirit and principles in the present invention, are all contained in protection scope of the present invention It is interior.

Claims (17)

1. a kind of data processing method based on streaming frame characterized by comprising
Obtain the corresponding task parameters of pending data；
According to the task parameters, the analytical unit and the analytical unit handled the pending data is determined Analysis sequence；Wherein, each analysis unit includes an independent thread；
Identified analytical unit is loaded；And according to the analysis sequence, each analysis unit loaded is carried out Connection obtains analytical unit combination；
The pending data is obtained, and the pending data is inputted into the analytical unit and is combined, it is single using the analysis The thread that each analysis unit includes in member combination, analyzes the pending data；
According to the output content that the analytical unit combines, processing result is obtained.
2. the method according to claim 1, wherein the corresponding task parameters of the acquisition pending data, packet It includes:
Obtain corresponding first configuration file of pending data；
It is described according to the task parameters, determine the analytical unit handled the pending data and the analysis The analysis sequence of unit, comprising:
According to the mark for the analytical unit for including in first configuration file and mark sequence, determine to the pending data The analysis sequence of the analytical unit and the analytical unit that are handled.
3. the method according to claim 1, wherein after obtaining the pending data, further includes: be directed to The pending data generates the structural body object of preset structure；
It is described that the pending data is inputted into the analytical unit combination, utilize each analysis list in analytical unit combination The thread that member includes analyzes the pending data, comprising:
The structural body object is inputted into first analytical unit in the analytical unit combination, is analyzed using described first The thread that unit includes analyzes the structural body object, is analyzed as a result, analysis result is added to the structure Body object obtains the output object of first analytical unit；
According to the analysis sequence, each analysis unit after first analytical unit is successively determined as present analysis Unit；The output object of a upper analytical unit for present analysis unit is inputted into present analysis unit, utilizes present analysis list The thread that member includes analyzes the output object of a upper analytical unit, is analyzed as a result, analysis result is added The output object for adding to a upper analytical unit, obtains the output object of present analysis unit；
The output content combined according to the analytical unit, obtains processing result, comprising:
According to the output object of the last one analytical unit in analytical unit combination, processing result is obtained.
4. according to the method described in claim 3, it is characterized in that, in the output pair for obtaining first analytical unit As later, further includes:
The output object of first analytical unit is added in the output queue of first analytical unit；
The output object of a upper analytical unit by present analysis unit inputs present analysis unit, comprising:
From the input rank of present analysis unit, according to default queue sequence, it is successively read a upper analytical unit Export object；
After the output object for obtaining present analysis unit, further includes:
The output object of active cell is added in the output queue of present analysis unit.
5. according to the method described in claim 3, it is characterized in that, described according to the last one point in analytical unit combination The output object of unit is analysed, processing result is obtained, comprising:
Judge whether the output object of the last one analytical unit in the analytical unit combination meets output condition；
If it is satisfied, obtaining processing result from the output object of the last one analytical unit；
If conditions are not met, returning to first analytical unit for executing and inputting the structural body object in the analytical unit combination The step of.
6. according to the method described in claim 5, it is characterized in that, before obtaining the pending data, further includes: obtain Corresponding second configuration file of pending data；
The mark for reading the interaction accessory library for including in second configuration file, interactive accessory corresponding to read mark Inventory loads, and includes input interface, output interface, assembling fittings and detection accessory in the interactive accessory library；
After obtaining the pending data, further includes:
The input interface is called, the pending data is input to the interactive accessory library；
It is described to be directed to the pending data, generate the structural body object of preset structure, comprising:
Using the assembling fittings, the pending data is assembled, generates the structural body object of preset structure；
Whether the output object for judging the last one analytical unit in the analytical unit combination meets output condition, wraps It includes:
The detection accessory is called, the output object of the last one analytical unit in analytical unit combination is detected, Judge whether testing result meets output condition；
After obtaining processing result in the output object of the last one analytical unit, further includes:
The output interface is called, the processing result is exported.
7. according to the method described in claim 6, it is characterized in that, further include in the interactive accessory library delete accessory, adjusting With the output interface, after exporting the processing result, further includes:
The deletion accessory is called, the output object of the last one analytical unit is deleted.
8. the method according to claim 1, wherein being obtained in the output content combined according to the analytical unit After taking processing result, further includes:
Unload the analytical unit in the analytical unit combination.
9. a kind of data processing equipment based on streaming frame characterized by comprising
First obtains module, for obtaining the corresponding task parameters of pending data；
Determining module, for according to the task parameters, determine the analytical unit handled the pending data and The analysis sequence of the analytical unit；Wherein, each analysis unit includes an independent thread；
First loading module, for being loaded to identified analytical unit；
Link block, for each analysis unit loaded being attached, analytical unit is obtained according to the analysis sequence Combination；
Second obtains module, for obtaining the pending data；
Analysis module is combined for the pending data to be inputted the analytical unit, using in analytical unit combination The thread that each analysis unit includes analyzes the pending data；
Third obtains module, and the output content for being combined according to the analytical unit obtains processing result.
10. device according to claim 9, which is characterized in that described first obtains module, is specifically used for: obtaining wait locate Manage corresponding first configuration file of data；
The determining module, is specifically used for:
According to the mark for the analytical unit for including in first configuration file and mark sequence, determine to the pending data The analysis sequence of the analytical unit and the analytical unit that are handled.
11. device according to claim 9, which is characterized in that described device further include:
Generation module generates the structural body object of preset structure for being directed to the pending data；
The analysis module, is specifically used for:
The structural body object is inputted into first analytical unit in the analytical unit combination, is analyzed using described first The thread that unit includes analyzes the structural body object, is analyzed as a result, analysis result is added to the structure Body object obtains the output object of first analytical unit；
According to the analysis sequence, each analysis unit after first analytical unit is successively determined as present analysis Unit；The output object of a upper analytical unit for present analysis unit is inputted into present analysis unit, utilizes present analysis list The thread that member includes analyzes the output object of a upper analytical unit, is analyzed as a result, analysis result is added The output object for adding to a upper analytical unit, obtains the output object of present analysis unit；
The third obtains module, is specifically used for:
According to the output object of the last one analytical unit in analytical unit combination, processing result is obtained.
12. device according to claim 11, which is characterized in that described device further include: the first adding module and second Adding module, wherein
First adding module, for the output object of first analytical unit to be added to first analytical unit In output queue；
The analysis module is also used to from the input rank of present analysis unit, according to default queue sequence, is successively read institute State the output object an of analytical unit；
Second adding module, for after the output object for obtaining present analysis unit, by the output pair of active cell As being added in the output queue of present analysis unit.
13. device according to claim 11, which is characterized in that described device further include:
Judgment module, for judging whether the output object of the last one analytical unit in the analytical unit combination meets output Condition；If it is satisfied, triggering the third obtains module；If conditions are not met, triggering the analysis module；
The third obtains module, is specifically used for: obtaining processing result from the output object of the last one analytical unit.
14. device according to claim 13, which is characterized in that described device further include: the 4th acquisition module, second add Carry module, input module and output module, wherein
4th obtains module, for obtaining corresponding second configuration file of pending data；
Second loading module, for reading the mark in the interaction accessory library for including in second configuration file, to read It identifies corresponding interactive accessory inventory to load, includes input interface, output interface, assembling fittings in the interactive accessory library With detection accessory；
The pending data is input to the interactive accessory library for calling the input interface by input module；
The generation module, is specifically used for: utilizing the assembling fittings, the pending data is assembled, generate default The structural body object of structure；
The judgment module, is specifically used for:
The detection accessory is called, the output object of the last one analytical unit in analytical unit combination is detected, Judge whether testing result meets output condition；
Output module, for from the output object of the last one analytical unit obtain processing result after, described in calling Output interface exports the processing result.
15. device according to claim 14, which is characterized in that further include deleting accessory, institute in the interactive accessory library State device further include:
Removing module deletes the output object of the last one analytical unit for calling the deletion accessory.
16. device according to claim 9, which is characterized in that described device further include:
Unload module, in the output content combined according to the analytical unit, after obtaining processing result, unloading is described to divide Analyse the analytical unit in unit combination.
17. a kind of electronic equipment, which is characterized in that including processor and memory；
Memory, for storing computer program；
Processor when for executing the program stored on memory, realizes any method and step of claim 1-8.
CN201711470644.7A 2017-12-29 2017-12-29 Data processing method, device and equipment based on streaming framework Active CN110007967B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711470644.7A CN110007967B (en) 2017-12-29 2017-12-29 Data processing method, device and equipment based on streaming framework

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711470644.7A CN110007967B (en) 2017-12-29 2017-12-29 Data processing method, device and equipment based on streaming framework

Publications (2)

Publication Number Publication Date
CN110007967A true CN110007967A (en) 2019-07-12
CN110007967B CN110007967B (en) 2022-05-06

Family

ID=67164072

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711470644.7A Active CN110007967B (en) 2017-12-29 2017-12-29 Data processing method, device and equipment based on streaming framework

Country Status (1)

Country Link
CN (1) CN110007967B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111611236A (en) * 2020-05-28 2020-09-01 宁波和利时智能科技有限公司 Data analysis method and system
CN112433866A (en) * 2020-11-25 2021-03-02 杭州视洞科技有限公司 Storm-DRPC-based face recognition performance optimization method
WO2021147289A1 (en) * 2020-01-22 2021-07-29 北京百度网讯科技有限公司 Stream computing job processing method, stream computing system, and electronic device
CN113282246A (en) * 2021-06-15 2021-08-20 杭州海康威视数字技术股份有限公司 Data processing method and device

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102542047A (en) * 2011-12-28 2012-07-04 奇智软件（北京）有限公司 Task logic library generating method, task logic library generating system, data processing method and data processing system
CN102546247A (en) * 2011-12-29 2012-07-04 华中科技大学 Massive data continuous analysis system suitable for stream processing
CN105260405A (en) * 2015-09-22 2016-01-20 北京云知声信息技术有限公司 Web crawler method and device
CN106201443A (en) * 2016-07-27 2016-12-07 福建富士通信息软件有限公司 A kind of method and system based on the Storm how concurrent written document of streaming Computational frame
US20170083378A1 (en) * 2015-09-18 2017-03-23 Salesforce.Com, Inc. Managing processing of long tail task sequences in a stream processing framework
CN106777009A (en) * 2016-12-07 2017-05-31 北京奇虎科技有限公司 Smart machine and task scheduling perform control method and device
CN107026851A (en) * 2017-03-22 2017-08-08 西安电子科技大学 A kind of real-time system guard method based on stream data processing
CN107229747A (en) * 2017-06-26 2017-10-03 湖南星汉数智科技有限公司 A kind of large-scale data processing unit and method based on Stream Processing framework

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102542047A (en) * 2011-12-28 2012-07-04 奇智软件（北京）有限公司 Task logic library generating method, task logic library generating system, data processing method and data processing system
CN102546247A (en) * 2011-12-29 2012-07-04 华中科技大学 Massive data continuous analysis system suitable for stream processing
US20170083378A1 (en) * 2015-09-18 2017-03-23 Salesforce.Com, Inc. Managing processing of long tail task sequences in a stream processing framework
CN105260405A (en) * 2015-09-22 2016-01-20 北京云知声信息技术有限公司 Web crawler method and device
CN106201443A (en) * 2016-07-27 2016-12-07 福建富士通信息软件有限公司 A kind of method and system based on the Storm how concurrent written document of streaming Computational frame
CN106777009A (en) * 2016-12-07 2017-05-31 北京奇虎科技有限公司 Smart machine and task scheduling perform control method and device
CN107026851A (en) * 2017-03-22 2017-08-08 西安电子科技大学 A kind of real-time system guard method based on stream data processing
CN107229747A (en) * 2017-06-26 2017-10-03 湖南星汉数智科技有限公司 A kind of large-scale data processing unit and method based on Stream Processing framework

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
VASANTH TOVINKERE.ETC: ""Flow Graph Designer: A Tool for Designing and Analyzing Intel® Threading Building Blocks Flow Graphs"", 《2014 43RD INTERNATIONAL CONFERENCE ON PARALLEL PROCESSING WORKSHOPS》 *
南海京: ""一种基于STORM的交通流数据实时处理系统设计与实现"", 《中国优秀博硕士学位论文全文数据库(硕士)工程科技Ⅱ辑》 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021147289A1 (en) * 2020-01-22 2021-07-29 北京百度网讯科技有限公司 Stream computing job processing method, stream computing system, and electronic device
CN111611236A (en) * 2020-05-28 2020-09-01 宁波和利时智能科技有限公司 Data analysis method and system
CN112433866A (en) * 2020-11-25 2021-03-02 杭州视洞科技有限公司 Storm-DRPC-based face recognition performance optimization method
CN113282246A (en) * 2021-06-15 2021-08-20 杭州海康威视数字技术股份有限公司 Data processing method and device
CN113282246B (en) * 2021-06-15 2023-07-04 杭州海康威视数字技术股份有限公司 Data processing method and device

Also Published As

Publication number Publication date
CN110007967B (en) 2022-05-06

Similar Documents

Publication Publication Date Title
CN110007967A (en) A kind of data processing method based on streaming frame, device and equipment
CN109034660B (en) Method and related device for determining risk control strategy based on prediction model
CN109145784B (en) Method and apparatus for processing video
CN104809062B (en) A kind of method of testing and system of artificial intelligence response system
CN103649907A (en) Registration and execution of highly concurrent processing tasks
CN107153584A (en) Method for detecting abnormality and device
CN106649120A (en) Data acquisition method, and data analysis method and system
CN112163468A (en) Image processing method and device based on multiple threads
CN112463859B (en) User data processing method and server based on big data and business analysis
CN110188717A (en) Image acquiring method and device
CN106528281A (en) Satellite telemetry data offline unified rapid processing system
CN109408345A (en) Operate replay method, device, computer equipment and storage medium
CN112906554B (en) Model training optimization method and device based on visual image and related equipment
CN110490079A (en) Inspection data processing method, device, computer equipment and storage medium
CN110245696A (en) Illegal incidents monitoring method, equipment and readable storage medium storing program for executing based on video
CN110826683A (en) Method and device for predicting user behavior occurrence probability
CN106599222B (en) Method and equipment for processing logs in streaming parallel
CN107220169A (en) A kind of emulating server returns to the method and apparatus for customizing data
CN107563257B (en) Video understanding method and device
CN105487982B (en) API method of testings and system
CN103927341B (en) A kind of method and device for obtaining scene information
CN110795993A (en) Method and device for constructing model, terminal equipment and medium
CN110599581A (en) Image model data processing method and device and electronic equipment
US6735751B2 (en) False path detecting apparatus and a false path detecting method in which a usage amount of memories is smaller and a process is carried out at higher speed, and that program
CN104462422A (en) Object processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant