CN116074293A - Real-time video analysis method based on stream computing - Google Patents

Abstract

The invention discloses a real-time video analysis method based on stream computing, which is applied to the field of video analysis and comprises the following steps: acquiring a video, and inputting the video into a Flink model; the video is decoded into a data stream in the link model; the data flow passes through a water line and then calls an asynchronous function; and after the data passes through the time window, outputting the data to a target receiver. The scheme can solve the problems of poor expandability and high real-time video analysis delay of the existing video analysis function.

Description

Real-time video analysis method based on stream computing

Technical Field

The invention belongs to the field of video analysis, and particularly relates to a real-time video analysis method based on stream computing.

Background

In recent years, technologies such as network transmission, video coding compression, computer vision and the like have all made great breakthrough, and intelligent monitoring systems have also been rapidly developed. The intelligent video monitoring system introduces various intelligent algorithms on the traditional video monitoring system, and can realize the functions of abnormal event detection, intrusion detection and the like by analyzing video images, and automatically send alarm information. Compared with the traditional monitoring system, the intelligent monitoring system can improve the monitoring efficiency, thereby reducing the working pressure of monitoring personnel.

With the rapid growth of monitoring video data, the defects of analyzing and processing the video in a traditional way begin to appear, and the defects mainly appear in the following aspects:

1. most of traditional intelligent video monitoring systems are based on single architecture, and video acquisition, analysis and processing and storage modules are highly coupled, so that reliability is low;

2. many intelligent algorithms and functions of the current video monitoring equipment are based on embedded type, and the intelligent video analysis function which can be used by the monitoring video is limited by the hardware condition of the equipment, so that the diversified requirements cannot be met, and the expansibility is poor;

3. because the unstructured data of the video image is complex to process, the processing time of the image processing algorithm is relatively long, so that the video output after analysis has high delay and low frame rate.

Disclosure of Invention

The embodiment of the invention aims to provide a real-time video analysis method based on stream computing, which can solve the problems of poor expandability and high real-time video analysis delay of the existing video analysis function.

In order to solve the technical problems, the invention is realized as follows:

in a first aspect, an embodiment of the present invention provides a real-time video analysis method based on streaming computation, including:

acquiring a video, and inputting the video into a Flink model;

the video is decoded into a data stream in the link model;

the data flow passes through a water line and then calls an asynchronous function;

and after the data passes through the time window, outputting the data to a target receiver.

Optionally, before the video is acquired, the method further includes: and designating monitoring equipment and calling a real-time video analysis interface exposed to the outside.

Optionally, the video is RTSP video.

Optionally, the video is decoded into a data stream in the link model, specifically including:

decoding the video into a sequence of successive image frames;

preprocessing the image frame sequence and generating a data stream.

Optionally, the preprocessing the image frame sequence and generating a data stream specifically includes:

preprocessing the image frame sequence and entering a frame window;

and uniformly processing the image frame sequences entering the same frame window to generate a data stream.

Optionally, after the video is decoded into a data stream in the link model, the method further includes:

and carrying out window division on the data stream.

Optionally, the data flow calls an asynchronous function after passing through a water line, which specifically includes:

assigning an event timestamp and a water line generation mechanism to the data stream;

and triggering a window function and calling an asynchronous function under the condition that the data flows all exceed the water line.

Optionally, the calling an asynchronous function specifically includes: and calling an external image processing algorithm service interface through the Http request, and putting a returned result into the data stream, wherein the returned result is a marking Box or log record.

Optionally, the data stream passing time window is that the data streams are arranged according to time sequence.

Optionally, the outputting to the target receiver includes:

for the data stream which needs to be drawn and displayed in real time, the data stream in RTMP format is provided for the client to play;

and for the notification prompt to be sent in real time, sending the returned result to the client through the WebSocket.

The invention has the advantages that the invention serves the image processing algorithm needed by video analysis, and solves the problem of data timing disorder in the asynchronous calling process by combining the mechanism methods of asynchronous calling, dynamic setting parallelism, time window, water line, frame window and the like. The patent can effectively reduce the delay of video analysis, improve the video frame rate, increase the expandability of video analysis, and support the calling of algorithm service with inter-frame dependence.

Drawings

Fig. 1 is a flow chart of a real-time video analysis method based on stream computing according to an embodiment of the present invention;

FIG. 2 is a flow chart of another method for real-time video analysis based on streaming computing according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of a real-time video analysis method based on stream computation according to an embodiment of the present invention.

The achievement of the object, functional features and advantages of the present invention will be further described with reference to the embodiments, referring to the accompanying drawings.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present invention may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type, and are not limited to the number of objects, such as the first object may be one or more. It should be understood that, in various embodiments of the present disclosure, the size of the sequence number of each process does not mean that the execution sequence of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

It should be understood that in this disclosure, "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements that are expressly listed or inherent to such process, method, article, or apparatus.

It should be understood that in this disclosure, "plurality" means two or more. "and/or" is merely an association relationship describing an association object, and means that three relationships may exist, for example, and/or B may mean: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship. "comprising A, B and C", "comprising A, B, C" means that all three of A, B, C comprise, "comprising A, B or C" means that one of the three comprises A, B, C, and "comprising A, B and/or C" means that any 1 or any 2 or 3 of the three comprises A, B, C.

It should be understood that in this disclosure, "B corresponding to a", "a corresponding to B", or "B corresponding to a" means that B is associated with a from which B may be determined. Determining B from a does not mean determining B from a alone, but may also determine B from a and/or other information. The matching of A and B is that the similarity of A and B is larger than or equal to a preset threshold value.

As used herein, "if" may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to detection" depending on the context.

The method for analyzing real-time video based on stream computation according to the embodiment of the present invention is described in detail below with reference to fig. 1 through a specific embodiment and an application scenario thereof. The embodiment of the invention provides a real-time video analysis method based on stream computing, which comprises the following steps:

s101: acquiring a video, and inputting the video into a Flink model;

s102: the video is decoded into a data stream in the link model;

s103: the data flow passes through a water line and then calls an asynchronous function;

s104: and after the data passes through the time window, outputting the data to a target receiver.

In an embodiment of the present solution, in order to decouple the video analysis algorithm from the platform, the algorithm needs to be converted from a traditional local call mode to a served mode, and the API of the algorithm is called by means of HTTP requests or RPCs.

The Flink model may also be referred to as a Flink real-time video analysis pipeline.

Optionally, before the video is acquired, the method further includes: and designating monitoring equipment and calling a real-time video analysis interface exposed to the outside. The interface will then submit and execute the Flink real-time video analysis pipeline task.

Optionally, the video is RTSP video. The RTSP video stream of the device is used as input data Source to enter a Flink real-time video analysis pipeline task.

Referring to fig. 2, another flow chart of a real-time video analysis method based on streaming computing according to an embodiment of the present invention is shown.

Optionally, the video is decoded into a data stream in the link model, specifically including:

decoding the video into a sequence of successive image frames;

preprocessing the image frame sequence and generating a data stream.

Optionally, the preprocessing the image frame sequence and generating a data stream specifically includes:

preprocessing the image frame sequence and entering a frame window;

and uniformly processing the image frame sequences entering the same frame window to generate a data stream.

In one embodiment of the scheme, the video enters a link real-time video analysis task, is decoded into a continuous image frame sequence, and the preprocessing process is mainly used for adapting to interface parameters of video analysis algorithm service and adjusting the size and format of the converted image according to the requirement. The preprocessed data stream is divided into a number window (CountWindow) as needed, and then an event time stamp and a water line generation mechanism are assigned to the data stream.

Optionally, after the video is decoded into a data stream in the link model, the method further includes:

and carrying out window division on the data stream.

In one embodiment of the scheme, for the algorithm service with inter-frame dependence, the invention designs a frame window, and is convenient for uniformly processing all frames in one window.

Optionally, the data flow calls an asynchronous function after passing through a water line, which specifically includes:

assigning an event timestamp and a water line generation mechanism to the data stream;

and triggering a window function and calling an asynchronous function under the condition that the data flows all exceed the water line.

In one embodiment of the present solution, the process of invoking the image processing algorithm is changed from a conventional synchronous mode to an asynchronous mode. In the processing of the asynchronous call algorithm, it is assumed that the average response time of processing each frame of data is

The concurrency of asynchronous processing is P, then in order for the average interval s' between frames to be less than or equal to a given value s, it is necessary to satisfy:

while because the response time per frame processing is not fixed, there is T _i ＞T _i-1 Possibly resulting in video stuck, a timeout time for asynchronous processing needs to be set>

The frame with overtime processing is directly returned to the original frame, so that the smoothness of the video is ensured.

But using an asynchronous approach results in out-of-order processing of the data, so the present invention uses a pipeline and time window mechanism to solve the out-of-order problem caused by the asynchronization. Recording event time stamp of each event (data of each frame), generating water line periodically, wherein the water line is a time stamp, and the algorithm is t _wm ＝t _max -t _de l _ay Wherein t is _max T is the maximum timestamp of the current event _delay For maximum allowable delay, the water line represents t _wm When the triggering condition of the time window is met, a window function is triggered, and the data in the window are ordered according to the event time stamp. For data that has not arrived beyond the maximum delay time, the discard is defaulted.

Under this pipeline, the delay and frame rate (FPS) of the video is calculated as follows:

T _delay ＝T _d +T _b +T _p +T _a +T _t +T _wc +T _wt +T _e

wherein T is _delay Overall delay for the processed video; t (T) _d Time consuming for video decoding; t (T) _b Time consuming for picture preprocessing, such as converting a picture frame into data in a byte array or Base64 format; t (T) _p Time consuming servicing the algorithm; t (T) _e Time consuming for video encoding; t (T) _a Time consuming for result processing, such as picture format conversion, and operations of drawing a binding Box; t (T) _t Time consuming network transmissions; t (T) _wc Delay generated by a frame window is introduced when a section of frame sequence is processed uniformly; t (T) _wt Delay generated by a time window is introduced to prevent frame disorder, namely the size of the time window; t (T) _wm Is the maximum allowable delay of the water line; p (P) _i Is the ithThe concurrency quantity supported by the algorithm service instance; p (P) _b Is the concurrency number of the pretreatment process; p (P) _a The concurrency number of the result processing process; f (F) _out Is the FPS for the output video stream.

If the output video is required to meet the requirements of smoothness at least, F is required _out Not less than 25, i.e

The parallelism set for asynchronous calls in the actual process is based on this. And for the processing result of the real-time rendering algorithm is not required, the FPS limitation is avoided, and the parallelism can be properly reduced.

Optionally, the calling an asynchronous function specifically includes: and calling an external image processing algorithm service interface through the Http request, and putting a returned result into the data stream, wherein the returned result is a marking Box or log record.

Optionally, the data stream passing time window is that the data streams are arranged according to time sequence.

Optionally, the outputting to the target receiver includes:

for the data stream which needs to be drawn and displayed in real time, the data stream in RTMP format is provided for the client to play;

and for the notification prompt to be sent in real time, sending the returned result to the client through the WebSocket.

In one embodiment of the present solution, since the returned result is typically a binding Box or log record, in the step of processing the result, further processing needs to be performed on the result of the previous step according to different types of the returned result, such as drawing the binding Box on a corresponding image frame or directly recording the analysis result on the log. The processed data flows through the processing of time window (TimeWindow) to guarantee the order of the data. And finally, outputting the data to different target receivers according to different requirements. I.e. the manner in which the data sink is different according to the different algorithm types. For example: for a video stream which needs to be drawn and displayed in real time, encoding a frame into an RTMP format for a client to play; and for the notification prompt to be sent in real time, sending the result returned by the algorithm to the client through the WebSocket.

The invention has the advantages that:

firstly, an image processing algorithm required by video analysis is served, the expandability is improved, and the frame rate of an output video is improved through asynchronous calling and dynamic setting parallelism.

And secondly, introducing a time window and a water line mechanism to solve the problem of data timing disorder in the asynchronous calling process.

Furthermore, a frame window is introduced to support invoking algorithmic services with inter-frame dependencies.

And finally, designing a complete video analysis pipeline based on the stream computing engine Flink, and reducing video analysis delay.

Example two

Referring to fig. 3, a schematic structural diagram of a real-time video analysis device 30 based on streaming computation according to an embodiment of the present invention is shown.

The real-time video analysis device 30 based on stream computation provided by the embodiment of the invention comprises:

the acquiring module 301 is configured to acquire a video, and input the video into a link model;

a decoding module 302, configured to decode the video into a data stream in the link model;

a calling module 303, configured to call an asynchronous function after the data stream passes through a water line;

and the output module 304 is configured to output the data to the target receiver after the data passes through the time window.

Example III

The embodiment of the invention provides equipment and a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to invoke the instructions stored in the memory to perform the method of embodiment one.

In the embodiment of the invention, an image processing algorithm required by video analysis is served, and the problems of data timing disorder in the asynchronous calling process are solved by combining mechanism methods such as asynchronous calling, dynamic setting parallelism, time window, water level line, frame window and the like. The patent can effectively reduce the delay of video analysis, improve the video frame rate, increase the expandability of video analysis, and support the calling of algorithm service with inter-frame dependence.

Example IV

An embodiment of the present invention provides a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method of embodiment one.

In the embodiment of the invention, an image processing algorithm required by video analysis is served, and the problems of data timing disorder in the asynchronous calling process are solved by combining mechanism methods such as asynchronous calling, dynamic setting parallelism, time window, water level line, frame window and the like. The patent can effectively reduce the delay of video analysis, improve the video frame rate, increase the expandability of video analysis, and support the calling of algorithm service with inter-frame dependence.

The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.

The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.

Computer program instructions for carrying out operations of the present invention may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information for computer readable program instructions, which can execute the computer readable program instructions.

Various aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer readable program instructions may be provided to a processing unit of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Note that all features disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic set of equivalent or similar features. Where used, further, preferably, still further and preferably, the brief description of the other embodiment is provided on the basis of the foregoing embodiment, and further, preferably, further or more preferably, the combination of the contents of the rear band with the foregoing embodiment is provided as a complete construct of the other embodiment. A further embodiment is composed of several further, preferably, still further or preferably arrangements of the strips after the same embodiment, which may be combined arbitrarily.

It will be appreciated by persons skilled in the art that the embodiments of the invention described above and shown in the drawings are by way of example only and are not limiting. The objects of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been shown and described in the examples and embodiments of the invention may be modified or practiced without departing from the principles described.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present disclosure, and not for limiting the same; although the present disclosure has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present disclosure.

Claims (10)

1. A real-time video analysis method based on stream computation, comprising:

acquiring a video, and inputting the video into a Flink model;

the video is decoded into a data stream in the link model;

the data flow passes through a water line and then calls an asynchronous function;

and after the data passes through the time window, outputting the data to a target receiver.

2. The method of real-time video analysis according to claim 1, further comprising, prior to the capturing the video: and designating monitoring equipment and calling a real-time video analysis interface exposed to the outside.

3. The real-time video analysis method according to claim 2, wherein the video is RTSP video.

4. A real-time video analysis method according to claim 3, characterized in that the video is decoded into a data stream in the link model, comprising in particular:

decoding the video into a sequence of successive image frames;

preprocessing the image frame sequence and generating the data stream.

5. The method according to claim 4, wherein the preprocessing the image frame sequence and generating a data stream specifically comprises:

preprocessing the image frame sequence and entering a frame window;

and after uniformly processing the image frame sequences entering the same frame window, generating the data stream.

6. The real-time video analysis method of claim 5, wherein the video, after being decoded into a data stream in the link model, further comprises:

and carrying out window division on the data stream.

7. The method of real-time video analysis according to claim 6, wherein the data stream passes through a water line and then invokes an asynchronous function, comprising:

assigning an event timestamp and a water line generation mechanism to the data stream;

and triggering a window function and calling the asynchronous function under the condition that the data flows all exceed the water line.

8. The real-time video analysis method according to claim 7, wherein the calling an asynchronous function specifically comprises: and calling an external image processing algorithm service interface through the Http request, and putting a returned result into the data stream, wherein the returned result is a marking Box or log record.

9. The method of claim 1, wherein the data stream passing through the time window is the data stream being arranged according to a time sequence.

10. The method of real-time video analysis according to claim 8, wherein the outputting to the target receiver comprises:

for the data stream which needs to be drawn and displayed in real time, the data stream in RTMP format is provided for the client to play;

and for the notification prompt to be sent in real time, sending the returned result to the client through the WebSocket.

Images