FR3126166A1 - METHOD FOR ANALYZING VIDEO CONTENT AND DEVICE EXECUTING THE METHOD. - Google Patents

Abstract

The invention relates to a method for analyzing video content comprising a determination (S1) of a list of objects and/or events to be detected in one or more video contents, a determination (S2) of a list of resources necessary for the detection of said objects and/or events to be detected, a breakdown (S3) of the resources necessary into a plurality of execution modules of a processing algorithm, and an automatic calculation (S4) of an optimized route of information between said execution modules, so as to minimize the number of said execution modules required to operate detections of elements among said list of objects and/or events to be detected. The invention also relates to a video content analysis device configured to perform the method. Figure to be published with abstract: Fig. 2

Description

VIDEO CONTENT ANALYSIS METHOD AND DEVICE EXECUTING THE METHOD.

The present invention relates to the field of video stream analysis. The invention relates more particularly to the allocation of hardware and/or software resources necessary for object detection operations or event detection in video content, from video streams.

STATE OF THE PRIOR ART

Video stream analysis systems make it possible to carry out automatic detection of objects or events, in real time or even in a deferred manner, from video streams generated by image capture devices such as, for example, CCTV cameras. Such systems are particularly frequent and useful in urban areas to ensure, for example, the proper use of road traffic infrastructures, the proper application of vehicle parking rules, the safety of goods and people, etc. The applications of video surveillance are numerous and varied, and the examples above are not limiting. Thus, video surveillance systems are commonly used for parking management, to encourage respect for traffic lights and respect for the direction of traffic, for securing pedestrian paths or railway tracks, but also to ensure the safety of people. in places where traffic is very high, such as, for example, train stations, airports, or shopping malls. Such systems are also appreciated for their contribution to traffic management, crowd management, and more broadly for their ability to help predict and plan urban development or changes in urban infrastructure. The success of these automated shooting and analysis systems, combined with the ease of use offered by the distribution of "turnkey" applications, is such that the need for the number of shooting devices is always croissants. The requirements for hardware and software resources implemented for the automated analysis of the video streams produced have increased accordingly. These systems are complex and expensive and there is a need to optimize the allocation of hardware and/or software resources necessary for analysis operations for predetermined or evolving needs. The situation can therefore be improved.

The object of the invention is to propose a method for allocating useful resources to video content analysis operations aimed at guaranteeing that the detections to be made can be carried out, or in other words that no saturation as to the The use of resources cannot harm proper detection of an object or an event from one or more video contents.

To this end, a video content analysis method is proposed, executed in a video content analysis device, comprising:

- determine a list of objects and/or events to detect in one or more video contents,

- determine a list of resources necessary for the detection of the objects and/or events to be detected,

- breaking down the resources into a plurality of modules for executing a processing algorithm, each of the modules being configured to process all or part of one or more images extracted from one of the video contents, or information representative of a video content obtained from one or more of the images and at least one of the execution modules being configured to deliver information representative of a result of a detection to be operated, the method further comprising,

- automatically determine an optimized route of information between the runtime modules so as to minimize the number of runtime modules required to operate element detections among the list of objects and/or events to be detected.

Advantageously, the use of hardware and/or software resources is optimized to make it possible to operate a large number of detections while minimizing the risk of not detecting an object or an event sought in one or more video contents.

According to one embodiment of the invention, the steps of the method for determining a list of necessary resources and for breaking down the resources into a plurality of execution modules of a processing algorithm comprise the determination or the consideration of an index representative of a processing capacity of each of the execution modules.

Thus, the assignment (or in other words the distribution) to execution modules of a processing algorithm of the global resources to be used to operate the scheduled analyzes and detections is facilitated.

According to one embodiment, the method further comprises a step of determining a rate of resources used with regard to all the available resources of the video content analysis device and at least one step of executing a video content analysis process is conditioned on whether or not the rate of resources used exceeds a predetermined threshold value.

According to one embodiment of the invention, the predetermined threshold value is between 80 and 90% of the total resources available in the video content analysis device.

According to one embodiment of the invention, the optimized data path is implemented thanks to asynchronous messaging functions for distributed or concurrent applications, preferably according to a protocol of a so-called ZeroMQ type or a protocol of an MQTT type (defined according to the ISO/IEC 20922:2016 standard or one of its evolutions).

The invention also relates to a video content analysis device comprising a set of electronic circuits configured among which:

- a module configured to determine a list of objects and/or events to detect in one or more video contents,

- a module configured to determine a list of resources necessary for the detection of objects and/or events to be detected,

- a module configured to break down the resources into a plurality of modules for executing a processing algorithm, each of the modules being configured to process all or part of one or more images extracted from one of the video contents, or information representative of a video content obtained from one or more of the images and at least one of the execution modules being configured to deliver information representative of a result of a detection to be operated, the device for analysis of video content further comprising:

- a module configured to automatically determine an optimized path of information between the execution modules of an algorithm, so as to minimize the number of execution modules required to operate detections of elements among the list of objects and/ or events to detect.

The invention also relates to a video surveillance system comprising one or more video content analysis devices such as the video content analysis device described above.

Finally, the subject of the invention is a computer program product comprising program code instructions for executing the steps of the aforementioned method, when the program is executed by a processor, as well as a storage device comprising such a program product of computer.

The characteristics of the invention mentioned above, as well as others, will appear more clearly on reading the following description of at least one embodiment, said description being made in relation to the attached drawings, among which:

schematically illustrates a video surveillance system comprising a video content analysis device executing a video content analysis method according to one embodiment;

is a flowchart representing a video content analysis method according to one embodiment, executed in the video surveillance system already illustrated in the ;

is a diagram representing the architecture of the video content analysis device represented on the and configured to perform a method described in .

DETAILED DESCRIPTION OF EMBODIMENTS

There schematically illustrates a video surveillance system 10 comprising a device 100 for analyzing video content. The video content analysis device 100 is configured to operate a detection of predefined objects and/or events from video content delivered by image capture devices 102, 104, 106 and 108. The image capture 102, 104, 106 and 108 are respectively connected to the video content analysis device 100 via communication buses 103, 105, 107 and 109. According to one embodiment, the image capture devices images 102, 104, 106 and 108 are so-called "IP camera" cameras configured to be remotely controlled and to each deliver one or more video contents from a shooting device in the form of a video stream digital. The terms "digital video stream" here designate a succession of identifiable information having a link between them and being representative of one or more video contents (for example one or more films or video sequences) and whose processing (decoding) allows a restitution of one or more video contents. More broadly, a video stream can also comprise an audio stream encoding a sound environment around a camera. In this case, the video stream is more precisely an audio/video or audiovisual stream. Thus, each of the communication buses 103, 105, 107 and 109 comprises communication lines useful for transporting video streams and communication lines useful for configuration operations of the image capture device to which it is connected. The communication lines useful for transporting video streams can also be used to transport data related to the video streams, such as, for example, camera identifiers, camera movement control data, video stream timestamp. The camera movement control data make it possible, for example, to control a rotating camera in order to modify or adjust its field of vision. According to one embodiment, the communication lines provided for the transport of video content coming from an image capture device are also provided for the configuration of the image capture devices. According to this embodiment, a communication bus thus produced is bidirectional and a data transfer sequencing is carried out to interleave (or in other words to alternate) configuration operations of an image capture device and video content transport operations between the image capture device concerned and the video content analysis device 100.

According to one embodiment, the video streams carrying the video content each use an application-level communication protocol of the RTSP type (from the acronym for “Real-Time Streaming Protocol” and which means “real-time streaming protocol”).

The terms “video content” designate here all the visible and identifiable elements in a video sequence generated by an image capture device among the image capture devices 102, 104, 106 and 108. According to the example described, a video sequence corresponds to a shooting plan or else, for example, to a shooting plan for a determined magnification factor (zoom level). Indeed, some of the processing applied to video content can interpret successive shooting shots when a rapid modification of the captured image occurs (for example a modification of a zoom level, or a modification by isometric or homothety, due to rotational or translational movements of a capture device). In other words, a video sequence is a series of images corresponding to the visual field of an image capture device, i.e. the total area of space perceived by the camera, possibly temporally delimited by a rapid modification of shooting characteristics, and therefore of the image as captured. Thus a video sequence comprises only one shot if an image capture device generating it is not deactivated, except possibly when this device is configured to operate a rapid modification of shooting characteristics, such as previously described, which then constitutes a transition between the end of a video sequence and the start of another video sequence, according to the various processing algorithms applied to the sequence.

The video surveillance system 10 further comprises, connected to the device 100 for analyzing video content, an analysis control device 110 and an analysis results restitution device 112. These two devices 110 and 112 are respectively connected to the device 100 for analyzing video content via bidirectional communication buses 111 and 113. The video surveillance system 10 further comprises a storage device 114 for analysis results which allows the recording of analysis results of video content produced by the device 100 for analyzing video content. The analysis results recorded in the storage device 114 are, for example, notifications linked to the detection of an object or an event, in the form of detection logs, images illustrating the detections made, or even combinations of notifications and one or more images illustrating the detections notified. The storage device 114 is connected to the device 100 for analyzing video content via a communication bus 115. According to one embodiment, the communication buses 111 and 113 are bidirectional communication buses and the communication 115 is a one-way or two-way communication bus. Each of the communication buses 103, 105, 107, 109, 111, 113 and 115 can be of the wired type or else of the wireless type depending on the configuration, and more particularly according to the topology of the video surveillance system 10, or, in other words, according to the geographical distance between the various elements described and which make up the video surveillance system 10.

The analysis control device 110 is provided and configured to program detections to be operated by the video surveillance system 10 in the video content delivered to the video content analysis device 100 via the communication buses 103, 105, 107 and 109 , from information entered by a user, who is still commonly called the system operator. Thus, a user of the system can configure the video content analysis device 100 to program a detection of one or more objects and/or one or more events in one or more of the different video contents transmitted by the devices. image capture devices 102, 104, 106 and 108 connected to the video content analysis device 100. For example, an operator can program a detection of a pedestrian in a video content transmitted by the image capture device 102 Similarly, an operator of the analysis control device 110 can program a detection of an isolated piece of baggage in this same video content. These detections then correspond, according to these examples, to detections of objects. Similarly, it is possible to program a detection of an event related to one or more objects. It should be noted that a detection of an event includes here, by definition, the consideration of a temporality in addition to one or more facts (or states) relating to one or more objects. For example, the video surveillance system 10 is configured to allow the detection of luggage abandoned by a person, which constitutes an event related to objects that are the person and the luggage. This implies, for example, detection of a person (object) and baggage (object), initially traveling the same path in time and space, then being separated from each other. other. An event that characterizes the abandonment of luggage by a person can be broken down into an event of a person carrying luggage, followed by an event corresponding to the fact that this person physically separates from this luggage, then an event corresponding to an absence of recovery of this baggage by a person for a predetermined period.

The video content analysis control device 110 makes it possible to program detection of objects or events from among a predefined set of objects or events from one or more identifiable video contents.

More generally, the video surveillance system 10 makes it possible, thanks to the analysis control device 110, to detect a predefined object or a predefined event in a video content. A detection of an object is then, in other words, a detection of an element likely to be identified as such (as this object) thanks to the shapes and visual characteristics (texture, color, etc.) that it presents to an image capture device (a camera, for example) and therefore to a form that it presents in a video content. By way of non-limiting examples, such an object can be, for example, a face, a pedestrian, a car, a bus, a road sign, luggage, a hat, a pair of glasses, a bottle of milk, etc. A detection of an event is a detection of a fact relating to one or more objects in a video content, and which therefore includes a temporality. For example, an event thus defined can be an appearance of an object in a video content, a displacement of an object in a video content, a disappearance of an object in a video content, a relative displacement of an object by relative to another object, in video content (according to the example of luggage carried by a person, for example), a variation in the displacement of an object relative to another object in video content (approach, distance, etc. …). Thus, and in other words, non-limiting examples of events likely to be searched for and detected in video content are: appearance of luggage, disappearance of luggage, appearance of a pedestrian, disappearance of a pedestrian, transport of luggage by a pedestrian, deposit of luggage by a pedestrian, gripping of luggage by a pedestrian, immobility of luggage for a predetermined time, but also , according to other non-limiting examples, a car crossing a red light, a vehicle entering a one-way traffic street, against the flow of traffic, a collision between two vehicles, a football football crossing a sideline of a football stadium, a candidate talking to a third person during a school exam monitored by video surveillance, etc.

Advantageously, object detection operations in video content can be carried out using so-called computer vision techniques based on the use of neural networks trained from samples. images during a learning phase.

According to one embodiment, an object detection module comprises a software or hardware implementation of a deep artificial neural network or of the DCNN type (from the acronym “Deep Convolutional Neural Network”). Such a DCNN module can be made up of a set of many artificial neurons, of the convolutional type or of the perceptron type, and organized by successive layers connected to each other. Such a DCNN module is classically inspired by a simplistic model of the functioning of a human brain where many biological neurons are connected to each other by axons.

For example, a module called YOLOv4 (from the English acronym “You Only Look Once version 4”) is a DCNN-type module which makes it possible to operate object detection in images, and which is said to be “one-step” ( or even One-Stage), ie whose architecture is composed of a single module of combined proposals of rectangles framing objects (“bounding boxes”) and classes of objects in the image. In addition to the artificial neurons previously described, the YOLOv4 module uses functions known to those skilled in the art such as batch normalization (or “ batch normalization ”), regularization by cutting blocks of pixels (or “ dropblock regularization ”, weighted residual connections between layers (or “ weighted residual connections ”) or even a step of non-maximal suppression (or “ non-maximal suppression ” in English) which eliminates the redundant proposals of detected objects.

According to one embodiment, the object detection module has the possibility of predicting a list of objects present in one or more images of one or more video contents by providing, for each object, a rectangle framing the object under form of coordinates of points defining the rectangle in the image, the type or class of the object from among a predefined list of classes defined during a learning phase, and a detection score representing a degree of confidence in the detection thus operated.

The analysis results restitution device 112 is a device provided and configured to operate a restitution of the results of one or more analyzes carried out by the video surveillance system 10 by means of the analysis device 100 of video content. Thus the analysis results restitution device 112 is provided to notify an operator of the detection of objects and/or events corresponding to the objects and/or events sought in the video content. According to one embodiment, the analysis results restitution device 112 displays an alert message for each detection carried out and which corresponds to a detection of an object and/or an event as programmed via of the analysis control device 110. According to one embodiment, the restitution of an alert message can be visual, audible, or even visual and audible and allows an operator to obtain information related to a detection carried out. For example, a notification of a detection via the analysis results restitution device 112 can be a short line of information, displayed on a screen, and comprising a summary of detailed information which can be accessed by selecting the line. The brief summary may include, for example, the type of detection made (object or event), a title of the detection (for example, crossing a red light on an urban traffic lane), a date and one hour.

According to one embodiment, the video content analysis device 100 is configured to determine a list of objects and/or events to be detected in one or more video contents from information entered by an operator, via the analysis control device 110. For example, the video content analysis device 100 is configured to detect an "left luggage" type event in video content transmitted from the image capture device 102, which device image capture 102 is positioned in the lobby of an airport. According to another example, the video content analysis device is configured to detect an event of the “ball out of touch” type in a video content transmitted from the image capture device 103, which image capture device 103 is positioned in a sideline of a football stadium. The video content analysis device 100 is further configured to determine, from the list of objects and/or events to be detected, entered or updated by an operator, a list of resources necessary for the detection of these objects and/or events to be detected, then to break down the resources into a plurality of execution modules of a processing algorithm, considering that each of the execution modules of a processing algorithm is configured to process all or part of one or more images extracted from one of the video contents, or else is configured to process information representative of a video content obtained from one or more of the images. The execution modules of a processing algorithm are also called “processors”. At least one of the execution modules is configured to deliver information representative of a result of a detection to be operated. In other words, this amounts to saying that the system is configured to use resources in the form of elementary image processing modules which are advantageously allocated dynamically to make it possible to operate all of the successive image processing operations required for analyze the video content and detect the various objects and/or events to be detected. For example, an elementary module for executing an image processing algorithm can be a module for detecting the presence of an object in a given image, extracted from video content or pointed to in video content. More precisely, an example of such an elementary module for executing an algorithm can be a module for detecting the presence of an object in an image identified in a given video content. Such a module can be defined with input data which are, for example, an identifier of a timestamped image in video content (or a link to it), an identifier of an object to be searched for in this image and output data from this module are then an indicator of the presence of the object sought, the coordinates of a precise point (reference point) of the object in the image and the dimensions of a frame delimiting the object around the precise point of the object. A reference point is for example the center of the object or the center of a median or the bottom leftmost point of a frame bordering the object, etc.

By way of example, an elementary module for executing object detection processing can be configured and used to detect the presence of an object of the "luggage" type in an image extracted from a video content originating of the image capture device 102, the image being time-stamped and captured at a specific date and time (for example July 12, 2021 at 12 h 16 min and 34 s), with a time-stamping precision of order of a second, for example.

According to a similar operation, this same object detection module can be used to detect the presence of a "pedestrian" type object in the same image or in another image, or even in another video content, at a moment temporally near or far from the search for baggage. In addition, an elementary algorithm execution module can be defined for the detection of an event related to two objects in the same image. For example, an elementary module for executing an event detection algorithm can be configured to detect an identical or almost identical displacement of two objects in the same image, the presence of which has already been detected by prior processing (and by an execution module of a different algorithm). Thus, such an elementary processing module can have as input data a link to an image, respective coordinates of reference points of each of the two previously detected objects, and a predefined time interval (for example, a start time and an analysis duration), and an output datum of this elementary module would then be an indicator of whether or not the displacements of the two objects match. Advantageously, the video content analysis device comprises a library of processing algorithm execution modules (also sometimes called processors) comprising numerous elementary algorithm execution modules using as input data all or part of 'an image extracted from a video content or even using as input data information representative of one or more elements contained in an image (for example, the coordinates of a reference point of a detected object, in a image, and the dimensions of a frame delimiting this object, with respect to the reference point).

Advantageously, the video content analysis device 100 is configured to automatically calculate or determine, from a list of detections to operate and from a list of elementary modules for executing an algorithm, an optimized route of information between the execution modules, so as to minimize the number of execution modules required to operate the detections of programmed elements among the list of objects and/or events to be detected.

A processing algorithm execution module can be implemented in software form or in hardware form. When it is implemented in software form, an execution module of an implemented processing algorithm can be a new instance of such an execution module, executed in a random access memory of the content analysis device 100 video, under the control of a control unit of the device 100 for analyzing video content. When it is implemented in hardware form, the execution module can be an ^nth instance of a determined module, implemented in a finite number of instances in a logic circuit, for example, of the video content analysis device 100, and configured to perform detection. Obviously, the resources are available in a finite number, whether due to the hardware architecture and/or the size of the RAM of the device 100, and it is therefore advisable to optimize their use so as not to risk an absence detection of an object or an event due to the temporary unavailability of a resource or to be able to increase the number of detections to be carried out with regard to a given architecture (and therefore a given number of resources ). This is why an automatic calculation, from a list of detections to operate and from a list of elementary modules for executing an algorithm, of an optimized path of information between the execution modules, of so as to minimize the number of execution modules required to operate detections of elements among the list of objects and/or events to be detected, makes it possible to optimize the allocation of resources and to limit as much as possible a risk of absence of detection in the presence of an object or an event to be detected in a video content.

According to one embodiment, the determination of a list of necessary resources and the decomposition into a plurality of execution modules of a processing algorithm uses a system of indices representative of a processing capacity for each of the modules of execution of an algorithm as defined in the library of elementary modules for the execution of a processing algorithm of the device 100 for analyzing video content. Thus a coefficient Xi expressed in processing capacity units is associated with each elementary module (indexed i) of execution of an algorithm as defined in the library of elementary modules of execution of an algorithm of the analysis device 100 video content. A coefficient Xi can also be expressed in units of capacity per unit of time. It should be noted that the resource requirements are not linear, i.e. the amount of resources required to detect ten events in a video content is not equal to ten times the amount of resources required to a single detection. In other words, the relationship between the resources needed to detect a number of objects or events in one or more video contents and this number of objects or events is not necessarily linear.

Advantageously, the device 100 for analyzing video content includes a so-called “supervisor” control module making it possible to define in a recurring manner a rate of resources used with regard to all the available resources of the device 100 for analyzing video content. It is then possible to operate a conditioned execution of a video content analysis process depending on whether or not said rate exceeds a predetermined threshold value. For example, the predetermined threshold value can be between 60% and 95% of the total resources available in the analysis device 100, preferably between 80 and 90% of these resources. In the event that the overall resources are insufficient to carry out all of the detections programmed by an operator, depending on other parameters of the video surveillance system, such as, for example, the number of image capture devices or the resolution video content images, the video content analysis device 100 is configured to send an alert to the operator, via the analysis control device 110 or via the analysis results restitution device 112 Advantageously, a module for executing a processing algorithm can be put into operation conditionally according to a level of use of the available resources, also called “saturation level” of the resources. Thus, a processing carried out by an algorithm execution module can be executed conditionally according to the level of use of the resources of the analysis device 100.

According to one embodiment, the optimized path of the data between the different analysis modules is implemented by using asynchronous messaging functions for distributed or concurrent applications, preferably according to a protocol of a type called ZeroMQ also commonly called ZMQ, a protocol of an MQTT type or another equivalent protocol. For this purpose, data transfers (also called here more broadly information) are operated according to a ZMQ, MQTT protocol, or according to another equivalent protocol between output registers of the execution modules of a processing algorithm and registers of data from execution modules of a processing algorithm, as soon as necessary, and according to, for example, a so-called “ round robin ” algorithm.

According to one embodiment, a characteristic is defined for each module of the library of elementary modules, so as to be able to determine whether a module is of a so-called “ stateful ” type or of a type called “ stateless ”.

A module defined as being “stateful” may require the use of information from or representative of previous processing, carried out by this same module, on all or part of one or more video contents or information representative of video content. Such a module comprises an internal memory capable of recording information representative of a state of the module at a given instant. This memory can, for example, be used to recreate an instance of this module when processing is resumed following an error (sometimes called “error recovery”). Such an error may be linked to the accidental unavailability of a hardware or software resource, for example following a breakdown, a "bug", or an interruption of a network connection. By way of example, a module tracking a "car" object in several successive images of video content can request previous positions of several "car" objects in previous images from the same video content in order to then associate from positions of “car” objects in a new image to “car” type objects tracked by the system in previously processed images.

Conversely, a module defined as being of the “stateless” type is only based on input data presented to it and which relate, for example, to all or part of one or more video contents or information representative of video content. By way of example, a module performing a detection of “car” type objects in an image may need only video content corresponding to this image but not video content corresponding to previous images.

According to one embodiment, an optimized path of information between the different execution modules is calculated by the control module, called "supervisor" so as to minimize the number of execution modules required to operate element detections from a list of objects and/or events to be detected on all or part of one or more video contents or information representative of a video content.

According to this embodiment, the so-called supervisor control module determines, to do this, a sequence of image processing operations to be carried out and respective configuration parameters to operate detections of elements among the list of objects and/or or events to detect.

The supervisor also determines a series of module types, each identified as being “stateful” or “stateless” and to which is associated a coefficient Xi as described previously and expressing a processing capacity associated with this module, necessary to carry out image processing operations.

The supervisor also defines the current use of the processing capacities of all the modules present in the device 100 for analyzing video content.

The supervisor can then determine the list of execution modules identified as being “stateful” existing and which have sufficient processing capacity available to carry out all the processing of a given type on one or more video contents or on representative information. of video content. The supervisor then selects, for each video content to be processed or each group of information representative of a video content to be processed, the execution module that performs the processing from a list of available execution modules using an algorithm called “ of the turnstile”. It should be noted that video content decomposed into several sub-streams and processed in several modules during a previous processing must be merged into a single stream and reordered to be processed by a module defined as “stateful”.

A distribution of detection tasks according to the turnstile algorithm amounts to distributing a first task (detection) to be processed to a first execution module in a list of available execution modules, then a second task to be processed to a second module execution in this list, and so on. The supervisor manages one or more dynamically updated allocation tables in a dedicated working memory, which is preferably internal or quick to access. The allocation tables make it possible to make the link between the tasks to be performed and the execution modules in charge of carrying them out.

In particular, when new detection tasks are to be performed, the allocation table(s) are updated accordingly.

In a similar way, the supervisor determines a list of existing execution modules defined as “stateless” and which can carry out all the processing of a given type applied to one or more video contents or to information representative of video content. The supervisor distributes the incoming flows, for a type of module, between each execution module in an equitable manner and so that the use of the processing capacity is best distributed between the modules of this type, so that each module has the least processing to be done. A video content can be divided into several streams, distributed between several processing modules, the operations to be carried out in an execution module defined as being “stateless” being temporally independent.

In a software embodiment, if the list of available execution modules defined as “stateful” does not contain enough modules to operate detections of elements among a list of objects and/or events to be detected in all or part of one or more video contents or in information representative of a video content, then the supervisor instantiates one or more new modules within the limit of the total resources of the device 100 for analyzing video contents. The supervisor then adds this new module (this new module instance) to the list of available modules to then determine, for each operation, the execution module(s) selected by the turnstile algorithm to process one or more video content or information representative of such content.

Similarly, if the list of available execution modules defined as “stateless” does not contain enough processing modules to perform element detections among a list of objects and/or events to detect in any or part of one or more video contents or in information representative of a video content, then the supervisor instantiates one or more new modules within the limit of the total resources of the device 100 for analyzing video contents.

For example, for detection of an event of the “parking of a bicycle” type, a video content can be analyzed by a sequence of execution modules comprising one or more so-called “stateless” modules for detecting an object “bicycle” in an image, one or more so-called “stateful” modules for tracking an “bicycle” object in a sequence of consecutive images of video content and one or more so-called “stateful” modules of a parking rules. In this example, the video content could be distributed between different so-called “stateless” modules for detecting a “bicycle” object to speed up processing and will be gathered and ordered in a single stream to be then processed in a so-called “to state” of bike tracking and in a so-called “state” module of parking rules engine.

A rules engine is defined as a set of so-called “business” rules, i.e. defined at an application level. For example, a rules engine can define that a traffic incident (for example an infraction) takes place if a vehicle is moving in a direction opposite to the only direction of traffic authorized on a traffic lane intended for this vehicle, or even that a motor vehicle is traveling on a lane reserved for the circulation of vehicles without a motor.

According to one embodiment, the determination of a list of resources necessary for given processing operations, and the breakdown into a plurality of resources necessary for the execution of a processing algorithm, uses a system of indices representative of a target processing capacity, less than or equal to the processing capacity expressed by the coefficient Xi, for each of the execution modules of an algorithm as defined in the library of elementary execution modules of a processing algorithm. Thus, a coefficient Yi expressed in units of target processing capacity is associated with each elementary module (indexed i) for executing an algorithm as defined in the library of elementary modules for executing an algorithm of the device 100 d analysis of video content. The supervisor, during a modification of the list of objects and/or events to detect, or even at regular intervals, determines the real (current) use of the processing capacities of each execution module, and then instantiates one or several new modules within the limit of the total resources of the device 100 for analyzing video content, if the consumption of a module reaches the coefficient Yi. This mechanism advantageously makes it possible to anticipate the needs of the device 100 for analyzing video content in order to reduce the risk of temporary interruption in the processing of video content or in the processing of information representative of video content, the time of instantiation of new processing modules, in the event of an increase in processing capacity requirements. Indeed, the implementation of a new instance of an execution module may, depending on the type of module, require a delay of several seconds.

There is a flowchart illustrating a video content analysis method executed in a video content analysis device according to one embodiment. The method comprises an initial step S0 at the end of which all the elements of the video surveillance system 10 are initialized and are able to operate nominally. In particular, at the end of step S0, an operator of the video surveillance system 10 can configure the system by means of at least the analysis control device 110 by choosing one or more objects and/or one or more events to be detected, from a set of predefined objects or events, in the video content carried by the video streams originating from the image capture devices 102, 104, 106 and 108, so as to determine a list of objects and/or of events to be detected via the video surveillance system 10. The term “list of objects and/or events” is to be interpreted here broadly, and such a list of objects and/or events can comprise a single element , of object type, or even a single element of event type, in the applicative sense of the term “event”, since it was explained previously that an event relates to at least one object considered in relation to time. Thus, for example, an operator can select during a step S1, by using, for example, a user interface made available by the analysis control device 110, one or more objects to be detected, and/or one or several events to be detected, indicating for each of the objects and for each of the events, where applicable, in which of the video contents the object or the event must be detected. For example, an operator can program a detection of a crossing of a red traffic light, by a land vehicle, in a video content coming from the image capture device 106, during the night, or even whatever the time of day. According to another example, an operator can program the detection of parking in a reserved parking space, by an unauthorized vehicle, in a video content delivered from the image capture device 106. According to yet another example, an operator can program a detection of a marauder near a room containing sensitive goods. Thus, at the end of step S1, a list of objects and/or events to be detected from all or part of the video content has been determined. This determination of a list can be either a creation of a list, starting from an empty list, or an updating of a list following an addition or following a withdrawal from the list. of elements to be detected. During a step S2, the video content device 100 determines, on the basis of known criteria and/or parameters, a number of resources necessary to operate the detections to be carried out, programmed during the preceding step S1. The device 100 notably considers the processing capacity of each of the elementary processing algorithm execution modules with regard to the characteristics of the video streams transmitted and of the video contents which are contained in the video streams. For example, a detection module configured for the detection of an object in a video content comprising 30 images per second can operate an object detection on n video contents in a given time interval, taking into account its processing capacity ( n being a positive integer). The required execution modules are selected from a library of execution modules, also sometimes called "processor" or "processing processor". Processing characteristics specific to each of the elementary execution modules are known so as to be able to define and consider a processing capacity index X for each of the execution modules. For example, a processing capacity index X ₁ can be determined for an elementary module for executing a processing algorithm operating on video content comprising 30 images per second and another processing capacity index X ₂ can be determined for the same elementary execution module operating on a video content comprising 50 images per second. Thus, having regard to the processing capacity of each of the elementary modules for executing a processing algorithm, operating on video content whose characteristics are known and possibly also considering "real-time" constraints of the needs of execution, it is possible to determine a number of elementary resources, which also corresponds to hardware and/or software resources of the device 100 for analyzing video content.

During a step S3, the necessary resources are broken down into elementary modules taking into account the detection operations to be carried out, for example, according to the number of objects to be detected and the number of events to be detected, as well as according to the nature objects and/or events to detect. The processing execution modules can be used according to an assembly specific to the objects and/or events to be detected and to the elementary modules defined in the library. Thus, a detection of an event of the "fire in a vehicle" type can be broken down into a detection of a vehicle, a detection of a fire, and a similar movement of the "vehicle" and "fire" objects detected in the time. An elementary object detection module can therefore be used to detect a vehicle in an image, when the input data of this module indicates to it that the object to be detected is a vehicle, then the same elementary module (equipment, or even the same software instance of this module) can be used to detect a fire object in the same image as that in which it detected a vehicle object, and finally another elementary processing execution module provided for detecting a motion similarity can be used to detect that the vehicle object and the fire object move in the same way, that is to say in other words substantially at the same places in space, and at the same time. According to the example described above, two elementary modules for executing a processing algorithm are to be used. One is an object detection algorithm execution module, the other is a time proximity detection algorithm execution module. Of course, many runtimes can be defined and combined, including hierarchically, so that one runtime can use a plurality of "lower" level modules combined together.

By proceeding in this way, execution modules of a processing algorithm use data, or more generally information, resulting from a processing result of one or more other execution modules having carried out prior processing.

Advantageously and according to one embodiment, the data, also called information here, is transmitted from and to the execution modules necessary to operate programmed detections using an asynchronous messaging system for distributed or concurrent applications. According to one embodiment, the messaging system operates according to a so-called ZeroMQ type protocol or according to a so-called MQTT type protocol. Variants can of course be used by using other similar or equivalent protocols.

Cleverly, during a step S4, an optimized route of the information is calculated (or determined) so as to optimize the number of resources necessary and to address only the smallest possible number of hardware resources or to only execute that the smallest possible number of software instances of an execution module of a processing algorithm, if necessary, to meet the needs for programmed detection of the objects and/or events identified and configured during the step S1.

Advantageously, an algorithm of the so-called “ round robin ” type applied to the various execution modules required and to the various information that must pass between these modules makes it possible to optimize the number of resources and to avoid or to limit as much as possible, consequently, the risk of not detecting a sought-after element in a video content of a video stream.

There schematically illustrates an example of internal architecture of the device 100 for analyzing video content. The architecture shown can also represent the internal architecture of the analysis control device 110, of the analysis results restitution device 112, of the analysis results storage device 114, or of any of the image capture devices (camera) 102, 104, 106 and 108.

According to the hardware architecture example shown in , the device 100 for analyzing video content then comprises, connected by a communication bus 1000: a processor or CPU (Central Processing Unit) 1001 comprising graphics processing units called GPUs; a random access memory RAM (Random Access Memory) 1002; a read-only memory ROM 1003; a storage unit such as a hard disk (or a storage medium reader, such as an SD ("Secure Digital" in English) card reader 1004; at least one communication interface 1005 allowing the device 100 to analysis of video content to communicate with third-party devices such as image capture devices 102, 104, 106 and 108, or analysis control devices 110 or analysis results restitution devices 112.

Processor 1001 is capable of executing instructions loaded into RAM 1002 from ROM 1003, external memory (not shown), storage medium (such as an SD card), or a communication network. When the video content analysis device 100 is powered up, the processor 1001 is able to read instructions from the RAM 1002, and to execute them. These instructions form a computer program causing the implementation, by the processor 1001, of all or part of a method described in relation to the .

All or part of the method implemented in the device 100 for analyzing video content, or its variants described, can be implemented in software form by the execution of a set of instructions by a programmable machine, for example a DSP ("Digital Signal Processor” in English) or a microcontroller, or be implemented in hardware form by a dedicated machine or component, for example an FPGA (“Field-Programmable Gate Array” in English) or an ASIC (“Application-Specific Integrated Circuit” in English). In general, the video content analysis device 100 comprises electronic circuitry configured to implement the methods described in relation to itself as well as to remote third-party equipment, and with any other device involved in the analysis. execution of the video content analysis method described, aiming to operate one or more detections of objects and/or events in video content coming from the image capture devices 102, 104, 106 and 108. Although obviously, the device 100 for analyzing video content also comprises all the elements usually present in a system comprising a control unit and its peripherals, such as a power supply circuit, a power supply supervision circuit, one or several clock circuits, a reset circuit, input-output ports, interrupt inputs, bus drivers. This list is not exhaustive.

At least each of the modules of the analysis device 100 from the following list:

- module configured to determine a list of objects and/or events to detect in one or more video contents,

- module configured to determine a list of resources necessary for the detection of objects and/or events to be detected,

- module configured to break down the resources into a plurality of processing algorithm execution modules,

- module configured to automatically determine an optimized path of information between the different execution modules of a processing algorithm,

can be considered as an electronic circuit or a control unit performing information and input data manager or supervisor functions comprising input and output data registers, memories, and processing capabilities of these data, either on its own or in combination with related resources. Each of these modules can be composed of hardware and software elements in different proportions from one module to another. The term module is not limiting and can be translated more broadly by processor, circuit, control unit, management unit, or any other synonym well known to those skilled in the art in information and signal processing or in the field of computer vision. The aforementioned modules can use common resources between them.

The invention is not limited only to the embodiments described but relates more generally to any video content analysis method, executed in a video content analysis device, comprising a determination of a list of objects and/or of events to be detected in one or more video contents, a determination of a list of resources necessary for the detection of the objects and/or events to be detected, from the determined list, then a breakdown of said necessary resources into a plurality of execution modules of a processing algorithm where each of said execution modules is configured to process all or part of one or more images extracted from one of the video contents, or to process information representative of a content video, obtained from one or more of the images, and at least one of the execution modules is configured to deliver information representative of a result of a detection to be operated, and an automatic calculation of a optimized path of information between the aforementioned execution modules, aimed at minimizing the number of execution modules required to operate detections of elements among the determined list of objects and/or events to be detected. The invention also relates more generally to any device configured to implement and execute such a method.

Claims (8)

A video content analysis method, executed in a video content analysis device, the method comprising:

• determining (S1) a list of objects and/or events to be detected in one or more video contents,
• determine (S2) a list of resources necessary for the detection of said objects and/or events to be detected,
• decomposing (S3) said resources into a plurality of modules for executing a processing algorithm, each of said modules being configured to process all or part of one or more images extracted from one of said video contents, or representative information of video content, obtained from one or more of said images, and at least one of said execution modules being configured to deliver information representative of a result of a detection to be operated, said method being characterized in that it further comprises:
• automatically determining (S4) an optimized path of information between said execution modules so as to minimize the number of said execution modules required to operate detections of elements among said list of objects and/or events to be detected.

Video content analysis method according to claim 1, in which the steps (S2, S3) for determining (S2) a list of necessary resources and for decomposing (S3) said resources into a plurality of execution modules of a processing algorithm include determining or considering an index representative of a processing capacity of each of said execution modules. Method for analyzing video content according to one of Claims 1 and 2, in which is also included a step of determining a rate of resources used with regard to all the available resources of said content analysis device video and a step of conditioned execution of a video content analysis process according to whether or not said rate exceeds a predetermined threshold value. Method for analyzing video content according to any one of Claims 1 to 3, in which the said optimized data path is implemented thanks to asynchronous messaging functions for distributed or concurrent applications, preferably according to a protocol of a type called ZeroMQ or MQTT. Video content analysis device comprising electronic circuits including:

• a module configured to determine (S1) a list of objects and/or events to be detected in one or more video contents,
• a module configured to determine 5S2) a list of resources necessary for the detection of said objects and/or events to be detected,
• a module configured to break down (S3) said resources into a plurality of modules for executing a processing algorithm, each of said modules being configured to process all or part of one or more images extracted from one of said video contents, or information representative of a video content obtained from one or more of said images and at least one of said execution modules being configured to deliver information representative of a result of a detection to be operated, said device analysis being characterized in that it further comprises:
• a module configured to automatically determine (S4) an optimized path of information between said execution modules of an algorithm, so as to minimize the number of said execution modules required to operate detections of elements among said list of objects and/or events to detect.

Video surveillance system comprising one or more video content analysis modules according to the preceding claim. Computer program product characterized in that it comprises program code instructions for executing the steps of the method according to any one of claims 1 to 4, when said program is executed by a processor. Information storage medium comprising a computer program product according to the preceding claim.