RU2653322C1 - Method of displaying objects in sequence of images received from stationary video camera - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: method of displaying objects in a sequence of images received from a stationary video camera is proposed. According to the method, in the initial sequence of frames, objects are detected, for each of the detected objects, the trajectory of its movement is constructed. Further, the queue of trajectories of the objects' movement is formed, make up a schedule for showing the objects on which they plan a synthetic personnel formation, form the next synthetic frame, show the operator on the screen the generated synthetic frames. Before forming the queue of object trajectories, make the patching of the trajectories of fragment motion into a single trajectory of motion.

EFFECT: improving the quality of synthetic frames by detecting a single object, using the object tracking algorithm.

1 cl

Description

FIELD OF THE INVENTION

The invention relates to the field of video surveillance, and in particular to technologies for detecting objects in a sequence of images received from a stationary video camera.

In digital video surveillance systems, they accumulate an archive of video data that stores a sequence of images represented by video frames, or, more simply, frames. Each frame contains one image and an indication of the time of its registration. This archive is used to analyze events that occurred in the field of view of a stationary video camera. For this analysis, it is necessary to search for events of interest to the operator, displayed in the sequence of images stored in this archive. An event means the presence of certain objects, motionless or moving, their location, the trajectory of their movement, and other signs of interest to the operator. An object can be a person, animal, vehicle, for example, a car, and any objects moving in the field of view of a video camera.

The search for events of interest to the operator is complicated by the fact that the duration of viewing the accumulated archive can reach several thousand hours for one video camera. Therefore, one of the most important tasks of constructing video surveillance systems is to reduce the time of this search.

The invention may find application in various video surveillance systems, for example, in security systems, in analytical systems for marketing research, etc.

State of the art

The obvious way to search for objects is based on sequential viewing at a constant speed of the entire accumulated archive of video data, that is, viewing the entire sequence of images received from a stationary video camera. The disadvantage of this method is the highest possible time required for the search. On average, it will take a time equal to half the time to view the accumulated archive to search for the desired event.

The prior art technology for recognition and image processing, implemented using a device, method, computer-readable medium for extracting image data (see application US 2016/0148070 A1, G06K 9/46, publ. 05.26.2016). This application discloses technologies for identifying objects in an image by category, searching for similar areas of images by similarity of features. The disadvantages of this technology is the inability to track the movement of an object in video images.

The prior art also knows a method for tracking and segmenting an object in images (see application US 2011/0158474 A1, G06K 9/00, publ. 06/30/2011). This application discloses a method that includes steps for determining the initial contour for tracking an object in images and dividing the original contour into many contour sections, and also includes estimating the weighted length of each of the many contour segments and generating the desired contour by combining many contour segments with multiple edges of the image of the object using the estimated weighted length. The disadvantages of this technology is the inability to track the movement of an object in video images.

The prior art also knows a method for tracking objects in a scene on images during any events (see application US 2011/0135149 A1, G06K 9/00, published 09.06.2011). In this method, players in a team game are tracked on images received from cameras. It also monitors the movement of objects in video images. The disadvantages of this technology is the low accuracy of detecting the movement of objects.

The prototype of the proposed method is a method of searching for objects in a sequence of images obtained from a stationary video camera (see patent RU 2471231 C1, G06K 9/00, published on December 27, 2012).

According to this method, for each of the objects of interest to the operator, a trajectory of its movement is built, each point of which corresponds to the position of the object in the frame and the time moment of its registration. These trajectories are ordered, an updated schedule for displaying the number of objects specified by the operator is compiled, for which time points for each trajectory are automatically selected, a synthetic frames formation plan is created taking into account the conditions of permissible mutual screening of objects, synthetic frames corresponding to this plan are formed and shown to the operator.

At the same time, the visual quality of the image formed by the known prototype method is highly dependent on the quality of the metadata, i.e. on the quality of the detection algorithm and inter-frame tracking of objects (tracking algorithm). The application of the tracking algorithm is effective for viewing large archives by cameras, in the field of view of which at the same time there are a small number of objects: instead of hours, the operator can spend just a few minutes searching for the desired fragment.

However, the tracking algorithm disclosed in the prototype method does not always work perfectly, and some errors in the operation of the tracking algorithm lead to degradation of the quality of the algorithm itself. Such errors in the operation of the algorithm include, for example, the fact that tracking breaks some real objects into several objects (fragments).

The objective of the claimed invention is to eliminate at least one of the disadvantages of the prior art.

Disclosure of invention

The technical result of the claimed invention is to improve the quality of the synthetic frames shown to the operator due to the "correct" detection of a single object using the tracking algorithm of the object (tracking algorithm), increasing the speed of searching for the desired image in the sequence of images due to the fact that when showing some objects are combined, and the displayed objects become smaller, and thus the time required to view them is reduced.

This technical result is achieved due to the proposed method of displaying objects in a sequence of images received from a stationary video camera, which consists in the fact that:

- in the initial sequence of frames received from a stationary video camera and representing images and indications of the moments of time of their registration, objects of interest to the operator are detected;

- for each of the detected objects, a trajectory of its movement is built, each point of which corresponds to the position of the object in the frame and the time moment of registration of this frame, and the position of the object in the frame is represented by a set of pixels of the frame representing the image of this object;

- form a queue of trajectories of motion of detected objects;

- make up a schedule for showing detected objects, in which for the trajectory of each object indicate the point in time from which it will be shown on the screen;

- according to the specified schedule, a plan for the formation of synthetic frames is built, in which several objects can be shown simultaneously in positions that, in the general case, are registered at different points in time;

- form the next synthetic frame, in accordance with the specified plan;

- when forming a synthetic frame, it includes images of objects that, in accordance with the specified plan, should be shown in this synthetic frame at the same time, and the background on which they should be shown;

- show the generated synthetic frames to the operator on the screen, while in the queue of the trajectories of the movement of objects, all trajectories are ordered by increasing time of the appearance of the corresponding object in the field of view of the camera;

- the sequence number of the trajectory in an ordered set of trajectories is assigned to this trajectory and its corresponding object;

- initially only the first object is included in the display schedule of ordered objects, and then this schedule is updated, removing objects from it, the trajectories of which are shown in full, and adding new ones, and the number of objects in the schedule does not exceed the value specified by the operator;

- on the basis of the specified schedule, when it is created and each update is made up, the specified plan for the formation of synthetic frames is made up of elements, each of which presents a set of points of the trajectories of different objects that will be displayed in one synthetic frame;

According to the claimed invention, before forming a synthetic image and in the case of dividing a single object while tracking its movement into several detected objects - fragments, the trajectories of the fragments are glued into a single motion path using an algorithm that includes the steps in which:

- determination for each trajectory of the movement of its life time, which is the duration of the time interval from the appearance to the disappearance of the corresponding object in the original sequence of frames;

- determination for each frame from the initial sequence of frames the pairwise distance between the positions of all objects detected on the frame;

- determination of the proximity time for each pair of motion paths during which the distance between the positions of the corresponding objects was less than the first specified threshold value;

- determination of the ratio of the found time of proximity of a pair of motion paths to the life time of each of them;

- if one of the two relationships found is greater than the second predetermined threshold value, a decision is made that the two motion paths are fragments generated by one object.

The implementation of the invention

Below will be a description of exemplary variations of the claimed invention. However, the present invention is not limited to these embodiments only. It will be apparent to those skilled in the art that other variations fall within the scope of the claimed invention described in the claims.

The main stages of the proposed method

The proposed method operates with a sequence of images received from a stationary video camera and stored in the memory of a video surveillance system. This sequence of images will be called the original one. The original sequence of images is presented in the original sequence of frames. Each frame contains one image and an indication of the time it was received. This image may contain images of one or more objects and always contains a background on which these objects are located. For simplicity, we say that the image of the frame contains one or more objects and the background, bearing in mind that the object here means its image. An object is understood as a person, a car, etc., moving in the field of view of a video camera.

The proposed method is based on the formation of synthetic frames from the source frames and showing them to the operator. Images of objects registered at different points in time are combined in one synthetic frame. presented in different source frames. When placing images of these objects in one synthetic frame, it must be ensured that they do not obscure each other or obscure each other to an acceptable extent.

The proposed method for searching for objects in a sequence of images obtained from a stationary video camera is as follows.

Browse the original sequence of frames and find in it objects of interest to the operator.

For each of the detected objects, a trajectory of its movement is built. Each point of this trajectory corresponds to the position of the object in the frame and the time moment of registration of this frame. The position of the object in the frame is represented by a set of pixels ("points") of the frame representing the image of this object.

The trajectory of a detected object is a sequence of its positions in frames. Each such position corresponds to a point in time of receipt of the frame.

Next, a queue of these trajectories is formed. In this queue, all the trajectories of the objects are ordered by increasing time instants of the appearance of objects corresponding to these trajectories in the field of view of the video camera. In other words, for example, these trajectories are placed in memory in the order in which the trajectories of one or more objects that simultaneously appear in the camera’s field of vision are first placed before all other objects. Following them are placed the trajectories of objects that simultaneously appeared in the field of view of the camera later than the first objects indicated, but earlier than all other objects, and so on.

The serial number of the trajectory in their ordered set is assigned to this trajectory and its corresponding object.

After that, a schedule for showing detected objects is compiled for the number of objects set by the operator. In this schedule, for the trajectory of each object included in it, indicate the point in time from which this object will be displayed on the screen.

Initially, only the first object from their ordered set, that is, an object with a minimum serial number, is included in the display schedule of ordered objects. Then they update this schedule, removing objects from it, the trajectories of which are shown in full, and adding new ones to it. In this case, the number of objects simultaneously in the schedule does not exceed the value h specified by the operator.

For each current schedule build a plan for the formation of synthetic personnel. In each synthetic frame, several objects in positions that, in the general case, are registered at different points in time, can be simultaneously shown. Moreover, on the basis of the specified schedule, when it is created and each update is made, the specified plan for the formation of synthetic frames is made. This plan consists of elements, each of which presents a set of points of the trajectories of different objects that will be shown in one synthetic frame. The points in this set can be represented in various ways depending on the adopted implementation of the proposed method.

In accordance with the plan form the next synthetic frame.

When forming the next synthetic frame, its background is built and images of objects are superimposed on it, which, in accordance with the specified plan, should be shown in this synthetic frame at the same time.

In accordance with this plan for the formation of synthetic frames, the next generated synthetic frame is shown to the operator on the screen.

After that, it is checked whether the display of objects whose path points were in the shown synthetic frame is not finished. If none of these objects is displayed, then a new (regular) synthetic frame is formed in accordance with the plan and then it is shown. If at least one of these objects is displayed, then check whether all detected objects are shown to the operator. If not all detected objects are shown, then the displayed object is deleted from the current display schedule and the object that has not yet been shown is added to it, next by its number. If the operator is shown on the screen all the detected objects, then the process of forming and displaying synthetic frames to the operator ends.

The timing of the start of the display of objects is selected when updating the schedule for displaying objects. Updating the schedule of objects consists in changing the set of objects that are in this schedule. This change can occur by adding another object not yet shown and by deleting the object already shown. When adding another object that has not yet been shown, an appropriate start time for its display is selected for it, at which the condition of permissible mutual screening of objects in all synthetic frames formed on the basis of this updated schedule is fulfilled.

This selection is made from time values ranging from the start time of showing the object with the previous serial number to the end time of showing all objects that are in the updated schedule. To do this, step by step check the fulfillment of this screening condition for time values from the specified range as follows.

We look at these time values in ascending order with a given discreteness, and at each verification step we test the next of them as the time of showing the added object. To do this, at each step of checking the next test time value, a test schedule for showing objects is added to which the specified next object is added, and in this test schedule, the next test time value is indicated as the display time for this object.

Then, according to this trial schedule, a plan for the formation of all the objects in this schedule is built. After that, check the condition of permissible mutual screening of objects on all synthetic frames corresponding to this plan. If this condition is not fulfilled, then go to the step of checking the next test time value. If this condition is fulfilled, then the test value is taken as the start time for showing this added object, and the trial schedule as an updated schedule.

Thus, a plan for the formation of synthetic frames is built for each trial schedule, one of which is taken as updated.

When forming the next synthetic frame corresponding to any element of the plan for the formation of synthetic frames, the following actions are carried out:

- find the frames of the original sequence corresponding to the points of the trajectories of the objects contained in the element of the specified plan corresponding to the generated synthetic frame; for this, at the time point corresponding to the point of the transferred path of the object having the number γ and indicated in this plan element, the previously stored transfer value Δγ is added and the frame of the original sequence of images is found from the obtained time moment of the original path;

- from these frames, select the frame corresponding to the object with the lowest serial number as the reference frame on the basis of which the synthetic frame will be built;

- based on the reference frame, an intermediate frame is formed containing the displayed objects of the reference frame and the background on which they should be shown;

- form the next synthetic frame by adding to the received intermediate frame images of other objects from other frames selected above according to the specified plan for the formation of synthetic frames.

The reference frame contains objects A and B, of which only object A should be shown in the synthetic frame; therefore, in the intermediate frame, object B is closed by the corresponding background fragment. A synthetic frame is formed from an intermediate frame into which images of objects C and D from other source frames are transferred.

The background of the synthetic frame is the sum of the background of the reference frame and fragments of the background, which replace the objects of the reference frame, which should not be shown.

To form the background of the synthetic frame, fragments of the reference frame occupied by any objects, except those that should be shown in the generated synthetic frame, are replaced by fragments from other frames of the original sequence corresponding to the points of the trajectories of the objects given in the element of the specified plan corresponding to the generated synthetic frame.

For example, the reference frame contains an image of object B that should not be shown. From the number of frames in the initial sequence corresponding to the points of the transferred object trajectories given in the element of the plan being generated, any first frame is found in which a fragment unoccupied by objects is contained in the place of object B, and object B is replaced with the "empty" fragment in the reference frame.

Each successive synthetic frame is shown to the operator on the screen after a period of time, the duration of which is equal to the minimum non-zero difference between the moments of registration time of any object presented in the previous and next synthetic frames. This condition imposes a restriction on the application of the proposed method: each object must be represented on at least two frames of the original sequence of images.

As you know, various implementations of the algorithm for detecting objects in the original sequence of frames have different quality of work. For example, some implementations can split a single CD object into two — fragments C and D. In order to reduce the influence of a particular implementation of the detection algorithm on the visual quality of the generated synthetic frames, it is necessary to glue the fragments C and D into a single object before forming a queue of paths using the following gluing algorithm.

Using this algorithm, they check for errors during the operation of the motion tracking algorithm and, if such errors are detected, carry out the “correct” detection of a single object. The following is an example of such a detection.

Below are the main steps of the algorithm:

- determination for each trajectory of the movement of its life time, which is the duration of the time interval from the appearance to the disappearance of the corresponding object in the original sequence of frames (1);

- determination for each frame from the initial sequence of frames the pairwise distance between the positions of all objects detected on the frame (2);

- determination of the proximity time for each pair of trajectories of movement, during which the distance between the positions of the corresponding objects was less than the first specified threshold value (3);

- determination of the ratio of the found time of proximity of a pair of motion paths to the life time of each of them (4);

- if one of the two found relations is greater than the second specified threshold value, they decide that the two motion paths are fragments generated by one object (5).

At step (1), there is a video sequence of 250 frames obtained from one camera, and containing images and indications of the time moments of their registration. Let objects A, B, C, and D be found in this sequence. For the motion path of the above objects, their lifetime is determined, i.e. the duration of the time interval from the appearance to the disappearance of objects in the original video sequence. For example, object A has a lifetime of 40 seconds, object B - 30 seconds, object C - 20 seconds, object D - 10 seconds.

At step (2), for each of the 250 frames of the video sequence, the pairwise distance between the positions of all detected objects is determined. Suppose object A is present on frames numbered 100-140, object B is 110-140, object C is 120-140, object D is 130-140. The distance between A and B in frame 110 is calculated as the distance between the positions of objects A and B in this frame. Let the calculated distance between A and B on frame 110 be 30 image points, and between C and D on frame 130 be 100 image points.

At step (3), the proximity time is determined for a pair of trajectories of objects A and B, during which the distance between the positions of the corresponding objects was less than the first specified threshold value, for example, less than 50 image points. We take the proximity time calculated in this way for A and B - 3 seconds, for C and D - 5 seconds.

At stage (4), the ratio of the found time of proximity of a pair of trajectories of motion A and B, C and D to the lifetime of the trajectory itself is determined. For A and B, the ratios are 0.075 and 0.1, and for C and D, 0.25 and 0.5. Let the value of the second threshold value be 0.33.

At step (5), it is determined if one of the two relationships found is greater than the second predetermined threshold value, a decision is made that the two motion paths are fragments generated by one object. That is, in our case, fragments C and D are fragments of one object. Further, they are glued together and a synthetic image with a single CD object is formed.

Thus, the proposed method prevents at least one error in the detection of objects, providing bonding of fragments of objects into a single object and helps to improve the quality of the generated synthetic image.

The above steps of the method are carried out by software, hardware, software logic, or a combination thereof. In an embodiment, program logic, software, or a set of instructions are stored on one of various traditional computer-readable media. In the context of this document, a “machine-readable medium” may be any medium or means that may contain, store, transmit, distribute or transport instructions for use by an instruction execution system, equipment or device, such as a computer. A computer-readable medium may include a non-volatile computer-readable medium, which may be any medium or means containing or storing instructions for use by, or for use in connection with, an instruction execution system, equipment or device, such as a computer.

In one embodiment, a circuit or user interface circuit may be provided that is configured to provide at least some of the control functions described above.

If at least part of the various functions discussed in this description is necessary, it can be performed in a different order from that presented and / or simultaneously with each other. In addition, if necessary, one or more of the functions described above may be optional or may be combined.

Claims (18)

The method of displaying objects in a sequence of images obtained from a stationary video camera, which consists in the fact that: in the initial sequence of frames received from a stationary video camera and representing images and indications of the moments of time of their registration, objects of interest to the operator are detected; for each of the detected objects, a trajectory of its movement is built, each point of which corresponds to the position of the object in the frame and the time moment of registration of this frame, and the position of the object in the frame is represented by a set of pixels of the frame representing the image of this object; form a queue of trajectories of motion of detected objects; make a schedule for showing detected objects, in which for the trajectory of each object indicate the point in time from which it will be shown on the screen; according to the specified schedule, they build a plan for the formation of synthetic frames in which several objects can be shown simultaneously in positions that, in the general case, are registered at different points in time; form the next synthetic frame, in accordance with the specified plan; when forming a synthetic frame include images of objects that, in accordance with the specified plan should be shown in this synthetic frame at the same time, and the background on which they should be shown; show the generated synthetic frames to the operator on the screen, while in the queue of the trajectories of the movement of objects, all trajectories are ordered by increasing time of the appearance of the corresponding object in the field of view of the camera; the sequence number of the trajectory in the ordered set of trajectories is assigned to this trajectory and its corresponding object; initially, only the first object is included in the display schedule of ordered objects, and then this schedule is updated, removing objects from it, the trajectories of which are shown in full, and adding new ones, and the number of objects in the schedule does not exceed the value specified by the operator; on the basis of the specified schedule, when it is created and each update is made up, the specified plan for the formation of synthetic frames, consisting of elements, each of which presents a set of points of the trajectories of different objects that will be displayed in one synthetic frame; characterized in that before forming a synthetic image and in the case of dividing a single object while tracking its movement into several detected objects - fragments, gluing the trajectories of the fragments into a single trajectory is carried out using an algorithm that includes the steps in which: determining for each trajectory of the movement the time of its life, which is the duration of the time interval from the appearance to the disappearance of the corresponding object in the original sequence of frames; determining for each frame from the original sequence of frames the pairwise distance between the positions of all objects detected on the frame; determining the proximity time for each pair of motion paths during which the distance between the positions of the corresponding objects was less than the first specified threshold value; determination of the ratio of the found time of proximity of a pair of motion paths to the life time of each of them; if one of the two relations found is greater than the second specified threshold value, they decide that the two motion paths are fragments generated by one object.