DE102010031429A1 - Method for providing a combination video - Google Patents

Abstract

The invention relates to a method for providing a combination video (40) for a scenario, in which at least one synopsis video (30) for a movement of at least one object that is composed of an input video (20) representing the scenario appears in the scenario, a trajectory video (34) being generated for a trajectory of the at least one object for the at least one object that appears in the scenario and being superimposed on the at least one synopsis video (30) , the at least one synopsis video (30) and the at least one trajectory video (34) being combined to form the combination video (40).

Description

The invention relates to a method and an arrangement for providing a combination video.

State of the art

In general, the results of an automatic video analysis, such. B. object events or events or trajectories, drawn directly in a video to be examined. This occurs either in live mode or during playback of the recorded video from a database. However, in many scenes over a long time usually nothing interesting happens. The sifting of the video material with frequent fast forward until a special event is found, is therefore tedious and time-consuming.

One way to condense the information into a short video in a very long video is achieved through a so-called Synopsis video. Techniques for providing such synopsis video are found in the article by Y. Pritch, A. Rev. Achs, S. Peleg, "Nonchronological video synopsis and indexing", IEEE Transactions On Pattern Analysis and Machine Intelligence, Vol. 11, 2008, from the international patent application WO 2007/057893 A2 or from the website http://www.briefcam.com the company BriefCam known. In a synopsis video, the objects from the long video are released from their temporal context and rearranged.

A method and a system for indexing videos and for providing synopsis videos are in the document WO 2008/093321 A1 described. Here, a synopsis video is generated from a source video, wherein at least three different source objects are selected taking into account at least one boundary condition. Furthermore, each source object is assembled with respect to a subset of pixels from at least three different frames of the source video. At least one synopsis object is selected from each selected source object using pixels from certain time periods. In addition, for each Synopsis object, a time at which it appears in the Synopsis video is determined. Furthermore, a color can be determined for each synopsis object.

Disclosure of the invention

Against this background, a method and an arrangement with the features of the independent claims are presented. Further embodiments of the invention will become apparent from the dependent claims and the description.

In the context of the invention, a so-called interactive synopsis player or a playback module is provided as a possible component of the arrangement, which plays a summary of an input video and thus a synopsis video of the input video. If a viewer selects an object in the synopsis video via a human-machine interface, eg. As by touching with the mouse pointer or by clicking on, so a complete trajectory or trajectory of the object is displayed, which is displayed as a, like a string of beads stringed together, a series of object images. In addition, specific events of the object may be noted at the particular location of the trajectory at which they occurred.

To create a synopsis video, each object in a video editing module that appears in an input video or source video is timed independently so that the objects are packed as tightly together as possible in a single synopsis video. If the input video shows a street that only occasionally drives a car, the synopsis video of that scene shows the street in which the cars are driving close together like in a traffic jam. This makes it much faster and easier to view the edited input video now, and z. For example, count the number of taxis that appear within the scene.

Although conventional synopsis video can be used to generate a short summary of a very long input or source video. However, the objects in the synopsis video are sometimes difficult to track by the viewer, as they may only be visible very briefly. Often, however, actions of a particular object are of interest to a viewer.

With the invention, it is possible in an embodiment to represent a single object specifically in an object-specific summary. This allows the viewer to capture the object in its overall motion sequence without having to review the synopsis video several times, or to fast forward or rewind it. For this purpose, the invention comprises as a module at least the interactive synopsis player or at least one playback module for a summary, each with an object trajectory overlay and thus an insertion of a trajectory of an object.

Thus, in one variant of the invention, the presentation of a trajectory or trajectory video for an object as an overlay (overlay) via a synopsis video or summary video is provided.

Furthermore, in the invention, the combination of several synopsis videos for providing a global background synopsis with at least one synopsis video for an object trajectory can be provided. In addition, the simultaneous display of these videos, ie the combination video with multiple synopsis videos and at least one trajectory video, which is blended one of the synopsis videos, can be made.

Typically, a synopsis video is a static video that does not present the information object centered. However, it is usually of interest to examine a specific object and the information about that object. In an embodiment of the invention, the concept of the synopsis video is extended by a second stage. While an overview of a scenario is provided in the first stage with a static synopsis video, by highlighting, for example, clicking on an object by operating the human-machine interface, an object centered representation may be provided by an overlay over a displayed one Synopsis video is realized and represents this object in a further embodiment with the associated meta-information and events.

In a possible implementation of the invention, the expansion of synopsis videos is thus provided to an interactive presentation to compactly represent both the trajectory and thus trajectory of at least one object, as well as object-related spatial and temporal metadata for the user. This allows faster browsing through large amounts of surveillance video footage to find the objects with relevant events.

In an embodiment of the invention, a module for synopsis generation or generation is provided with the data as various videos are generated. This module for synopsis generation can be designed as a component of the arrangement according to the invention.

At least one generated synopsis video comprises a very condensed version of the input video provided by concurrently displaying objects from an observed scenario that are not necessarily simultaneous in the input video. This is generated by a similar process as in synopsis systems known in the art.

Furthermore, the input video is usually provided by a surveillance camera observing a scenario and thus an environment and / or location, eg a street scene, under a static, unaltered viewing angle. The input video typically shows a static, unmodified background that includes static, immovable objects. In addition, the input video also shows dynamic objects that move relative to the background over time. For at least one such dynamic object, the combination video is provided.

At least one label video generated in the embodiment of the method includes object labels of the observed scenario objects that are seen in the synopsis video.

For each object of the scenario or a scene, a very short, so-called trajectory video is generated individually in a further embodiment, which represents the object along its trajectory, as strung on a string of beads, several times. The trajectory video is specially created so that it can be displayed fluidly in an endless loop. This trajectory video is later presented as an overlay over a synopsis video to dynamically display various object trajectories.

An optional event detection module or event detection module detects for each object conspicuous actions, such. For example: object stands still, object falls over, object interacts with other object as well as other actions. These events are located spatially (x, y) as well as temporally (t) and are also passed as metadata to the synopsis player or the synopsis player module, so that this action provides video object events.

In contrast to known synopsis systems, which only represent the actual synopsis video, the interactive synopsis player provided in the context of the invention or the interactive playback module for displaying summaries plays at least two synopsis videos which are inserted within the combination video so-called background synopsis video form, simultaneously.

In addition to the at least one actual synopsis video, the synopsis player internally also plays an associated label video, but does not present it to the viewer on a display device of a video player (video player).

If the user moves a pointer, for example a mouse pointer, by actuating the man-machine interface via the played combination video comprising a plurality of synopsis videos and at least one displayed trajectory video, then the video player looks in and thus Play module as a component of an embodiment of the arrangement according to the invention for the combination video at a current position (mouse position) in the label video according to which object is shown at the current position. If the label image contains an object label or an object marking at the current position, then the trajectory video of the corresponding selected object is additionally displayed, and a trajectory previously inserted by another object is deactivated. Does the label video contain the special background label? B. zero, the current trajectory remains active, if active. If an object trajectory has been activated, the third video is the respective trajectory video played. This trajectory video overlays the synopsis video.

The trajectory videos are usually a special form of synopsis video, but each represents only a single object on a transparent background. The generation process of trajectory videos also deviates slightly from the generation of synopsis videos. While video length is an input parameter in the generation of conventional synopsis videos, trajectory video is a central, automatically determined parameter of the generation process. Trajectory videos with a display rate n are usually generated in such a way that in each case the object is copied from every nth input image or frame into an image. That means z. For example, a trajectory video with the display rate n = 3 contains the object images from the following frames represented by numbers:
1st picture: 0, 3, 6, 9, 12, 15, 18, 21, ...
2nd picture: 1, 4, 7, 10, 13, 16, 19, 22, ...
3rd picture: 2, 5, 8, 11, 14, 17, 20, 23, ...

The following frames are considered for a k-th input image: k, k + n, k + 2n, k + 3n, k + 4n, k + 5n, k + 6n, ...

Typically, for small display rates n of the trajectory videos, there will be more object overlaps than for large display rates n, since the multiple representations of the object are closer together. On the other hand, for large display rates n, the free distances between the multiple representations of the object may increase. This trade-off can be optimized for the generation of the trajectory videos by trying out different values for the display rates n for a good ratio.

In a further embodiment of the invention, interactions or interactions of objects can be taken into account. If an interaction of an activated first object (a) with another second object (b) is detected, the trajectory of the second object (b) is displayed in addition to the trajectory of the first object (a). Possibly. In this case, not the entire trajectory of the second object (b) but only the part relevant for the interaction with the first object (a) is displayed.

In addition, occlusions of an object can be marked. If the first object (a) in the input video or original video is obscured by the second object (b), then it may happen in the synopsis video that both objects (a), (b) are sent to different times in the synopsis Video to be pushed. This creates an inexplicable hole for the first object (a). However, since the generation of the synopsis video is known to have resulted from occlusion with the second object (b), the area, e.g. B. by hatching or other suitable emphasis, be specially marked.

Alternatively or additionally, additional synopsis videos with different settings can be generated, eg. With weaker temporal compression or with guaranteed temporal coherence, so that after selection of the object optionally additionally these synopsis videos instead of the original synopsis video to display the background of the scenario are displayed. This makes it easier for the user to understand the interaction of the selected object with its environment within the scenario, as it can distort causality in a highly compressed synopsis video.

A time-independent selection of objects can also be provided. It is provided that instead of looking at an interactive selection of the object only in the current image of the label video, it can be looked up for each image position, which objects regardless of the current time for consideration in question. This is especially useful for scenarios in which many objects occur at a particular location, e.g. B. on a door or in a corridor of a room. The user or viewer can then, after selecting the position in the image of the combination video, one by one look at the objects in question.

One use of the invention is typically provided for forensic search in large quantities of surveillance videos and / or for applications in which locations and / or scenarios are to be monitored over long periods of time.

The arrangement according to the invention is designed to carry out all the steps of the presented method. In doing so, individual steps of this process can also be carried out by individuals Components of the arrangement are performed. Furthermore, functions of the arrangement or functions of individual components of the arrangement can be implemented as steps of the method. In addition, it is possible for steps of the method to be realized as functions of at least one component of the arrangement or of the entire arrangement.

Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

It is understood that the features mentioned above and those yet to be explained below can be used not only in the respectively specified combination but also in other combinations or alone, without departing from the scope of the present invention.

Brief description of the drawings

1 shows a diagram for providing a synopsis video according to the prior art.

2 shows a diagram of a first embodiment of the method according to the invention for providing a combination video, which can also be referred to as extended video synopsis system.

3 shows a schematic representation of a label image that is provided in a second embodiment of the method according to the invention.

4 shows a diagram of a third embodiment of the method according to the invention.

5 shows an example of images from a combination video, which is provided in a fourth embodiment of the method according to the invention.

6 shows an example of a composite video image provided in a fifth embodiment of the inventive method.

7 shows a schematic representation of an embodiment of an arrangement according to the invention.

Embodiments of the invention

The invention is schematically illustrated by means of embodiments in the drawings and will be described in detail below with reference to the drawings.

The figures are described in a coherent and comprehensive manner, like reference numerals designate like components.

Like the diagram 1 shows is in the known method for providing a synopsis video 2 for a source or input video 4 a segmentation 6 performed. Whereby so-called object masks 8th to be provided. From the object masks 8th and the input video 4 becomes at a synopsis generation 10 the synopsis video 2 provided, for example, with a video player 12 can be represented.

In the first embodiment of the method according to the invention, by the diagram of 2 is shown, is for an input video 20 a segmentation 22 performed. This will be object masks 24 provided. These object masks 24 become in a further step an event detection or an event detection 26 subjected. Also, for the input video 20 and the object masks 24 a synopsis production 28 or a synopsis generation performed. There will be several synopsis videos 30 , at least one label video 32 as well as at least one trajectory video 34 provided. The Synopsis Videos 30 that has at least one label video 32 as well as the at least one trajectory video 34 are usually using an interactive video player 36 , which may also be referred to as a video player and possibly provided via software of a computing device, shown. In addition to the video player 36 metadata 38 for local (x, y) and temporal (t) parameters of an object as well as a description of the object to the video player 36 be supplied. Accordingly, the synopsis videos 30 that has at least one label video 32 that has at least one trajectory video 34 and the metadata 38 in the present embodiment of the method according to the invention to a combination video 40 summarized.

Unlike the simple synopsis method ( 1 ) will be in addition to the synopsis videos 30 even the at least one label video 32 , as well as for at least one object in the original input video 20 shows up, a trajectory video 34 generated. In addition, optional object-related events are metadata 38 collected. All this information is stored in a synopsis player or player as components of the video player 36 summarized, collected and presented to a viewer with a provided composite video 40 interact via a man-machine interface to select information to be displayed, usually objects.

3 shows a schematic representation of the contents of a label image 50 from a label video as provided in a second embodiment of the method of the invention. Shown above is a space-time cube 52 with an x-axis 54 as well as a y-axis 56 for the room and a t-axis 58 for the time that includes the entire input video as a source, looking at the x-t plane of the space-time cube 52 , An image of the synopsis video includes parts of this space-time cube 52 , where the pixels of a synopsis image of the synopsis video are not necessarily from the same time slice of the space-time cube 52 must come.

In the 3 are z. B. three objects 60 . 62 . 64 in the synopsis image 66 indicated from three different times of the space-time cube 52 come. The label video, unlike the synopsis video, does not contain the image data itself, but the object label that appears at a particular point in the synopsis video.

Between the frames or input images in the synopsis video and those in the label video there is a 1: 1 correspondence.

In the example, cubes are in the space-time 52 three object trajectories 68 . 70 . 72 located. An image of a synopsis video is created by adding portions of the space-time cube 52 into the synopsis picture 66 be copied, the pixels or parts can come from different time slices. Usually such a time slice comprises an image and thus a snapshot of an input video. In the example, the synopsis image shown below is included 66 also the three objects 60 . 62 . 44 however, they come from three different times of the input video. The label picture 50 Now saves not the image data, but the object labels, as shown in the example. Empty areas 74 that is not an object 60 . 62 . 64 are indicated by a special label or a special mark, z. B. null.

In a third embodiment of the method according to the invention, as the diagram 4 shows a combination of a synopsis video 76 with a trajectory video 82 in a synopsis player as a component of an inventive arrangement. The Synopsis video 76 becomes in a usually relatively long endless loop 78 played. If the user selects a first object 80 off, so does the associated trajectory video 82 of the object 80 by fading in 84 via the synopsis video 76 placed. The trajectory video 82 contains the trajectory 86 of the selected object 80 , Also the trajectory video 82 runs in an infinite loop 88 but taking the length of the trajectory video 82 usually much shorter than the synopsis video 76 is. The two videos do not run synchronously but asynchronously. By fading in 84 of the trajectory video 82 in the synopsis video 76 becomes the combination video 90 provided, in addition to the trajectory 86 of the selected first object 80 also other objects 92 contains.

5a and 5b each show an image from a combination video, as provided in a fourth embodiment of the method according to the invention. In both pictures the scenario is a traffic intersection. It is in the picture off 5a a trajectory 94 of a car. In the picture 5b is a trajectory 96 of a cyclist.

A picture of another combination video as provided in a fifth embodiment of the method according to the invention is shown in FIG 6 shown. In this picture is a trajectory 98 a person represented as an object. In the embodiment shown here are optional to the trajectory 98 Video Events 100 . 102 for the selected object to the trajectory 98 written. The first video event 100 indicates that the person is hidden behind another object for about 5 seconds. A second video event 102 indicates that the person turns and waits, which takes about 15 seconds. This embodiment of the combination video allows a viewer to quickly get an overview of the activities of the person and thus of the object.

7 shows a schematic representation of an embodiment of an arrangement according to the invention 200 , which is a first module 202 for editing videos, this first module 202 can also be referred to as a video player or video player. Furthermore, the first module comprises 202 for editing videos a second module 203 for segmentation, a third module 204 for synopsis generation (synopsis video generation) and a fourth module 206 for trajectory generation (trajectory video generation).

As part of an embodiment of the method according to the invention, with the arrangement shown here 200 is performed, it is provided that as source video an input video 208 provided by a surveillance camera filming a scenario. This scenario includes a background of static, immobile objects within which time-varying moving dynamic objects appear and move relative to the background.

Upon execution of the method according to the invention, the input video 208 with the second module 203 segmented for segmentation, with typically dynamic objects included in the input video 208 emerge, object masks are generated. With the third module 204 For synopsis generation, the object masks for multiple objects become synopsis videos 210 . 212 generated, which have different settings.

Also, for at least one object is directly from the input video 208 and / or from the object masks, which are also referred to as segmentation masks, with the fourth module 206 for trajectory generation in each case a trajectory video 214 generated. Alternatively or additionally, at least one trajectory video 214 from at least one synopsis video 210 . 212 be generated.

A combination video 216 for the scenario arises by combining at least one generated synopsis video 210 . 212 and at least one generated trajectory video 214 , When playing the combination video 216 the at least one synopsis video is running 210 . 212 in the background. The at least one trajectory video 214 will when playing the combination video 216 displayed as overlay or overlay.

As a rule, only a synopsis video 210 . 212 which contains the background of the scenario, ie static objects, and a summary of all dynamic objects. In the at least one synopsis video 210 . 212 may or may in embodiment of the invention at least one trajectory video 214 and thus also several trajectory videos 214 to be displayed. If several synopsis videos 210 . 212 can be generated to display a specific dynamic object, a synopsis video 210 . 212 through interaction of the user with the man-machine interface 220 to be selected. Thus, for each dynamic object, a synopsis video 210 . 212 available. Usually, this serves at least one synopsis video 210 . 212 to represent the background, usually the static objects, of the scenario.

It is not necessarily the case that for each dynamic object a synopsis video 210 . 212 is created, although that is also conceivable. If the user selects a trajectory for an object, the synopsis video may be 210 . 212 be replaced in the background as desired by another, which meets certain criteria, the z. For example, it preserves temporal causality, is weaker in compression, displays only objects that the selected object interacts with, and so on. However, it does not necessarily need an extra set of synopsis videos for each dynamic object 210 . 212 give.

The first module 202 for editing videos is a display device 218 associated with the single synopsis videos 210 . 212 and the trajectory video 214 separately but also in combination can be considered. Furthermore, via the display device 218 also the input video 208 and the combination video generated in the process 216 considered.

Besides, here is a man-machine interface 220 shown with the first module 202 for editing videos and the display device 218 interacts. About this man-machine interface 220 In the context of one embodiment of the method according to the invention, the object is displayed by a viewer by marking an object from the scenario on the display device 218 selected this object and the generated composite video 216 with inserted trajectory video 214 representing the trajectory of the selected object.

Using the method can create a scenario focusing on the selected object with the combination video 216 be considered, allowing an automatic analysis of the input video 208 and thus the scenario is possible.

In a synopsis video 210 . 212 be at least two objects in the input video 208 emerge within the scenario at different times, displayed at the same time. A synopsis video 210 . 212 may alternatively or additionally include trajectory portions of a trajectory of at least one object included in the input video 208 within the scenario by moving the at least one object at different times and thus in the input video 208 appear at different times, be presented at the same time.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• WO 2007/057893 A2 [0003]
• WO 2008/093321 A1 [0004]

Cited non-patent literature

• http://www.briefcam.com [0003]

Claims (10)

Method for providing a combination video ( 40 . 90 . 216 ) for a scenario in which an input video ( 20 . 208 ) representing the scenario, at least one synopsis video ( 30 . 76 . 210 . 212 ) for a movement of at least one object ( 60 . 62 . 64 . 80 . 92 ) that appears in the scenario is generated, wherein for the at least one object ( 60 . 62 . 64 . 80 . 92 ), which appears in the scenario, in each case a trajectory video ( 34 . 82 . 214 ) for a trajectory ( 68 . 70 . 72 . 86 . 94 . 96 . 98 ) of the at least one object ( 60 . 62 . 64 . 80 . 92 ) and into the at least one synopsis video ( 30 . 76 . 210 . 212 ), the at least one synopsis video ( 30 . 76 . 210 . 212 ) and the at least one trajectory video ( 34 . 82 . 214 ) to the combination video ( 40 . 90 . 216 ). Method according to Claim 1, in which in a synopsis video ( 30 . 76 . 210 . 212 ) at least two objects ( 60 . 62 . 64 . 80 . 92 ) in the input video ( 20 . 208 ) appear within the scenario at different times, be displayed at the same time and / or in which at least one trajectory video ( 34 . 82 . 214 ) Trajectory sections of a trajectory ( 68 . 70 . 72 . 86 . 94 . 96 . 98 ) at least one object ( 60 . 62 . 64 . 80 . 92 ) in the input video ( 20 . 208 ) within the scenario by moving the at least one object ( 60 . 62 . 64 . 80 . 92 ) appear at different times, be displayed at the same time. Method according to Claim 1 or 2, in which at least two of the synopsis videos ( 30 . 76 . 210 . 212 ) with different settings. Method according to one of the preceding claims, in which the at least one object ( 60 . 62 . 64 . 80 . 92 ) in the trajectory video ( 34 . 82 . 214 ) at the trajectory ( 68 . 70 . 72 . 86 . 94 . 96 . 98 ) of the at least one object ( 60 . 62 . 64 . 80 . 92 ), along which the at least one object ( 60 . 62 . 64 . 80 . 92 ) is moved in the scenario, shown multiple times. Method according to one of the preceding claims, in which the trajectory video ( 34 . 82 . 214 ) for the at least one object ( 60 . 62 . 64 . 80 . 92 ) has a display rate n and in which to generate the trajectory video ( 34 . 82 . 214 ) every nth input image of the at least one object ( 60 . 62 . 64 . 80 . 92 ) from the input video ( 20 . 208 ) into the trajectory video to be generated ( 34 . 82 . 214 ) is copied. Method according to one of the preceding claims, in which, in the event that the at least one object ( 60 . 62 . 64 . 80 . 92 ) with at least one further object ( 60 . 62 . 64 . 80 . 92 ) interacts in the trajectory video ( 34 . 82 . 214 ) of the at least one object ( 60 . 62 . 64 . 80 . 92 ) additionally at least a portion of a trajectory ( 68 . 70 . 72 . 86 . 94 . 96 . 98 ) of the at least one further object ( 60 . 62 . 64 . 80 . 92 ), where the objects ( 60 . 62 . 64 . 80 . 92 ), is displayed. Method according to one of the preceding claims, in which, in the event that the at least one object ( 60 . 62 . 64 . 80 . 92 ) in the input video ( 20 . 208 ), an occlusion in a generated synopsis video ( 30 . 76 . 210 . 212 ) is marked. Method according to one of the preceding claims, in which the at least one object ( 60 . 62 . 64 . 80 . 92 ), for which the trajectory video ( 34 . 82 . 214 ) is selected, independent of time. Arrangement for providing a combination video ( 40 . 90 . 216 ) for a scenario involving at least one module ( 202 ) for editing videos, wherein the at least one module ( 202 ) for editing videos the combination video ( 40 . 90 . 216 ) from an input video ( 20 . 208 ) produced by a method according to any one of the preceding claims. Arrangement according to Claim 9, which has a human-machine interface ( 220 ) by an observer from an input video ( 20 . 208 ) and / or a synopsis video ( 30 . 76 . 210 . 212 ) at least one object ( 60 . 62 . 64 . 80 . 92 ) is to be selected from a scenario, wherein the at least one module ( 202 ) for editing videos the combination video ( 40 . 90 . 216 ) taking into account the at least one selected object ( 60 . 62 . 64 . 80 . 92 ) generated.

Images