TWI536838B - Video playback method and apparatus - Google Patents

Description

Video playing method and device

The present invention relates to a video device, and more particularly to a video playback method and apparatus.

The video surveillance system can capture the scene using the camera module to obtain the original video and store the original video on the hard disk. For traditional video playback systems, playback is a commonly used feature. The playback function allows the user to view the contents of the original video stored on the hard drive. The user can view the content of the original video in a specific time section to search for an object of interest or an abnormal event. However, the length of the original video is often extremely long. For example, the length of the original video may be hours or even days. The user can use the preset constant speed to speed up playing/viewing the original video stored on the hard disk. Although the conventional video playback system can shorten the video playback time, the conventional video playback system cannot display all the objects of the original video for a preset length of time by the user.

The invention provides a video playback method and device, which can shorten the video playing time and display all the objects of interest in the preset playing time length.

An embodiment of the present invention provides a video playing method, including: providing an original video, wherein the original video is obtained by capturing a scene by a photography module; and providing a length of playing time to determine a length of the synthesized video, where the composite The length of time of the video is less than the length of time of the original video; at least one object path is extracted from the original video; and the at least one object path is selectively adjusted to composite the at least one object path into the composite video.

In an embodiment of the invention, the video playing method further includes: playing the composite video.

In an embodiment of the invention, the step of extracting the at least one object path from the original video comprises: performing an object detection and a background extraction process to extract at least one object and at least one background image from the original video. Creating the object path according to a relationship of the at least one object of a current frame in the original video with the at least one object of a previous frame; and storing the background image and the object path In the storage device.

In an embodiment of the invention, the step of creating the path of the at least one object includes: if the object of the current frame is not in a parent object of the previous frame, or in the current frame and other objects If there is a parent object, or the object of the current frame has multiple parent objects, a new object path is created, where the mesh The object of the previous frame is the first object of the new object path; if the object of the current frame has a unique parent object, and the object of the current frame is the only child object of the parent object, then The object of the current frame is added to an existing object path to which the parent object belongs; and when the last object of the at least one object path has no child objects, or when the last object of the at least one object path has more than one The at least one object path ends when the sub-object, or when the last object of the at least one object path and the other object path share the sub-object.

In an embodiment of the present invention, the video playing method further includes: providing a start time T _b and an end time T _e in the original video; and if T _b P _t T _e , or T _b P _t + P _l T _e , or P _t T _b and T _e P _t + P _l , where P _t and P _l are the occurrence time and length of a candidate object path in the storage device, respectively, and the candidate object path is selected as the at least one object path.

In an embodiment of the present invention, the video playing method further includes: calculating a length of the suggested time according to the crowding situation of the at least one object path at different pixels in a scene; and prompting the suggested time length to use In order to assist the user to determine the length of the play time.

In an embodiment of the present invention, the video playing method further includes: generating a crowdedness map according to the original video to describe an extruding value of the object path at different pixels; calculating an equation Where C _m represents the associated value of the crowded values, C _th represents the threshold; and the equation is calculated Where T _p represents the suggested length of time and R _f represents the frame rate of the composite video.

In an embodiment of the present invention, the video playing method further includes: providing an object attribute; and screening the at least one object path according to the item attribute.

In an embodiment of the invention, the step of selectively adjusting the at least one object path to synthesize the at least one object path to the composite video comprises: following the appearance of the object path in the original video Sequence, rearranging the object path in the composite video; and synthesizing the object path and the background image into the composite video.

In an embodiment of the invention, the step of rearranging the object path in the composite video comprises: selectively adjusting the object path to obtain at least one adjusted object path; according to the object path An order of occurrence in the original video, initializing a temporal position of the adjusted object path in the composite video; and adjusting the adjusted object path in the composite video in accordance with an overlap condition of the adjusted object path in the composite video Time location.

In an embodiment of the invention, the step of obtaining the at least one adjusted object path includes: if the object path has a parent object path, merging the object path and the parent object path as the adjusted An object path; if the time length of the object path is greater than the threshold length, speeding up the playback speed of the object path according to a speedup factor to shorten the time length of the object path, as the adjusted object path, wherein The threshold length is less than or equal to the length of the play time, and the acceleration factor is a real number; and if the time length of the object path is greater than The threshold length splits the object path into a plurality of sub-paths as the adjusted object path.

In an embodiment of the present invention, the video playing method further includes: calculating a first factor S _g = ( P _l / P _th ), where P _l is a time length of the object path, and P _th represents the threshold length If the first factor S _{g is} greater than the maximum acceleration value S _max , the first factor S _{g is} set to the maximum acceleration value S _max , wherein the maximum acceleration value S _max is a real number greater than 1 and less than 4; if the first factor S _g If less than 1, the first factor S _{g is} set to 1; if the representative forced value C _{p of} the object path in the crowd map is greater than or equal to the upper limit forced value C _U , the second factor S _{c is} set to The maximum acceleration value S _max , wherein the upper limit forced value C _U is a real number; if the representative forced value C _{p is} less than or equal to the lower limit forced value C _L , the second factor S _{c is} set to 1, wherein the lower limit forced value C _L is a real number, and the lower limit forced value C _{L is} less than the upper limit forced value C _U ; if the representative forced value C _{p is} greater than the lower limit forced value C _L and less than the upper limit forced value C _U , the second factor S _c is set Is [( C _p - C _L )/( C _U - C _L )]*( S _max -1)+1; and the larger one is selected from the first factor S _g and the second factor S _c as the acceleration factor S _p .

In an embodiment of the present invention, the step of splitting the at least one object path into a plurality of sub-paths includes: adjusting a first sub-path of the sub-paths to other sub-paths of the sub-paths Frame shift to reduce the overlap area of the sub-paths.

In an embodiment of the invention, the step of initializing the time position of the adjusted object path in the composite video comprises: if between the first object path and the second object path in the adjusted object path If there is a gap in time, the time of the first object path and the subsequent occurrence in the second object path is advanced, and the The time of the subsequent occurrence is later than the time of the first object path and the first occurrence in the second object path; and the time offset of the adjusted object path is respectively multiplied by the adjustment value to make the adjusted object path The time range is included in the range of the length of the play time.

In an embodiment of the invention, the step of adjusting the temporal position of the adjusted object path in the composite video comprises: adjusting a first object path in the adjusted object path to the adjusted object path The frame offset of the other object paths in the middle to reduce the overlapping area of the adjusted object paths.

In an embodiment of the invention, the step of synthesizing the object path and the background image into the composite video comprises: mixing an object image of the object path with the background by a Gaussian blending method image.

In an embodiment of the invention, the step of synthesizing the object path and the background image into the composite video comprises: mixing the object images overlapping each other in the object path in a translucent manner by using an Alpha blending method.

In an embodiment of the invention, the step of synthesizing the object path and the background image into the composite video comprises: calculating a z-axis distance of the plurality of objects in the object path; and z according to the objects The descending order of the axial distances, the objects are pasted on the background image.

Embodiments of the present invention provide a video playback device, including an object path extraction module and a video synthesis module. The object path extraction module is configured to extract at least one object path from the original video. The video synthesis module is coupled to the object path extraction module to receive the object path. Video synthesis module configured to play time The object path is selectively adjusted in length to synthesize the object path into a composite video, wherein the length of play time determines the length of time of the composite video, and the length of time of the composite video is less than the length of time of the original video.

In an embodiment of the invention, the video playback device further includes a camera module. The photography module is coupled to the object path extraction module. The photography module is configured to capture the scene to obtain the original video.

In an embodiment of the invention, the video playback device further includes a display module. The display module is coupled to the video synthesis module. The display module is configured to play the composite video.

In an embodiment of the invention, the object path includes a first object path and a second object path, and the playback speed of the first object path in the composite video is different from the playback speed of the second object path.

In an embodiment of the invention, the object path includes a first object path and a second object path, and the time of the first object of the first object path and the second object of the second object path appear in the original video Do not overlap, and the first object overlaps with the time at which the second object appears in the composite video.

In an embodiment of the invention, the spatial position of the object path in the composite video is the same as the spatial position of the object path in the original video.

In an embodiment of the invention, the video playback device further includes a storage module. The storage module is coupled to the object path extraction module. The object path extraction module includes an object detection and background extraction unit and an object path generation unit. The object detection and background extraction unit is configured to receive the original video and extract from the original video Taking at least one object and at least one background image, and storing the background image in the storage module. The object path generating unit is coupled to the object detecting and background extracting unit. The object path generation unit is configured to create the object path based on the relationship of the object of the current frame in the original video with the object of the previous frame, and store the object path in the storage module.

In an embodiment of the present invention, if the object of the current frame does not have a parent object of the previous frame, or the object in the current frame shares a parent object with other objects, or the object of the current frame has multiple parent objects, the object The path generation unit creates a new object path in which the object of the current frame is the first object of the new object path. If the object of the current frame has a unique parent object, and the object of the current frame is the only child of the parent object, the object path generating unit adds the object of the current frame to the existing object path to which the parent object belongs. When the last object of the object path has no child objects, or when the last object of the at least one object path has more than one child object, or when the last object of the object path and other object paths share the child object At this time, the object path generating unit ends the object path.

In an embodiment of the present invention, the information of the object path includes: a time length of the object path, a time stamp of the first object of the object path, and each member object of the object path corresponds to the first The time shift of an object, the position of each member object, the size of each member object, or the parent object path.

In an embodiment of the invention, the video playback device further includes a user interface. Configured to provide a user interface in the original video start time end time T _b and T _e. If T _b P _t T _e , or T _b P _t + P _l T _e , or P _t T _b and T _e P _t + P _l , where P _t and P _l are the occurrence time and length of the candidate object path in the storage device, respectively, and the video synthesis module selects the candidate object path as the object path.

In an embodiment of the invention, the video synthesizing module includes an object path collecting unit and a video length estimating unit. The object path collection unit is configured to collect some or all of the object paths from the object path generated by the object detection and background extraction unit. The video length estimating unit is coupled to the object path collecting unit to receive the collected result of the object path collecting unit. The video length estimating unit is configured to estimate a recommended length of time according to an overcrowding situation of the object path at different pixels in a scene, wherein the suggested length of time is used to prompt a user to assist the user in determining the The length of play time.

In an embodiment of the invention, the video length estimating unit generates an extruded map according to the original video to describe the crowded value of the object path at different pixels. Video length estimation unit calculation equation Where C _m represents the associated value of the crowded values and C _th represents the threshold. Video length estimation unit calculation equation Where T _p represents the suggested length of time and R _f represents the picture playback rate of the composite video.

In an embodiment of the invention, the associated value C _m is an average of the crowded values of all pixels in the crowded map.

In an embodiment of the invention, the associated value C _m is an average of the top 50% of the forced values of the crowded map, or an average of the top 20% of the forced values, or The average of the 10% crowded value.

In an embodiment of the invention, the threshold C _{th is} greater than 0 and less than the associated value C _m .

In an embodiment of the invention, the video synthesizing module includes an object path collecting unit and an object filter. The object path collection unit is configured to collect a portion or all of the object paths from the object path generated by the object path extraction module. The object filter is coupled to the object path collecting unit to receive the collected result of the object path collecting unit. The object filter is configured to filter the object path based on the item properties.

In an embodiment of the invention, the object attributes include size, color, texture, material, face, direction of motion, or behavior.

In an embodiment of the invention, the video synthesis module includes an object path rearrangement unit and a video synthesis unit. The object path rearrangement unit is configured to rearrange the object path in the composite video in accordance with the order of appearance of the object path in the original video. The video synthesis unit is coupled to the object path rearrangement unit to receive the rearrangement result of the object path rearrangement unit. The video synthesis unit is configured to synthesize the object path and the background image into a composite video.

In an embodiment of the invention, the object path rearranging unit selectively adjusts the object path to obtain at least one adjusted object path. The object path rearrangement unit initializes the temporal position of the adjusted object path in the composite video in accordance with the order of appearance of the object path in the original video. According to the adjusted object The object path rearrangement unit adjusts the temporal position of the adjusted object path in the composite video.

In an embodiment of the invention, if the object path has a parent object path, the object path rearrangement unit merges the object path with the parent object path as the adjusted object path. If the time length of the object path is greater than the threshold length, the object path rearranging unit accelerates the playing speed of the object path according to the acceleration factor to shorten the time length of the object path as the adjusted object path. The threshold length is less than or equal to the length of the play time, and the acceleration factor is a real number. If the time length of the object path is greater than the threshold length, the object path rearrangement unit splits the object path into a plurality of sub-paths as the adjusted object path.

In an embodiment of the invention, the threshold length is greater than or equal to a quarter of the length of the play time.

In an embodiment of the invention, the threshold length is equal to one-half of the length of the play time.

In an embodiment of the present invention, the above-described acceleration factor _{S p = (P l / P} th), where P _l is the length of time the article path, P _th denotes the threshold length. When the acceleration factor is greater than the maximum acceleration value S _p S _max, then the acceleration factor S _p as the maximum acceleration value S _max, where S _max value is the maximum acceleration is greater than 1 and less than 4. When the acceleration factor is less than S _p 1, S _p factor is set to 1 will be accelerated.

In an embodiment of the present invention, when the above-described object path in congested congestion graph represent C _p value greater than or equal to the upper limit value of the congestion C _U, the acceleration factor S _p as the maximum acceleration value S _max, wherein The upper limit forced value C _U and the maximum acceleration value S _max are real numbers. If the representative value of the congestion C _p value of less than or equal to the lower limit C _L congestion, then the acceleration factor S _p is 1, wherein lower limit value of the congestion C _L is a real number, and the lower limit value C _L crowded crowded than the upper limit value C _U. If the representative congestion crowded than the lower limit value of C _p and C _L than the upper limit value of the congestion value C _U, then the acceleration factor S _p is set to _{[(C p - C L)} / (C U - C L)] * ( S _max -1) +1.

In an embodiment of the invention, the object path rearrangement unit calculates a first factor S _g =( P _l / P _th ), where P _l is the length of time of the object path, and P _th represents the threshold length. If the first factor S _{g is} greater than the maximum acceleration value S _max , the object path rearrangement unit sets the first factor S _g to the maximum acceleration value S _max , wherein the maximum acceleration value S _max is a real number greater than 1 and less than 4. If the first factor S _{g is} less than 1, the object path rearrangement unit sets the first factor S _g to 1. If the representative forced value C _{p of} the object path in the crowded map is greater than or equal to the upper limit forced value C _U , the object path rearranging unit sets the second factor S _c to the maximum acceleration value S _max , wherein the upper limit is squeezed The forced value C _U is a real number. If the representative forced value C _{p is} less than or equal to the lower limit forced value C _L , the object path rearrangement unit sets the second factor S _c to 1, wherein the lower limit forced value C _L is a real number, and the lower limit forced value C _L Less than the upper limit of the forced value C _U . If the representative forced value C _{p is} greater than the lower limit forced value C _L and less than the upper limit forced value C _U , the object path rearrangement unit sets the second factor S _c to [( C _p - C _L )/( C _U - C _L )]*( S _max -1)+1. Rearrangement unit selected object path from the first factor and the second factor S _g S _c as the greater of the acceleration factor S _p.

In an embodiment of the present invention, the object path rearranging unit adjusts a frame offset of the first sub-paths of the sub-paths to other sub-paths of the sub-paths to reduce overlapping of the sub-paths. area.

In an embodiment of the invention, if there is a gap between the first object path and the second object path in the adjusted object path, the object path rearranging unit will first object path and the second object The time of the subsequent occurrence in the path is advanced, and the time of the subsequent occurrence is later than the time of the first object path and the first occurrence in the second object path. The object path rearranging unit multiplies the time offset of the adjusted object path by the same adjustment value such that the time range of the adjusted object path is included in the range of the playing time length.

In an embodiment of the invention, the object path rearranging unit adjusts a frame offset of the first object path in the adjusted object path to other object paths in the adjusted object path to reduce the The overlapping areas of the object paths are adjusted.

In an embodiment of the invention, the video synthesizing unit mixes the object image and the background image of the object path by a Gaussian blending method.

In an embodiment of the invention, the video synthesizing unit mixes the object images overlapping each other in the object path in a translucent manner by using an Alpha blending method.

In an embodiment of the invention, the video synthesizing unit calculates a z-axis distance of a plurality of objects in the object path. The video synthesizing unit pastes the objects onto the background image according to the order of power reduction of the z-axis distances of the objects.

In an embodiment of the invention, the z-axis distance of the object is inversely proportional to the maximum value of the y-coordinate of the object, or proportional to the minimum distance from the object to the center of the image.

An embodiment of the present invention provides a video playing method, including: providing an original video, wherein the original video is obtained by capturing a scene by a photography module; and the object path extraction module extracts at least one object path and a background image from the original video. And calculating a z-axis distance of the plurality of objects in the object path; and affixing the objects to the background image according to a descending power order of the z-axis distances of the objects to synthesize the object path to the composite In the video, the length of time of the composite video is less than the length of time of the original video.

In an embodiment of the invention, the z-axis distance of the object is inversely proportional to the maximum value of the y-coordinate of the objects, or proportional to the minimum distance from the objects to the center of the image.

Based on the above, the video playing method and the video playing device in the embodiments of the present invention can shorten the video playing time, that is, the length of the synthesized video is smaller than the length of the original video. The video playing method and the video playing device may determine the length of the synthesized video according to a fixed setting or a playing time length dynamically determined by the user. The video playback method and video playback device may extract at least one object path from the original video and selectively adjust the object path to composite the object path into the composite video. Therefore, the video playing method and the video playing device can display all the items of interest in a preset playing time length.

The above described features and advantages of the invention will be apparent from the following description.

11‧‧‧ original video

12‧‧‧ Object path

13‧‧‧Composite video

100‧‧‧Video player

110‧‧‧Photography module

120‧‧‧Object Path Extraction Module

121‧‧‧ Object detection and background extraction unit

122‧‧‧object path generation unit

130‧‧‧Storage Module

140‧‧‧Video Synthesis Module

141‧‧‧Video Length Estimation Unit

142‧‧‧Object Path Collection Unit

143‧‧‧Object path rearrangement unit

144‧‧‧Video Synthesis Unit

145‧‧‧ Object Filter

150‧‧‧User interface

160‧‧‧ display module

411, 412, 413, 421, 422, 423, 431, 432, 433‧‧‧ bounding box

1100, 1200‧‧‧ image frames

1101, 1102, 1103, 1201, 1202, 1203‧‧‧ objects

1210‧‧‧Image Center

FS2, FS3‧‧‧ frame offset

L _N ‧‧‧Time Range

P _a , P _b , P _c , P _d , P _e , P _f , P1, P2, P3, P4, P9, P101, P102, P103‧‧‧ object path

P9_1, P9_2‧‧‧ subpath

R‧‧‧ Radius

S210~S250, S241~S246,

S810, S1310~S1340‧‧‧ steps

SC‧‧‧ adjustment value

T _L ‧‧‧Play time length / length of composite video

The length of the original video of _ov ‧‧‧

FIG. 1 is a block diagram showing a circuit of a video playing device according to an embodiment of the invention.

2 is a schematic flow chart of a video playing method according to an embodiment of the invention.

FIG. 3 is a block diagram showing the circuit of the object path extraction module and the video synthesis module shown in FIG. 1 according to an embodiment of the invention.

4A, 4B, and 4C illustrate the relationship between objects that appear in different frames.

Figure 5A is a diagram showing an image of an office scene in accordance with an embodiment of the present invention.

Figure 5B shows an extruded view corresponding to the video shown in Figure 5A.

6A is a schematic diagram showing an image of a train station platform scene in accordance with another embodiment of the present invention.

Fig. 6B shows an overcrowded view corresponding to the video shown in Fig. 6A.

7A and 7B are schematic diagrams illustrating an original video and a composite video in accordance with an embodiment of the present invention.

FIG. 8 is a schematic flow chart showing the detailed implementation of step S240 shown in FIG. 2 according to an embodiment of the invention.

FIG. 9 is a schematic diagram showing the operation of step S243 shown in FIG. 8 according to an embodiment of the present invention.

10A, FIG. 10B and FIG. 10C are schematic diagrams showing the time position of the object path rearrangement unit initializing the adjusted object path according to an embodiment of the invention.

11 is a schematic diagram illustrating an image obtained by a general camera in accordance with an embodiment of the present invention.

Figure 12 is a schematic illustration of an image obtained by a fisheye camera in accordance with an embodiment of the present invention.

FIG. 13 is a schematic flowchart diagram of a video playing method according to another embodiment of the present invention.

FIG. 14 is a block diagram showing the circuit of the video synthesizing module of FIG. 1 according to another embodiment of the present invention.

The term "coupled" as used throughout the specification (including the scope of the patent application) may be used in any direct or indirect connection. For example, if the first device is described as being coupled to the second device, it should be construed that the first device can be directly connected to the second device, or the first device can be connected through other devices or some kind of connection means. Connected to the second device indirectly. In addition, wherever possible, the elements and/ Elements/components/steps that use the same reference numbers or use the same terms in different embodiments may refer to the related description.

The following embodiments will explain a video playback method and/or a video playback device. The video playing method and/or video playing device can solve The problem of playing long videos, which are taken by the fixed camera for a user-predetermined length of time. In some embodiments, but not limited to, the video playing method and/or the video playing device may estimate a suitable shortest playing length (proposed length of time) before synthesizing the image for playing, and The suggested length of time is prompted to the user. The user can refer to the length of time suggested by the device to set the expected length of play time. The video playing method and/or the video playing device may generate a composite video having a length of time desired by the user according to the length of the playing time. A composite video can contain all the objects that appear in the original video (or all objects of interest). Objects in the composite video appear in the same order as the original video. In some embodiments, but not limited to, the video playback method and/or video playback device may display overlapping objects in the composite video in a semi-transparent manner. In other embodiments, but not limited to, the video playing method and/or the video playing device may occluding a distant object with a near object to make the synthesized video look It looks like a video taken from a camera.

FIG. 1 is a block diagram showing a circuit of a video playback apparatus 100 according to an embodiment of the invention. The video playback device 100 includes a camera module 110, an object path extraction module 120, a storage module 130, a video synthesis module 140, a user interface 150, and a display module 160. The photography module 110 is coupled to the object path extraction module 120. The photography module 110 can capture a scene to obtain an original video 11. The object path extraction module 120 can extract at least one object path 12 from the original video 11. The storage module 130 is coupled to the object path extraction module 120. The object path extraction module 120 can provide the object path 12 directly to the video synthesis module 140 or store the object path 12 in the storage module 130 according to different design requirements.

The video synthesis module 140 is coupled to the object path extraction module 120 and the storage module 130 to receive the object path 12 . The video synthesis module 140 can selectively adjust the object path 12 according to a preset play time length to synthesize the object path 12 into the composite video 13. The length of the play time determines the length of time of the synthesized video 13 , and the length of the synthesized video 13 is less than the length of time of the original video 11 . According to different design requirements, the length of the play time may be a preset value of a fixed setting or a play time length value determined dynamically by the user.

The user interface 150 is coupled to the video synthesis module 140. The user interface 150 can transmit the start time T _b and the end time T _e input by the user to the video synthesis module 140. The user can determine the time range of the object to view the original video 11 by setting the start time T _b and the end time T _e . After determining the time range of the original video 11 to be viewed, the video composition module 140 may synthesize the object path to which the time range belongs into the composite video 13. The length of the synthesized video 13 is in accordance with the preset playing time length. The length of time in which the video 13 is synthesized is not related to the content of the original video 11. The display module 160 is coupled to the video synthesis module 140. The display module 160 can play the synthesized video 13 generated by the video synthesis module 140 for viewing by the user.

2 is a flow chart of a video playing method according to an embodiment of the invention. Schematic diagram. Step S210 provides an original video, wherein the original video is obtained by shooting a scene by the photography module. In step S220, at least one object path is extracted from the original video by the object path extraction module. In step S230, a play time length is provided to determine a time length of the synthesized video, wherein the synthesized video has a time length smaller than a length of the original video. In step S240, the object path is selectively adjusted by the video synthesis module to synthesize the object path into the composite video. In step S250, the composite video is played for viewing by the user. 2 and FIG. 1 can be referred to each other, and therefore will not be described again. In some embodiments, the video playback method of FIG. 2 can be implemented in a hardware circuit (such as video playback device 100 shown in FIG. 1). In other embodiments, the video playback method of FIG. 2 can be implemented in firmware. This firmware can run on a central processing unit, microcontroller or other firmware operating platform. In other embodiments, the video playing method described in FIG. 2 can be implemented in a software. This software can be stored or run on a computer, smart phone or other software running platform.

In some application scenarios, the object path may include a first item path and a second item path. After the first object path and the second object path are selectively adjusted via the video synthesis module 140, the playback speed of the first object path in the composite video 13 may be different from the playback speed of the second object path. The playback speed of the object path depends on the set playback time length T _L . For example, in a practical application scenario, if the playing time length of the first object path and the second object path are both less than the threshold length P _th (the threshold length P _{th is} less than or equal to the playing time length T _{L of the} synthesized video 13), The playback speed of the first object path and the second object path may be the same. If the playing time length of the first object path and/or the second object path is greater than the threshold length P _th , the video synthesizing module 140 may adjust the playing speed of the first object path and/or the second object path according to the threshold length P _th So that the playback speed of the first object path and the second object path may be different (or the same).

In some application scenarios, the object path may include a first item path and a second item path. The time at which the first object of the first object path and the second object of the second object path appear in the original video 11 does not overlap. After the first object path and the second object path are selectively adjusted via the video synthesis module 140, the first object overlaps with the time when the second object appears in the composite video 13. For example, the object of the object path P1 shown in FIG. 7A does not overlap with the time when the object of the object path P2 appears in the original video 11. After the object path P1 and the object path P2 are selectively adjusted via the video synthesis module 140, the object of the object path P1 overlaps with the time when the object of the object path P2 appears in the composite video 13, as shown in FIG. 7B. It should be noted that the spatial position of the object path in the composite video 13 is the same as the spatial position of the object path in the original video 11. For example, the object paths P1, P2, P3, and P4 shown in FIG. 7B have the same spatial position in the composite video 13 as the spatial positions of the object paths P1, P2, P3, and P4 shown in FIG. 7A in the original video 11. 7A and 7B will be described later in detail.

FIG. 3 is a block diagram showing the circuit of the object path extraction module 120 and the video synthesis module 140 shown in FIG. 1 according to an embodiment of the invention. The embodiment shown in FIG. 3 can be analogized with reference to the related description of FIG. 1 and FIG. 2.

Referring to FIG. 3, the object path extraction module 120 includes object detection and back An object detection and background extraction unit 121 and an object path generation unit 122. Objects should be detected first before the object path is generated. As described in U.S. Patent No. 8,599,255, "object" is defined as the foreground of a scene in a frame of a video stream. In contrast, "background" is defined as a static scene that is nearly constant or has only a slight difference in the time series frame of the video. The object detection and background extraction unit 121 can receive the original video 11 and extract at least one object and at least one background image from the original video 11. The object detection and background extraction unit 121 can extract the object and the background image from the original video 11 using any algorithm. For example, but not by way of limitation, in some embodiments, the object detection and background extraction unit 121 can be from the original video 11 using the method described in US Pat. No. 8,599,255 or other known methods. Extract objects and background images.

Each object and each background image has a timestamp corresponding to its source frame. The object detection and background extraction unit 121 may store the background image in the storage module 130. According to different design requirements, the storage module 130 may include a storage device (such as a hard disk, a solid state drive, etc.), a memory, a buffer, or other data storage media. In some embodiments, the object detection and background extraction unit 121 can store background images of all source frames in the storage module 130. In other embodiments, not every background image is stored in order to save storage space. For example, every other one The background image is selected for storage during the constant period of the segment.

The object path generating unit 122 is coupled to the object detecting and background extracting unit 121. The object path generating unit 122 may create the object path 12 according to the relationship between the object of the current frame in the original video 11 and the object in the previous frame, and store the object path 12 in the storage module 130. After object detection is performed on a frame, all detected objects in the current frame will be checked for their relationship to objects in the previous frame. A bounding box of objects can be used to establish the relationship. For example, if the bounding box of an object in the current frame overlaps the bounding box of the object in the previous frame, the two objects that appear in the sequence frame have a relationship therebetween. An object in the current frame is considered a child object, and an object in the previous frame is treated as a parent object.

Figure 4A illustrates the relationship between objects that occur in different frames. The plurality of blocks depicted in Figure 4A each represent a bounding box of objects that appear in different frames. Since these boxes (the bounding boxes of the objects) each overlap the box in its previous frame (the bounding box of the object), the boxes (the bounding box of the object) depicted in Figure 4A are related. The object path generation unit 122 may make these boxes (boundary boxes of objects) having an association into one object path.

4B and 4C respectively illustrate the relationship between objects appearing in different frames. The plurality of blocks depicted in Figures 4B and 4C each represent a bounding box of objects that appear in different frames. The generation of the object path includes three cases: 1) creating a new object path; 2) adding members to the existing object path; and 3) ending the object path. The three cases will cooperate with Figures 4A, 4B and 4C The description is as follows.

There are three conditions for creating a new object path: 1) the object in the current frame has no parent object (the corresponding object in the previous frame); 2) the object in the current frame shares the same parent object as the other objects; or 3) the current The object of the frame has multiple parent objects. The object path generating unit 122 may create a new object path according to an object satisfying at least one of the above conditions, and the object of the current frame is the first object of the new object path. For example, if the bounding box 411 of the object shown in FIG. 4A has no parent object, the object path generating unit 122 may use the bounding box 411 of the object as the first object of the newly created object path. 4B, the bounding box 422 of the object in the current frame shares the same parent object (the bounding box 421 of the object) with the bounding box 423 of the object in the current frame, and the object path generating unit 122 can newly create the bounding box 422 of the object. The first object of the object path P _b and the bounding box 423 of the object are the first object of the newly created object path P _c . 4C shows that the bounding box 433 of the object in the current frame has a plurality of parent objects (the bounding boxes 431 and 432 of the object), and the object path generating unit 122 can use the bounding box 433 of the object as the newly created object path P _f An object.

If the object of the current frame has a unique parent object and the object of the current frame is the only child of the parent object, the object path generation unit 122 adds the object of the current frame to the existing object path to which the parent object belongs. For example, the bounding box 412 of the object shown in FIG. 4A has a unique parent object (the bounding box 411 of the object), and the bounding box 412 of the object is the only child of the parent object (the bounding box 411 of the object), then the object path generating unit 122 can add the bounding box 412 of the object to the parent object (The bounding box 411 of the object) belongs to the existing object path.

The object path generating unit 122 ends the object path when at least one of the following conditions occurs: 1) the last object of the object path has no child objects; 2) the last object of the object path has more than one child object; Or 3) the last object of the at least one object path and the other object path share the sub-object. For example, if the last object of the object path shown in FIG. 4A (the bounding box 413 of the object) has no child objects, the object path generating unit 122 ends the object path shown in FIG. 4A. The last object of the object path P _a shown in Fig. 4B (the bounding box 421 of the object) has a plurality of sub-objects (bounding frames 422 and 423 of the object), and the object path generating unit 122 ends the object path P _a shown in Fig. 4B. As shown in the last object (bounding box object 431) object path P _d and a last object path object (object boundary block 432) share the same sub-object P _e in Figure 4C, the path of the object generation unit 122 in FIG. 4C ends The object paths P _d and P _{e are shown} . When the first object of the object path has a parent object, the object path of the parent object is considered to be the parent object path of the current object path.

After the object path 12 is generated, the data of the object path 12 is stored in a storage module 130 (eg, a memory or storage device). The data of the object path 12 includes: a time length of the object path 12, a timestamp of the first object of the object path 12, and each member object of the object path 12 corresponds to the first object Time shift, the position of each member object, the size of each member object, and/or the parent object path. Thus, in some embodiments, the object path 12 can be three dimensional data, including time information and in two Location information in the dimension.

The user interface 150 shown in FIG. 3 can set parameters of the video synthesis program. The parameters include the start time T _b and the end time T _{e of the} original video 11 , the play time length of the synthesized video 13 (expected length), the parameters of the object filter (such as the object size, the color of the object, the movement of the object.... . . and so on, which will be explained in detail in other embodiments to be described later, and/or parameters for generating a frame of synthesis video.

The video synthesis module 140 shown in FIG. 3 includes a video length evaluation unit 141, an object path collection unit 142, an object path rearrangement unit 143, and a video synthesis. Unit 144. In some embodiments, the object path collection unit 142 may collect some or all of the object paths from the object path 12 generated by the object path extraction module 120. In other embodiments, the object path collecting unit 142 may collect some or all of the objects from the object path in the storage module 130 according to the start time T _b and the end time T _e output by the user interface 150 . path. In the case where the length of the original video 11 is too long, or for a surveillance application, the user usually views a segment of the original video 11 instead of the full video. The user can use the user interface 150 to set a time period of interest (eg, set the start time T _b and the end time T _e ). The object path collecting unit 142 can collect/select the corresponding object path according to the start time T _b and the end time T _e provided by the user interface 150. It is assumed that the occurrence time and length of a certain candidate object path in the storage device (for example, the storage module 130) are P _t and P _{l , respectively} . If T _b P _t T _e , or T _b P _t + P _l T _e , or P _t T _b and T _e P _t + P _l, then the video object path synthesis module 140 collection unit 142 may select this path as a candidate object of interest to the user object path.

The video length estimating unit 141 is coupled to the object path collecting unit 142 to receive the collected result of the object path collecting unit 142. The video length estimating unit 141 can estimate the recommended length of time according to the crowding situation at the different pixels in one scene of the collected object path of the object path collecting unit 142. For the user, it is difficult to determine the appropriate video length for shortening the play time because the complexity of the original video 11 varies in different time periods. Therefore, suggestions to facilitate the user are necessary in order to determine the appropriate length of play. The recommended length of time estimated by the video length estimating unit 141 can be presented to the user to assist the user in determining the length of play time of the composite video 13.

In this embodiment, the video length estimating unit 141 may generate a crowdedness map according to the original video 11 to describe the crowded value of the at least one object path at different pixels. The concept of a crowded map is to compress all objects of interest in one frame. The crowd map can be used as a counter for the crop piece path. The initial value of the squeeze map at each pixel C _ij is zero, where C _ij represents the squeezed value at the position ( i,j ) of the frame. An object path is defined as a bounding box of a set of objects in a time sequence. If the position ( i, j ) is in the bounding box of an object path, then C _ij = C _ij +1. After counting all the bounding boxes of all interested object paths, the crowded map is created.

5A is a diagram showing an office scene in accordance with an embodiment of the present invention. Like a schematic. Figure 5B shows an extruded view corresponding to the video shown in Figure 5A. As shown in Figure 5B, a high crowding value will appear in the center of the frame (corresponding to the aisle in the office shown in Figure 5A) as all object paths pass through the aisle. In contrast, FIG. 6B is not crowded as in FIG. 5B. 6A is a schematic diagram showing an image of a train station platform scene in accordance with another embodiment of the present invention. Fig. 6B shows an overcrowded view corresponding to the video shown in Fig. 6A. As shown in Fig. 6B, the distribution of the crowding values is relatively uniform (corresponding to the train station platform shown in Fig. 6A). Comparing FIG. 5B with FIG. 6B, the playback time of the video shown in FIG. 5A should be longer than the video shown in FIG. 6A.

In the embodiment, the video length estimating unit 141 can calculate the equation. Where F _n represents the number of suggested frames, C _m represents the associated value of the crowded values in the crowded map, C _th represents the threshold, T _p represents the suggested length of time, and R _f represents the frame playback rate of the composite video (frame Rate). The maximum value of the crowded map is related to determining the appropriate length of play time. The threshold value C _th is smaller than the correlation value is greater than 0 and C _m. When the correlation value C _m is smaller than the threshold value C _th, the user can easily distinguish the different objects in a scene. In order to eliminate the influence of noise, the correlation value C _m and the threshold C _th can be determined according to design requirements. For example (but not limited to), C _m associated value may be an average value of the congestion in the congested all pixels in FIG. In other embodiments, the correlation value C _m may be crowded in the crowded FIG these values in descending order, the front 50% of the average value of congestion. In other embodiments, the correlation value C _m may be an average value of the top 20% crowded, crowded or average of the first 10% of the value. In some embodiments, the threshold value C _th may be set to 72 to conform to the human visual perception (visual perception).

The object path rearranging unit 143 may rearrange the object path in the synthesized video 13 in accordance with the order of appearance of the object path selected by the object path collecting unit 142 in the original video 11. For example, Figures 7A and 7B are schematic diagrams illustrating an original video 11 and a composite video 13 in accordance with an embodiment of the present invention. In Figs. 7A and 7B, the vertical axis represents space (position of objects), and the horizontal axis represents time. It is assumed here that the object paths P1, P2, P3 and P4 appear at different times in the original video 11. The object path rearranging unit 143 may rearrange the object paths P1, P2, P3, and P4 in the synthesized video 13 in accordance with the order of appearance of the object paths P1, P2, P3, and P4 in the original video 11. As can be seen from FIG. 7A and FIG. 7B, the time length T _{L of the} composite video 13 is smaller than the time length T _{ov of the} original video 11, so that the video playing method and the video playing device 100 of the embodiment can shorten the video playing time.

FIG. 8 is a schematic flow chart showing the detailed implementation of step S240 shown in FIG. 2 according to an embodiment of the invention. In the embodiment shown in FIG. 8, step S240 includes sub-steps S810, S244, S245, and S246. In step S810, the object path rearranging unit 143 may selectively adjust the object path selected by the object path collecting unit 142 to obtain at least one adjusted object path. For example, but not limited to, step S810 shown in this embodiment includes sub-steps S241, S242, and S243.

In step S241, if the object path has a parent object path, the object path and its parent object path are merged as the adjusted object path. When multiple objects span each other, the object path is divided into several object paths. For example, the condition shown in Figure 4C may be that two items move from different positions to the same position, while the condition shown in Figure 4B may be that two items move from the same position to a different position. Therefore, the first step in the object path rearrangement is to merge the relative object paths (step S241 shown in Figure 8) to recover the entire story of the cross-over object path. Since each object path records its parent object path, this information is used to construct an inheriting tree structure to merge all relative object paths together. For example, the object path rearranging unit 143 may merge the object paths P _a , P _b and P _c shown in FIG. 4B into an individual object path, which is referred to herein as an adjusted object path, in step S241. . Object path rearrangement unit 143 may also object path P _d shown in FIG. 4C in the step S241, P _e P _f and merged into a path of objects adjusted. In addition, an object path without a parent object path can also be considered an adjusted object path.

The next step in the rearrangement of the object path is to place all individual object paths in the composite video 13. When the user wants to perform compressed playback in a short period of time, the length of the object path may exceed the length of the composite video 13, so the embodiment shown in Fig. 8 uses two methods to shorten the object path. The two methods are object path speedup (step S242 shown in Fig. 8) and object path splitting (step S243 shown in Fig. 8).

If the time length of the adjusted object path provided in step S241 is greater than the threshold length P _th , the object path rearranging unit 143 may accelerate the adjusted object path provided in step S241 according to a speedup factor S _p in step S242. The playback speed is shortened by the length of time. Threshold P _th length of less than or equal composite video playback time length T _L 13, and the acceleration factor S _p is a real number. This embodiment does not limit the length of the threshold P _th acceleration factor S _p embodiments. For example (but not limited to), it may be greater than the threshold P _th length equal to a quarter or synthetic video playback time length T _L of 13. In the present embodiment, the length of the threshold P _th is set equal to the playback time of one-half the length _L T. The following Examples will illustrate various embodiments of the acceleration factor S _p embodiments.

In some embodiments, the acceleration of the object path, concern is the ratio of the length _L of the object path P _l time and 13 times the length of the composite video T. If the time length P _l of the object path is longer than half the time length T _L of the composite video 13, the object path will be shortened. Object path rearrangement unit 143 in step S242 may calculate the equation _{S p = (P l / P} th), in order to achieve acceleration factor S _p. In the present embodiment, the threshold length P _th = (T _L /2). When the acceleration factor is greater than the maximum acceleration value S _p S _max, then the acceleration factor S _p as the maximum acceleration value S _max, where S _max is greater than the maximum acceleration value of 1 and less than 4. When the acceleration factor is less than S _p 1, S _p factor is set to 1 will be accelerated. Therefore, if the time length P _l of the object path is less than half of the time length T _L of the composite video 13, the object path is not shortened.

It can be seen that the playback speed of the object path depends on the set playback time length T _L . For example, in a practical application scenario, if the playing time length of the first object path and the second object path are both less than the threshold length P _th , the playback speed of the first object path and the second object path may be the same as in the original video. The playback speed in 11, that is, the playback speed of the first object path and the second object path may not need to be accelerated. If the playing time length of the first object path and/or the second object path is greater than the threshold length P _th , the video synthesizing module 140 may adjust the playing speed of the first object path and/or the second object path according to the threshold length P _th So that the playback speed of the first object path and the second object path may be different (or the same).

In other embodiments, the object path that traverses the hot zone of crowdedness map requires acceleration to reduce overlap with other object paths. Here, the "hot zone" is defined as a region where the crowded value is relatively larger than the crowded value of other regions, and the certain region may be referred to as a hot zone. FIG calculated by using the object path congestion in a crowded figures representative of a congestion value C _p. In this embodiment, the bounding box of all the object paths is first projected onto the crowded map, and then the representative forced value C _{p of the} pressed map is found in the projected area. The present embodiment assumes a representative of object path C _p value of the congestion is congestion at the maximum value in the object path crowded FIG. If the path to the object on behalf of overcrowding FIG overcrowded C _p value of equal to or greater than the upper limit value of the congestion C _U, the object path rearrangement unit 143 may be set to the maximum value of acceleration at step S acceleration factor S _p S242 will _Max , wherein the upper limit forced value C _U and the maximum acceleration value S _max are real numbers. The upper limit forced value C _U and the maximum acceleration value S _max can be determined according to design requirements. If the representative value of the congestion C _p value of less than or equal to the lower limit C _L congestion, the object path rearrangement unit 143 may be set in a step in the acceleration factor S _p S242 1, wherein the lower limit value of the congestion C _L is a real number, and the lower limit The forced value C _{L is} less than the upper limit forced value C _U . The lower limit of the forced value C _L can be determined according to the design requirements. If the representative congestion crowded than the lower limit value of C _p and C _L than the upper limit value of the congestion value C _U, the object path rearrangement unit 143 may be set in a step in the acceleration factor S _p S242 [(C _p - C _L) /( C _U - C _L )]*( S _max -1)+1.

In other embodiments, article 143 may route rearrangement unit may calculate the first factor S _g at step _{S242 = (P l / P th} ), in order to achieve acceleration factor S _p. In the present embodiment, the threshold length P _th = (T _L /2). If the first factor S _{g is} greater than the maximum acceleration value S _max , the object path rearranging unit 143 sets the first factor S _g to the maximum acceleration value S _max , wherein the maximum acceleration value S _max is a real number greater than 1 and less than 4. If the first factor S _{g is} less than 1, the object path rearranging unit 143 sets the first factor S _g to 1. If the representative forced value C _{p of} the object path in the forced map is greater than or equal to the upper limit forced value C _U , the object path rearranging unit 143 sets the second factor S _c to the maximum acceleration value S _max . If the representative forced value C _{p is} less than or equal to the lower limit forced value C _L , the object path rearranging unit 143 sets the second factor S _c to 1. If the representative forced value C _{p is} greater than the lower limit forced value C _L and less than the upper limit forced value C _U , the object path rearranging unit 143 sets the second factor S _g to [( C _p - C _L )/( C _U - C _L )]*( S _max -1)+1. Object path rearrangement unit 143 may be selected from the big to the first factor and the second factor S _g S _c in step S242, the acceleration factor as S _p.

After completing the accelerated object path (step S242), there may be some object paths longer than the playback time length T _{L of the} composite video 13. In order to handle an extremely long object path, step S243 can split this extremely long object path into several short sub-paths. In step S243, if the time length P _l of the object path processed in step S242 is greater than the threshold length P _th , the object path rearranging unit 143 may split the object path into a plurality of sub paths as the adjusted Object path. The object path rearranging unit 143 can adjust the frame shift of the first sub-paths of the sub-paths to other sub-paths to reduce the overlapping area of the sub-paths.

For example, FIG. 9 is a schematic diagram showing the operation process of step S243 shown in FIG. 8 according to an embodiment of the present invention. P9 object path length of time shown in FIG. 9 is assumed here that the length is greater than the threshold P _th. The object path rearranging unit 143 may split the object path P9 into a plurality of sub-paths, for example, sub-paths P9_1 and P9_2 in step S243. The two adjacent sub-paths P9_1 and P9_2 in the time series have a small overlap in time space (as shown in FIG. 9), that is, the tail end of the first sub-path P9_1 is identical to the second sub-path. The head end of P9_2. The object path rearrangement unit 143 may rearrange the times of the sub-paths P9_1 and P9_2 to advance the occurrence time of the sub-path P9_2 that occurs later to be the same as the sub-path P9_1.

After rearranging the sub-paths P9_1 and P9_2, in some cases (such as a wandering object), there may be a large amount of overlap between these sub-paths P9_1 and P9_2 in the position space. To solve this problem, a time shift is added to each subpath. This problem can be formulated as a minimum cost problem. The cost function E(t) is defined as . Where t = { t ₀ , t ₁ , ..., t _N } represents a set of frame shifts corresponding to each sub-path of the first sub-path P9_1, and t ₀ =0, t ₀ t ₁ ... t _N , N is the number of sub-paths. For example, the frame offset of the first sub-path P9_1 is t ₀ , and the frame offset of the sub-path P9_2 is t ₁ . P _i ( t _i ) is the i-th sub-path with the frame offset t _i , and the function O ( P _x , P _y ) calculates the overlap region between the two sub-paths P _x and P _y . To minimize the cost function E(t), the following conditions must be met: R x T _L > L _N , where L _N is the time range of all sub-paths and R is a constant less than one. If the R value is smaller, the length of the subpath is shorter. Therefore, the object path rearranging unit 143 can adjust the frame offsets of the first sub-paths P9_1 to the other sub-paths P9_2 among the sub-paths in step S243 to reduce the overlapping area of the sub-paths.

After the step S243 is completed, the object path rearranging unit 143 can proceed to step S244. In accordance with the order of appearance of the object path in the original video 11, the object path rearranging unit 143 may initialize the time position of the adjusted object path provided in step S243 in the synthesized video 13 in step S244. All object paths are rearranged first. If there is a gap between the first object path and the second object path in the adjusted object path, the object path rearranging unit 143 advances the time of the first object path and the second object path. And the time of the subsequent occurrence is later than the time of the first object path and the first occurrence in the second object path. The object path rearranging unit 143 multiplies the time offsets of the adjusted object paths by the same adjustment value, respectively, such that the time range of the adjusted object path is included in the range of the playback time length T _L of the composite video 13 .

For example, FIG. 10A, FIG. 10B and FIG. 10C are schematic diagrams illustrating the time position of the object path rearranging unit 143 initializing the adjusted object path according to an embodiment of the present invention. FIG. 10A illustrates object paths P101, P102, and P103 with time gaps between object paths P102 and P103. Therefore, referring to FIG. 10B, the object path rearranging unit 143 can advance the time of the object path P103 in step S244 so that there is no gap between the object paths P102 and P103. The time of the object path P103 advanced is later than the time of the object path P102. As shown in FIG. 10B, the time range L _{N of} all object paths exceeds the playback time length T _{L of} the composite video 13. The object path rearranging unit 143 can determine the adjustment value SC2 of the object path P102 according to the frame offset FS2 of the object path P102, the length of the object path P102, and the playback time length T _{L of the} synthesized video 13 (a real number less than 1) ). Wherein, after the frame offset FS2 is multiplied by the adjustment value SC2, the object path P102 can be included in the range of the playback time length T _L of the synthesized video 13. The object path rearranging unit 143 can also determine the adjustment value SC3 of the object path P103 according to the frame offset FS3 of the object path P103, the length of the object path P103, and the playing time length T _{L of the} synthesized video 13 (less than 1). Real number). Wherein, after the frame offset FS3 is multiplied by the adjustment value SC3, the object path P103 can be included in the range of the playback time length T _L of the synthesized video 13. Next, the object path rearranging unit 143 may select the smallest one of these adjustment values (for example, SC2 and SC3) as the adjustment value SC. The object path rearranging unit 143 multiplies the time offsets FS2 and FS3 of the object paths P102 and P103 by the same adjustment value SC, respectively, in step S244, so that the time range L _{N of} the object paths P101, P102, and P103 is included in The range of the playback time length T _L of the composite video 13 is as shown in FIG. 10C. Since the time offsets of all object paths are multiplied by the same adjustment value, the order of appearance of the object paths in the original video 11 can be preserved.

If it is determined in step S245 that the time length of the object path is greater than the threshold length P _th , or the time range L _{N of} all the object paths exceeds the play time length T _{L of} the synthesized video 13, the steps S242, S243, and S244 are performed again. And S245.

The final step in the object path rearrangement is to optimize the position of the object path in the time domain (step S246 shown in Figure 8). In accordance with the overlapping situation of the object path in the composite video 13 provided in step S244, the object path rearranging unit 143 adjusts the time position of the object path in the composite video 13 in step S246. For example, the object path rearranging unit 143 can adjust a frame offset between the first object path and the other object path in the adjusted object path to reduce overlapping areas of the adjusted object paths. The purpose of the optimizer is to get the best position of the object path in the time domain. The optimal position of multiple object paths in the time domain indicates that these object paths have minimal overlap areas. Apply the cost function again The result of the object path rearrangement is a set of frame offsets S = { S ₀ , S ₁ , ..., S _x } corresponding to the object path, where S ₀ =0, S ₀ S ₁ ... S _x , and x is the number of object paths. P _i ( t _i ) is the i-th object path with the frame offset t _i , and the function O ( P _x , P _y ) calculates the overlap region between the two object paths P _x and P _y .

Referring to FIG. 3, the video synthesizing unit 144 is coupled to the object path rearranging unit 143 to receive the rearrangement result of the object path rearranging unit 143. The video synthesizing unit 144 can synthesize the object path provided by the object path rearranging unit 143 and the background image provided by the storage module 130 into the synthesized video 13. When a frame of composite video 13 contains multiple objects, these objects may be from different frames of the original video 11. For example, considering two object paths P ₁ and P ₂ , it is assumed that T _b ¹ and T _b ² are each the object paths P ₁ and P ₂ at the start time of the original video 11 and are rearranged by the object path rearranging unit 143. The corresponding frame offsets are S ₁ and S _{2 , respectively} . If _{S 1 + k = S 2 +} m, the object path P k-th frame and _an article path P ₂ m-th frame in the composite video will be displayed in the same frame 13. The time stamps of the two objects are T _b ¹ +k and T _b ² +m. T _b ¹ + k and T _b ² + m is not the same, unless ^{_{T b 1 -T b 2 = S}} 1 -S 2.

To synthesize the video, the first step is to determine which objects are in the video frame. Assuming that the composite video 13 has N frames in the user-predetermined time length T _L , the object of the k- th frame of the object path P _i is a member of the (k+ S _i )th frame of the composite video 13 Where S _i is the frame offset of the object path P _i and k+ S _i N. The video playback device 100 can synthesize all of the objects of interest into the composite video 13.

Background images are another key factor in video synthesis. The video compositing program pastes the object onto the background image to produce a video frame. In this embodiment, the selection of the background image is based on the timestamp of the object in the frame. For a video frame, it contains n objects O = { O ₀ , O ₁ ,..., O _{n 1} }, and the time stamp of these objects is T = { T ₀ , T ₁ ,..., T _{n 1} } . The video synthesizing unit 144 may select a background image whose time stamp is closest to the time stamp value T ^bg from among the plurality of background images in the storage module 130 to perform video frame synthesis. The timestamp value T ^bg is obtained from the time stamp of each object in the video frame. For example, but not limited to, the timestamp value T ^bg is equal to the average of the time stamps of the objects in a video frame, ie . In other applications, the timestamp value T ^bg may be the intermediate value of the time stamp T of these objects, or one of the values of the time stamp T. For example, if the object O _i is an important significant object, the video synthesis unit 144 can select a time stamp of the background image that is closest to T _i .

After the selection of the background image is completed, the video synthesizing unit 144 can perform object and background blending. This embodiment does not limit the manner in which the object and the background are mixed. For example, in some embodiments, video synthesis unit 144 can blend the background image with the object image of the object path using a Gaussian blending method. The object image (boundary frame area) may contain the complete object and partial background of the original video 11. Therefore, if the object image is directly pasted on the background image, the boundary of the object image may become very noticeable/unnatural. In order to solve this problem, the present embodiment can mix a background image and an object image by a Gaussian mixture method. The Gaussian mixture applied to the object image and the background image is defined as F' _ij =w _ij *F _ij +(1-w _ij )*B _ij , where F _ij is the image of the object at position (i,j) Pixels, B _ij are the pixels of the background image at position (i, j), F' _ij is the pixel of the object image after Gaussian blending is applied, and w _ij is the weight value of the Gaussian mixture. When th |F _ij -B _ij |, w _ij =1, otherwise . Where th is the threshold value used to determine the background similarity, and b is the constant value for Gaussian function.

In other embodiments, video synthesis unit 144 may utilize an alpha blending method to blend object images that overlap each other in the object path in a translucent manner. When the frame of the composite video 13 contains a plurality of objects, there is a possibility of overlapping regions between the object images. Let one object overlap to cover another, which will lose information about the object. The alpha blending method is used to display overlapping objects in a semi-transparent manner to avoid information loss issues. The alpha blending method is expressed as a formula , where F' _ij is the pixel (pixel) at position (i, j) after applying alpha blending, and n represents the number of overlapping objects at position (i, j); Is the pixel at the position (i, j) of the kth object in the overlap region.

In other embodiments, video synthesis unit 144 may utilize z-ordering to generate composite video 13. The concept of the Z sequence is that a near object will cover a distant object. This method requires defining the z-distance of the object in a two-dimensional image. The video synthesis unit 144 can calculate the z-axis distance of the plurality of objects in the object path. Based on the order of power reduction of the z-axis distance of these objects, the video synthesizing unit 144 can sequentially paste the objects onto the background image. That is, the object with the largest z-axis distance is first pasted on the background image, and the object with the smallest z-axis distance is finally pasted on the background image.

For an image obtained from a general camera, the z-axis distance of the object is inversely proportional to the maximum value of the y-coordinate of the object. For example, Figure 11 is a schematic diagram illustrating an image obtained by a general camera in accordance with an embodiment of the present invention. In FIG. 11, the vertical axis represents the Y-axis of the image frame 1100, and the horizontal axis represents the X-axis of the image frame 1100, wherein the origin of this X-Y coordinate system is located at the upper left corner of the image frame 1100. The closer the object is to the photographic module 110 (ie, the smaller the z-axis distance), the larger the maximum value of the Y coordinate of the object (ie, the Y coordinate of the lower edge of the object image). Taking FIG. 11 as an example, the maximum value of the Y coordinate of the object 1101 is greater than the maximum value of the Y coordinate of the object 1102, and the maximum value of the Y coordinate of the object 1102 is greater than the maximum value of the Y coordinate of the object 1103. Therefore, it can be seen that the z-axis distance of the object 1101 is smaller than the z-axis distance of the object 1102, and the z-axis distance of the object 1102 is smaller than the z-axis distance of the object 1103. After calculating the z-axis distance of the object, the objects 1101, 1102, and 1103 are ordered according to their z-axis distance. According to the order of power reduction of the z-axis distances of the objects 1101, 1102 and 1103, the video synthesizing unit 144 may The object 1103 is first pasted on the background image, and then the object 1102 is pasted on the background image, and then the object 1101 is pasted on the background image.

For an image obtained from a fisheye camera, the z-axis distance of the object is proportional to the minimum distance from the object to the image centroid. For example, Figure 12 is a schematic illustration of an image obtained by a fisheye camera in accordance with an embodiment of the present invention. In Figure 12, the fisheye image frame 1200 is circular with a radius r. The closer the object is to the photographic module 110 (ie, the smaller the z-axis distance), the smaller the minimum distance of the object from the image center 1210. Taking FIG. 12 as an example, the minimum distance from the object 1201 to the image center 1210 is less than the minimum distance from the object 1202 to the image center 1210, and the minimum distance from the object 1202 to the image center 1210 is less than the minimum distance from the object 1203 to the image center 1210. . Therefore, it can be seen that the z-axis distance of the object 1201 is smaller than the z-axis distance of the object 1202, and the z-axis distance of the object 1202 is smaller than the z-axis distance of the object 1203. After calculating the z-axis distance of the object, the objects 1201, 1202, and 1203 are ordered according to their z-axis distance. According to the order of power reduction of the z-axis distances of the objects 1201, 1202 and 1203, the video synthesizing unit 144 may first paste the object 1203 on the background image, and then paste the object 1202 on the background image, and then the object 1201 is next. Paste on the background image.

FIG. 13 is a schematic flowchart diagram of a video playing method according to another embodiment of the present invention. Step S1310 provides an original video, wherein the original video is obtained by shooting a scene by the photography module. In step S1320, the object path extraction module extracts at least one object path and a background image from the original video. Steps S1310 and S1320 shown in FIG. 13 may refer to steps S210 and S220 shown in FIG. 2. Related instructions and so on. Step S1330 calculates a z-axis distance of the plurality of objects in the object path. According to the power-down sequence of the z-axis distances of the objects, the object is sequentially pasted on the background image in step S1340 to synthesize the object path into the composite video. The length of the synthesized video is less than the length of the original video. The process of synthesizing the object path to the synthesized video in step S1340 can be analogized with reference to the related description of step S240 shown in FIG. 2, and therefore will not be described again.

In some embodiments, the z-axis distances of the objects are inversely proportional to the maximum of the y-coordinates of the objects. In other embodiments, the z-axis distances of the objects are proportional to the minimum distance from the objects to the center of the image.

FIG. 14 is a block diagram showing the circuit of the video synthesizing module 140 of FIG. 1 according to another embodiment of the present invention. The embodiment shown in FIG. 14 can be analogized with reference to the related description of FIG. 1 to FIG. 13 and will not be described again. In the embodiment shown in FIG. 14, the video synthesis module 140 further includes an object filter 145. The object filter 145 is coupled to the object path collecting unit 142 to receive the collected result of the object path collecting unit 142. The user interface 150 shown in FIG. 14 can also set parameters of the object attributes. The object filter 145 is applied when the user wants to search for an object using the attributes of the object. Object properties may include size, color, texture, material, face, direction of motion, other physical properties, or behavior, depending on the application requirements. The object filter 145 may filter the object path provided by the object path collecting unit 142 according to the object attribute. The object filter 145 checks each object path provided by the screen object path collecting unit 142 and selects an object path that conforms to the object attribute. The selected object path may be part of a complete object path or object path. Object The object path selected by the piece filter 145 will be provided to the object path rearranging unit 143. The object path rearranging unit 143 may rearrange the object paths in the synthesized video 13 in accordance with the order of appearance of the object paths selected by the object filter 145 in the original video 11. The object path rearranging unit 143 and the video synthesizing unit 144 shown in FIG. 14 can be analogized with reference to the related description of FIG. 3, and therefore will not be described again.

In summary, the video playing method and the video playing device according to the embodiments of the present invention can determine the length of time of the synthesized video according to a fixed setting or a playing time length dynamically determined by the user. The video playback method and video playback device may extract at least one object path from the original video and selectively adjust the object path to composite the object path into the composite video. Therefore, the video playing method and the video playing device can display all the items of interest in a preset playing time length. The video playing method and the video playing device can shorten the video playing time, that is, the length of the synthesized video is less than the length of the original video.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

S210~S250‧‧‧Steps

Claims (13)

A video playing method includes: providing an original video, wherein the original video is obtained by capturing a scene by a photography module; providing a length of playing time to determine a length of time of a composite video, wherein the length of the composite video a time length less than the original video; extracting at least one object path from the original video; and selectively adjusting the at least one object path according to the play time length to synthesize the at least one object path into the composite video . The video playing method of claim 1, wherein the at least one object path comprises a first object path and a second object path, and the playback speed of the first object path is different in the composite video. The playback speed of the second object path. The video playing method of claim 1, wherein the at least one object path comprises a first object path and a second object path, and a first object and the second object path of the first object path The time at which a second object appears in the original video does not overlap, and the first object overlaps with the time at which the second object appears in the composite video. The video playing method of claim 1, wherein the spatial position of the at least one object path in the composite video is the same as the spatial position of the at least one object path in the original video. The video playing method according to claim 1, wherein the slave The step of extracting the at least one object path in the original video comprises: performing an object detection and a background extraction process to extract at least one object and at least one background image from the original video; according to a current frame in the original video Generating the at least one object path in relation to the at least one object of a previous frame; and storing the at least one background image and the at least one object path in a storage device. The video playing method of claim 5, wherein the step of creating the at least one object path comprises: if the object of the current frame is not in a parent object of the previous frame, or in the current frame If the object in the current object shares the parent object with other objects, or the object of the current frame has multiple parent objects, a new object path is created, wherein the object of the current frame is the first object of the new object path; If the object of the current frame has a unique parent object, and the object of the current frame is the only child object of the parent object, adding the object of the current frame to an existing object path to which the parent object belongs; And when the last object of the at least one object path has no child objects, or when the last object of the at least one object path has more than one child object, or when the last object of the at least one object path and the other When the object path shares a child object, the at least one object path ends. The video playing method of claim 5, wherein the at least one object path data comprises: a time length of the at least one object path, a time stamp of a first object of the at least one object path, a time offset of each member object of the at least one object path, a position of each member object, a position of each member object Size, or parent object path. The video playing method of claim 5, further comprising: providing a start time T _b and an end time T _e in the original video; and if T _b P _t T _e , or T _b P _t + P _l T _e , or P _t T _b and T _e P _t + P _l , where P _t and P _l are the occurrence time and length of a candidate object path in the storage device, respectively, and the candidate object path is selected as the at least one object path. The video playing method of claim 1, further comprising: calculating a recommended length of time according to the crowded situation of the at least one object path at different pixels in a scene; and prompting the suggested time length A user to assist the user in determining the length of the play. A video playing method includes: providing an original video, wherein the original video is obtained by capturing a scene by a photography module; providing a length of playing time to determine a length of time of a composite video, wherein the length of the composite video Less than the length of time of the original video; extracting a plurality of object paths from the original video; and rearranging the object paths in the composite video in accordance with the order of appearance of the object paths in the original video. A video playing method includes: Providing an original video, wherein the original video is obtained by capturing a scene by a photography module; providing a length of playing time to determine a length of time of the composite video, wherein the length of the composite video is less than the length of the original video Extracting at least one object path from the original video; selectively adjusting the at least one object path to synthesize the at least one object path into the composite video; different in one scene according to the at least one object path The crowded situation at the pixel calculates a suggested length of time; and prompts the suggested length of time to a user to assist the user in determining the length of the play. A video playback device includes: an object path extraction module configured to extract at least one object path from an original video; a video synthesis module coupled to the object path extraction module to receive the at least one object a path configured to selectively adjust the at least one object path to synthesize the at least one object path into a composite video according to a length of play time, wherein the length of the play time determines a length of time of the composite video, and the path The length of the composite video is less than the length of the original video; a camera module coupled to the object path extraction module configured to capture a scene to obtain the original video; and a display module coupled to the Video synthesis module configured to play the composite a video, wherein the at least one object path includes a first object path and a second object path, and in the composite video, a playback speed of the first object path is different from a playback speed of the second object path, the first A time when a first object of the object path and a second object of the second object path appear in the original video do not overlap, and the first object overlaps with a time when the second object appears in the composite video. A video playing method includes: providing an original video, wherein the original video is obtained by capturing a scene by a camera module; extracting at least one object path and a background image from the original video by an object path extraction module Calculating a z-axis distance of the plurality of objects in the at least one object path; and pasting the objects on the background image according to a power-down sequence of the z-axis distances of the objects, to at least one object The path is synthesized into a composite video, wherein the length of the composite video is less than the length of time of the original video.