EP2904541A1 - Détection de scène vidéo - Google Patents

Détection de scène vidéo

Info

Publication number
EP2904541A1
EP2904541A1 EP13763412.7A EP13763412A EP2904541A1 EP 2904541 A1 EP2904541 A1 EP 2904541A1 EP 13763412 A EP13763412 A EP 13763412A EP 2904541 A1 EP2904541 A1 EP 2904541A1
Authority
EP
European Patent Office
Prior art keywords
frame
target
shot
shots
scene
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP13763412.7A
Other languages
German (de)
English (en)
Inventor
Maneli NOORKAMI
Yi Linda Chan
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Publication of EP2904541A1 publication Critical patent/EP2904541A1/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/147—Scene change detection
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
    • G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording

Definitions

  • the video may be composed of individual video frames that may be grouped into a number of shots.
  • a shot may be characterized as a sequence of frames that are captured with a certain visual angle of a camera.
  • a scene may be characterized as a collection of shots that may be related in action, place, context, and/or time, with such relationship perhaps corresponding to the nature of the content or program. For example, in some examples of situation comedies, soap operas, and/or dramatic programs, a scene may be characterized as a continuous set of shots that capture a certain action taking place in a particular location.
  • a user While watching or browsing video content, a user may desire to access a particular scene or portion of the content related to a scene.
  • One approach to locating scenes within video content may involve grouping individual frames into shots by detecting shot boundaries at shot transitions.
  • Hard cut shot transitions in which the first frame of an appearing shot immediately follows the last frame of a disappearing shot, may be located by detecting differences in consecutive frames.
  • gradual shot transitions typically span multiple frames over which the disappearing shot gradually transitions to the appearing shot.
  • temporally adjacent frames may be a combination of the disappearing shot and the appearing shot.
  • a gradual shot transition may include smaller and nonlinear differences between consecutive frames, making it more challenging to accurately identify a shot boundary.
  • the shots may be clustered into scenes.
  • Algorithms that use K-mean clustering to cluster shots into scenes are known. These algorithms, however, typically depend upon an estimation of the number of expected clusters. As such, these approaches are highly sensitive to a correct estimation of the number of expected clusters. The corresponding algorithms are also relatively complicated and computationally expensive.
  • the correlation among individual frames that constitute a shot may be fairly reliable, the correlation among shots that comprise a scene may be more unpredictable, and may depend on the angle of the camera, the nature of the scene, and/or other factors. Accordingly, it can prove challenging to reliably and repeatedly identify scenes.
  • a scene detection system and related method for detecting a scene in video content may comprise a computing device including a processor and memory.
  • a scene detection program is executed by the processor using portions of the memory.
  • the scene detection program may be configured to identify a plurality of shots in the video content.
  • the scene detection program may select a target shot in the plurality of shots.
  • the scene detection program may then build a forward window including the target shot and having a first number of shots that are temporally ahead of the target shot, and a rearward window having a second number of shots that are temporally behind the target shot. For each of the shots in the forward window, the scene detection program may determine a dissimilarity between a selected shot and each of the other shots in the rearward window. If one of the determined dissimilarities is less than a scene boundary threshold, the scene detection program may determine that the scene does not begin at the target shot. If none of the determined dissimilarities is less than the scene boundary threshold, the scene detection program may determine that the scene begins at the target shot.
  • FIG. 1 is a schematic view of a scene detection system according to an embodiment of the present disclosure.
  • FIG. 2 is a schematic illustration of a series of consecutive video frames.
  • FIG. 3 is a flow chart of a method for detecting a hard cut shot transition according to an embodiment of the present disclosure.
  • FIG. 4 is a flow chart of a method for detecting a first category of gradual shot transitions according to an embodiment of the present disclosure.
  • FIG.5 is a flow chart of a method for detecting a second category of gradual shot transitions that may be not detected by the method shown in FIG. 4, according to an embodiment of the present disclosure.
  • FIG. 6 is a schematic illustration of a series of consecutive shots including a target shot, a temporally forward window of shots and a temporally rearward window of shots.
  • FIG. 7 is a flow chart of a method for detecting a scene according to an embodiment of the present disclosure.
  • FIG. 8 is a simplified schematic illustration of an embodiment of a computing device.
  • FIG. 1 shows a schematic view of one embodiment of a scene detection system 10 for detecting one or more scenes in video content.
  • the scene detection system 10 includes a scene detection program 14 stored in mass storage 18 of a computing device 22.
  • the scene detection program 14 may be loaded into memory 26 and executed by a processor 30 of the computing device 22 to perform one or more of the methods and processes described in more detail below.
  • Video content 34 may be stored in mass storage 18.
  • video content 34 may be received by the computing device 22 from removable computer-readable storage media 50, shown here in the form of a DVD.
  • the removable computer-readable storage media 50 may be used to store and/or transfer data, including but not limited to the video content 34, scene detection program 14, and other media content and/or instructions executable to implement the methods and processes described herein.
  • the removable computer-readable storage media 50 may also take the form of CDs, HD-DVDs, Blu-Ray Discs, EEPROMs, and/or floppy disks, among others.
  • the video content 34 may be received from and/or accessed at a remote source, such as server 38 via network 42.
  • the remote source may take the form of a cloud-based service.
  • Computing device 22 may take the form of a desktop computer, laptop computer, tablet computing, mobile computer, networking computer, gaming console, set- top box (e.g. cable television box, satellite television box) or any other type of suitable computing device. Additional details regarding the components and computing aspects of the computing device 22 are described in more detail below with respect to FIG. 8.
  • set- top box e.g. cable television box, satellite television box
  • the computing device 22 also may be operatively connected with one or more additional devices, such as server 38 via network 42.
  • Network 42 may take the form of a local area network (LAN), wide area network (WAN), wired network, wireless network, personal area network, or a combination thereof, and may include the Internet.
  • LAN local area network
  • WAN wide area network
  • wired network wireless network
  • personal area network personal area network
  • Internet personal area network
  • FIG. 2 a schematic diagram of several individual frames (i-1), (i), (i+1), (i+2), etc., from video content 34 is provided.
  • frame (i-1) is the frame temporally adjacent and behind frame (i)
  • frame (i+1) is the frame temporally adjacent and ahead of frame (i)
  • frame (i+2) is the frame temporally adjacent and ahead of frame (i+1), etc.
  • each of the frames (i-1), (i), (i+1), (i+2), etc. may represent an image captured by an image capture device, such as a video camera, or
  • a shot may be characterized as a continuous sequence of frames that are captured with a common visual angle of a camera. Accordingly, the frames constituting a shot may be visually correlated. It will also be appreciated that detecting a shot boundary at a transition between a first shot and a temporally adjacent second shot may be useful for performing higher-level video segmentation, such as scene detection.
  • a shot transition is a hard cut transition, in which the first frame of the appearing shot immediately follows the last frame of the disappearing shot.
  • FIG. 3 an embodiment of a method 300 for analyzing each frame of a plurality of frames in the video content 34 to identify one or more hard cut transitions is provided. The following description of method 300 is provided with reference to the software and hardware components of the scene detection system 10 described above and shown in FIG. 1. It will be appreciated that method 300 may also be performed in other contexts using other suitable hardware and software components.
  • the method 300 may include calculating a color histogram of each frame (i) in the video content 34.
  • the red, green, and blue pixel values of a frame are converted to the YCbCr color space.
  • a YCbCr color histogram of each frame (i) is then calculated using a number B of bins.
  • the number B of bins may be 16.
  • the number B of bins may be 4, 8, 12, 20, 32, 64, 128 or other suitable number of bins.
  • the method 300 may include normalizing each frame histogram by dividing the value in each bin by the number of pixels of the frame being analyzed.
  • the method 300 may include, for all frames in the video content 34, calculating a frame histogram similarity H between a first frame and a temporally adjacent second frame.
  • a frame histogram similarity H between a first frame and a temporally adjacent second frame.
  • the color histogram similarity between a first frame (i) and a temporally adjacent frame (i+1) may be calculated as follows:
  • i (b) is the ratio of the number of pixels whose Y value falls in bin b to the total number of pixels in frame (i).
  • the method 300 may include determining whether the following first condition is satisfied:
  • T1 is a first histogram similarity threshold value.
  • T1 may be approximately 2.85. In other examples, T1 may be 2.25, 2.45, 2.65, 3.05, 3.25, 3.45 or other suitable threshold value.
  • the method 300 may include determining whether all of the frames in the video content 34 have been examined to identify a hard cut transition. If all of the frames in the video content 34 have been examined, then the method 300 may end. If all of the frames of the video content 34 have not been examined to identify a hard cut transition, then the method 300 may proceed to the next frame comparison.
  • the method 300 may include determining whether the following second condition is satisfied:
  • T 2 is a second histogram similarity threshold value.
  • T 2 may be approximately 2.9.
  • T2 may be 2.3, 2.5, 2.7, 3.1, 3.3, 3.5 or other suitable threshold value.
  • the method 300 may include determining whether all of the frames in the video content 34 have been examined to identify a hard cut transition. If all of the frames in the video content 34 have been examined, then the method 300 may end. If all of the frames of the video content 34 have not been examined to identify a hard cut transition, then the method 300 may proceed to the next frame comparison.
  • the method 300 may determine that a shot boundary exists at frame (i). Alternatively expressed, the method 300 may use a frame histogram similarity to identify a hard cut transition between the frame (i) and the temporally adjacent frame (i + 1). [0028] It will be appreciated that the first condition described above may identify a hard cut transition at frame (i) when the similarity between frame (i) and the temporally adjacent frame (i+1) is less than the first histogram similarity threshold value T 1 .
  • the second condition described above may be utilized to rule out a fast camera movement condition which may cause a large difference between temporally adjacent frames (i-1) and (i) and result in H i-1,i ⁇ T 2, and also a large difference between temporally adjacent frames (i) and (i+1) and result in Hi,i+1 ⁇ T1.
  • a shot transition is a gradual shot transition, in which two shots are concatenated and transitioned in a gradual fashion over multiple frames.
  • gradual shot transitions include, but are not limited to, dissolving from one shot to another shot, fading from one shot to another shot, and wiping from one shot to another shot.
  • methods 400 and 500 for identifying a gradual shot transition and gradual transition start frame are provided.
  • the following description of methods 400 and 500 is provided with reference to the software and hardware components of the scene detection system 10 described above and shown in FIG. 1. It will be appreciated that methods 400 and 500 may also be performed in other contexts using other suitable hardware and software components.
  • the methods 400 and 500 for identifying a gradual shot transition may utilize the same frame histogram data that is calculated and utilized in the method 300 for identifying hard cut transitions as described above.
  • frame histogram data may be computed only once. Accordingly, the methods 400 and 500 for identifying a gradual shot transition described below may add only minor additional computational complexity to the systems and methods for detecting a scene in video content as described herein.
  • the method 400 may be used to identify gradual shot transitions to black (a first category). In this category of transitions, frames of a first shot transition to black with the last frame of the first shot being followed by the first frame of second shot.
  • the method 500 may be used to identify gradual shot transitions in which the last frame of a first shot transitions directly to the first frame of a second shot (a second category).
  • the first category of gradual shot transitions may be easier to detect, and may have a longer length, as compared the second category of gradual shot transitions.
  • the methods 400 and 500 may achieve improved results in detecting the second category of gradual transitions, including a better estimation of the gradual transition length.
  • the method 400 may include, for each frame (i) in the video (the target frame), determining a cardinality of a first bin of a normalized Y component color histogram of the target frame (i). More particularly, at 408 and with respect to the target frame (i), the method 400 may include determining if the luma value Y of more than 98% of the total frame pixels is in the first bin of the histogram. It will be appreciated that pixels in the first bin of the histogram, i.e. having luma values Y zero or close to zero, are black or nearly black. Such a determination may be represented by:
  • the method 400 may proceed to the next frame after target frame (i).
  • the method 400 may make a similar determination of the cardinality of the first bin of the normalized Y component color histogram of next frame (i+1). If H Y
  • the method 400 may set GTL to GTL + 1.
  • the method 400 may proceed to the next frame (i+2) and make a similar determination of the cardinality of the first bin of the normalized Y component color histogram of next frame (i+2).
  • the method 400 may continue to cycle through 420, 424, and 416 until, at 420 the cardinality of the first bin of the normalized Y component color histogram for the current frame is ⁇ 0.98. Upon this occurrence, at 428 the method 400 may determine if a frame histogram similarity between a temporally adjacent previous frame (i-1) and a temporally forward frame (i+ GT L + 1) is less than a false positive similarity threshold T3. In one example, T3 may be 2.9 or the same value as the second histogram similarity threshold value T2. This determination may be represented by:
  • the method 400 may include determining whether all of the frames in the video content have been examined. If all of the frames in the video content have been examined, then the method 400 may end. If all of the frames of the video content have not been examined, then at 436 the method 400 may proceed to the next frame comparison.
  • the method 400 may include determining that the target frame (i) is a gradual transition start frame, and setting the length of the gradual transition to GTL. At 432 the method 400 may then include determining whether all of the frames in the video content have been examined.
  • the method 500 may include initializing a count to 0.
  • the method 500 may include calculating a second set of frame histogram differences between the end frame (i+X) and the target frame (i) and between the end frame (i+X) and each of the consecutive following frames between the target frame and the end frame (i+X).
  • the second set of frame histogram differences may include: [0040]
  • the method 500 may include determining whether the frame histogram differences in the first set are increasing towards the end frame (i+5).
  • this determination may be expressed as follows: [0042] At 520 the method 500 may include determining whether the frame histogram differences in the second set are decreasing towards the end frame (i+5). In one example, this determination may be expressed as follows: [0043] At 524 the method 500 may include determining whether a false positive frame histogram difference between the target frame (i) and the end frame (i+5) is greater than a false positive difference threshold T 4 . In one example, this determination may be expressed as follows:
  • T 4 0.1. It will be appreciated that other suitable examples of T4 may also be utilized.
  • the method 500 may include setting the count to 0.
  • the method 500 may next include determining whether all of the frames in the video content have been examined. If all of the frames in the video content have been examined, then the method 500 may end. If all of the frames of the video content have not been examined, then at 536 the method 500 may proceed to the next frame.
  • the method 500 may include setting the count to equal count + 1.
  • the method 500 may determine whether all of the frames in the video content have been examined. If all of the frames in the video content have been examined, then the method 500 may end. If all of the frames of the video content have not been examined, then at 536 the method 500 may proceed to the next frame.
  • the method 500 may include determining if a blur value of a sample frame taken from any of the frames (i), (i+1), (i+2), (i+3), (i+4) and (i+5) is greater than a blur value threshold T 5 .
  • this determination may be used to identify certain false positives, such as a moving camera, that may be mistaken for a gradual transition. It will be appreciated that this determination may detect an amount of blur or sharpness at the edge frames (i) and (i+5).
  • frames in a moving camera sequence have sharper qualities, while frames in a gradual transition include greater blur. It will also be appreciated that any suitable blur value associated with a corresponding blurring technique may be utilized.
  • FIG.6 schematically illustrates a series of temporally adjacent shots that are temporally rearward and temporally forward of a target shot 604. More specifically, in this example six shots 608, 610, 612, 614, 616, and 618 that are temporally rearward of the target shot 604 are shown. Five shots 620, 622, 624, 626, and 628 that are temporally forward of the target shot 604 are shown.
  • method 700 for detecting a scene in video content is provided.
  • the following description of method 700 is provided with reference to the software and hardware components of the scene detection system 10 described above and shown in FIG. 1. It will be appreciated that method 700 may also be performed in other contexts using other suitable hardware and software components.
  • the method 700 may include identifying a plurality of shots in the video content 34. To identify the plurality of shots and as described above, the method 700 may include analyzing each frame of a plurality of frames in the video content 34 to identify the plurality of shots. In one example, the method 700 may include identifying a shot boundary by identifying a hard cut transition between a first frame and a second as described above with respect to the method 300. The method 700 may also include identifying one or more gradual transition start frames in the plurality of frames, indicating the beginning of a gradual transition shot boundary, as described above with respect to method 400. It will also be appreciated that the method 700 may utilize other methods and techniques for identifying a plurality of shots, identifying hard cut transitions and/or identifying gradual shot transitions.
  • the method 700 may include calculating color shot histograms for each shot in the video content 34.
  • the color shot histograms for each shot may be calculated as follows:
  • the frame histogram of one representative frame in the shot can be used as the color histogram of the shot.
  • the individual frame histograms for each frame in the video content 34 may have already been computed in identifying the plurality of shots via identifying hard cut transitions and/or gradual shot transitions as described above.
  • the same frame histograms may be utilized to calculate the shot histograms. Accordingly, by making use of this pre-calculated frame histogram data, the method 700 for detecting a scene in video content may utilize minimal additional computational resources.
  • the method 700 may include selecting a target shot in the plurality of shots.
  • the method 700 may determine if the target shot includes a gradual transition start frame. If the target shot does not include a gradual transition start frame, then at 716 the method 700 may include, beginning with the target shot, building a forward window including the target shot and containing a first number A of following shots that are temporally ahead of the target shot, and building a rearward window containing a second number B of previous shots that are temporally behind the target shot.
  • the forward window and rearward window may have a different number of shots such as 5, 6, 7, 8, 10, or other suitable number of shots. Additionally, the total number of shots in the forward window may be different than the total number of shots in the rearward window.
  • the number of shots in the forward and rearward window may be selected to be large enough to capture one scene boundary defining two separate scenes.
  • the number of shots in the forward and rearward window may also be selected to be small enough to avoid capturing two scene boundaries defining three separate scenes.
  • factors and/or characteristics of the type of video content 34 may be utilized to select the number of shots in the forward and rearward windows. For example, video content comprising a television soap opera program may utilize a different number of shots in the forward and rearward windows than video content comprising a television sitcom program.
  • the method 700 may include, for each of the shots in the forward window, determining a dissimilarity Dx,y between a selected shot and each of the other shots in the rearward window.
  • the selected shot may be shot 620 in the forward window.
  • Shot 620 may be compared to the other 6 shots in the rearward window. Each such comparison may yield a dissimilarity Dx,y between shot 620 and the other shot in the comparison.
  • shot 622 may be compared to the other 6 shots in the rearward window, and so forth until each of the 6 shots in the forward window has been compared to each of the 6 shots in the rearward window..
  • the method 700 may include determining a dissimilarity D x,y between a selected shot in the forward window and each of the other shots in the rearward window by calculating a Euclidean distance between the shot histograms of the selected shot and each of the other shots.
  • the Euclidean distance between shot histograms of two shots x and y may be calculated as follows:
  • the method 700 may include determining whether any of the determined dissimilarities Dx,y are less than a scene boundary threshold T4.
  • T 4 may be approximately 1, 1.1, 1.2, 1.3, or 1.75. It will be appreciated that other suitable values of T 4 may also be utilized.
  • the method 700 may include determining that no new scene begins at the target shot 620. At 736 the method 700 may then determine whether all of the shots in the video content 34 have been examined. If all of the shots in the video content 34 have been examined, then the method 700 may end. If all of the shots of the video content 34 have not been examined, then at 738 the method 700 may proceed to the next target shot comparison.
  • the method 700 may include determining that a new scene begins at the target shot 620. In one example, the method 700 may then insert metadata into the video content indicating that a new scene begins at the target shot 620.
  • the method 700 may then determine whether all of the shots in the video content 34 have been examined. If all of the shots in the video content 34 have been examined, then the method 700 may end. If all of the shots of the video content 34 have not been examined, then the method 700 may proceed to the next target shot comparison. In this manner, the method 700 may examine all of the shots in the video content 34 and efficiently determine one or more scene boundaries at one or more target shots, where a new scene begins at each scene boundary.
  • the method 700 may include determining that a new scene begins at the target shot 620. In one example, the method 700 may then insert metadata into the video content indicating that a new scene begins at the target shot 620. The method 700 may then proceed to 736 to determine whether all of the shots in the video content 34 have been examined, and continue as described above.
  • the methods and processes described above may be tied to a computing system of one or more computing devices.
  • such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.
  • API application-programming interface
  • FIG. 8 schematically shows a nonlimiting embodiment of a computing device 800 that may perform one or more of the above described methods and processes.
  • Computing device 800 is shown in simplified form. It is to be understood that virtually any computer architecture may be used without departing from the scope of this disclosure.
  • computing device 800 may take the form of a mainframe computer, server computer, desktop computer, laptop computer, tablet computer, home entertainment computer, gaming console, set-top box (e.g. cable television box, satellite television box), network computing device, mobile computing device, mobile communication device, etc.
  • set-top box e.g. cable television box, satellite television box
  • computing device 800 includes a logic subsystem 804, a data-holding subsystem 808, a display subsystem 812, and a communication subsystem 816.
  • Computing device 800 may optionally include other subsystems and components not shown in FIG. 8.
  • Computing device 800 may also optionally include other user input devices such as keyboards, mice, game controllers, and/or touch screens, for example.
  • the methods and processes described herein may be implemented as a computer application, computer service, computer API, computer library, and/or other computer program product in a computing system that includes one or more computers.
  • Logic subsystem 804 may include one or more physical devices configured to execute one or more instructions.
  • the logic subsystem may be configured to execute one or more instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs.
  • Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more devices, or otherwise arrive at a desired result.
  • the logic subsystem 804 may include one or more processors that are configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic subsystem may be single core or multicore, and the programs executed thereon may be configured for parallel or distributed processing. The logic subsystem may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and/or configured for coordinated processing. One or more aspects of the logic subsystem may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.
  • Data-holding subsystem 808 may include one or more physical, persistent devices configured to hold data and/or instructions executable by the logic subsystem 804 to implement the herein described methods and processes. When such methods and processes are implemented, the state of data-holding subsystem 808 may be transformed (e.g., to hold different data). Data-holding subsystem 808 may be configured to hold, for example, the video content 34.
  • Data-holding subsystem 808 may include removable media and/or built-in devices.
  • Data-holding subsystem 808 may include optical memory devices (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory devices (e.g., RAM, EPROM, EEPROM, etc.) and/or magnetic memory devices (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), among others.
  • Data-holding subsystem 808 may include devices with one or more of the following characteristics: volatile, nonvolatile, dynamic, static, read/write, read-only, random access, sequential access, location addressable, file addressable, and content addressable.
  • logic subsystem 804 and data- holding subsystem 808 may be integrated into one or more common devices, such as an application specific integrated circuit or a system on a chip.
  • FIG. 8 also shows an aspect of the data-holding subsystem 808 in the form of removable computer-readable storage media 820, which may be used to store and/or transfer data and/or instructions executable to implement the methods and processes described herein.
  • Removable computer-readable storage media 820 may take the form of CDs, DVDs, HD-DVDs, Blu-Ray Discs, EEPROMs, and/or floppy disks, among others.
  • data-holding subsystem 808 includes one or more physical, persistent devices.
  • aspects of the instructions described herein may be propagated in a transitory fashion by a pure signal (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for at least a finite duration.
  • a pure signal e.g., an electromagnetic signal, an optical signal, etc.
  • Display subsystem 812 may be used to present a visual representation of data held by data-holding subsystem 808.
  • the state of the display subsystem 812 may likewise be transformed to visually represent changes in the underlying data.
  • the display subsystem 812 may visually depict such scene boundaries in a visual representation of the video content.
  • the display subsystem 812 may depict representative frames from each scene in a browser bar panel, where a user can navigate to a particular scene by selecting the corresponding frame.
  • the display subsystem 812 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic subsystem 804 and/or data-holding subsystem 808 in a shared enclosure, or such display devices may be peripheral display devices.
  • Communication subsystem 816 may be configured to communicatively couple computing device 800 with one or more networks, such as network 42, and/or one or more other computing devices.
  • Communication subsystem 816 may include wired and/or wireless communication devices compatible with one or more different communication protocols.
  • the communication subsystem 816 may be configured for communication via a wireless telephone network, a wireless local area network, a wired local area network, a wireless wide area network, a wired wide area network, etc.
  • the communication subsystem may allow computing device 800 to send and/or receive messages to and/or from other devices via a network such as the Internet.
  • the computing device 800 may operate in a cloud-based service that delivers video content to a client display device.
  • the video content sent to the client display device may also include scene markers that denote one or more scenes that are detected using the above described systems and methods.
  • the above described systems and methods may be used in a computationally efficient manner to accurately identify scenes in video content, thereby addressing inefficiencies identified in the Background.
  • viewers of video content may be provided with an enjoyable user experience in browsing the content and locating desired portions of the content.
  • program may be used to describe an aspect of the scene detection system 10 that is implemented to perform one or more particular functions. In some cases, such a program may be instantiated via logic subsystem 804 executing instructions held by data-holding subsystem 808. It is to be understood that different programs may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same program may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc.
  • the term“program” is meant to encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

Abstract

L'invention concerne des systèmes et des procédés associés destinés à détecter une scène dans un contenu vidéo. Dans un exemple, un programme de détection de scène identifie une pluralité de plans et sélectionne un plan cible. Le programme crée une fenêtre avant comprenant le plan cible et ayant des plans provisoirement avant le plan cible, et une fenêtre arrière ayant des plans provisoirement après le plan cible. Pour chacun des plans dans la fenêtre avant, le programme établit une dissemblance entre un plan sélectionné et chacun des autres plans dans la fenêtre arrière. Si une ou plusieurs des dissemblances sont inférieures à un seuil de limite de scène, le programme établit que la scène ne commence pas au plan cible. Si aucune des dissemblances n'est inférieure au seuil de limite de scène, le programme établit que la scène commence au plan cible.
EP13763412.7A 2012-10-01 2013-09-06 Détection de scène vidéo Withdrawn EP2904541A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US13/633,021 US8818037B2 (en) 2012-10-01 2012-10-01 Video scene detection
PCT/US2013/058431 WO2014055203A1 (fr) 2012-10-01 2013-09-06 Détection de scène vidéo

Publications (1)

Publication Number Publication Date
EP2904541A1 true EP2904541A1 (fr) 2015-08-12

Family

ID=49213142

Family Applications (1)

Application Number Title Priority Date Filing Date
EP13763412.7A Withdrawn EP2904541A1 (fr) 2012-10-01 2013-09-06 Détection de scène vidéo

Country Status (6)

Country Link
US (1) US8818037B2 (fr)
EP (1) EP2904541A1 (fr)
JP (1) JP2015536094A (fr)
KR (1) KR20150067160A (fr)
CN (1) CN104854600A (fr)
WO (1) WO2014055203A1 (fr)

Families Citing this family (55)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013023063A1 (fr) 2011-08-09 2013-02-14 Path 36 Llc Édition multimédia numérique
US9760768B2 (en) 2014-03-04 2017-09-12 Gopro, Inc. Generation of video from spherical content using edit maps
US9792502B2 (en) 2014-07-23 2017-10-17 Gopro, Inc. Generating video summaries for a video using video summary templates
US9685194B2 (en) 2014-07-23 2017-06-20 Gopro, Inc. Voice-based video tagging
US9799376B2 (en) * 2014-09-17 2017-10-24 Xiaomi Inc. Method and device for video browsing based on keyframe
US9734870B2 (en) 2015-01-05 2017-08-15 Gopro, Inc. Media identifier generation for camera-captured media
US9514502B2 (en) * 2015-01-21 2016-12-06 Interra Systems Inc. Methods and systems for detecting shot boundaries for fingerprint generation of a video
US9679605B2 (en) 2015-01-29 2017-06-13 Gopro, Inc. Variable playback speed template for video editing application
US9955193B1 (en) 2015-02-27 2018-04-24 Google Llc Identifying transitions within media content items
US10186012B2 (en) 2015-05-20 2019-01-22 Gopro, Inc. Virtual lens simulation for video and photo cropping
US9894393B2 (en) 2015-08-31 2018-02-13 Gopro, Inc. Video encoding for reduced streaming latency
US10204273B2 (en) 2015-10-20 2019-02-12 Gopro, Inc. System and method of providing recommendations of moments of interest within video clips post capture
US9721611B2 (en) 2015-10-20 2017-08-01 Gopro, Inc. System and method of generating video from video clips based on moments of interest within the video clips
US10095696B1 (en) 2016-01-04 2018-10-09 Gopro, Inc. Systems and methods for generating recommendations of post-capture users to edit digital media content field
US10109319B2 (en) 2016-01-08 2018-10-23 Gopro, Inc. Digital media editing
US10769442B1 (en) * 2017-09-11 2020-09-08 Amazon Technologies, Inc. Scene change detection in image data
US9812175B2 (en) 2016-02-04 2017-11-07 Gopro, Inc. Systems and methods for annotating a video
US9972066B1 (en) 2016-03-16 2018-05-15 Gopro, Inc. Systems and methods for providing variable image projection for spherical visual content
US10402938B1 (en) 2016-03-31 2019-09-03 Gopro, Inc. Systems and methods for modifying image distortion (curvature) for viewing distance in post capture
US9794632B1 (en) 2016-04-07 2017-10-17 Gopro, Inc. Systems and methods for synchronization based on audio track changes in video editing
US9838731B1 (en) 2016-04-07 2017-12-05 Gopro, Inc. Systems and methods for audio track selection in video editing with audio mixing option
US9838730B1 (en) 2016-04-07 2017-12-05 Gopro, Inc. Systems and methods for audio track selection in video editing
US10250894B1 (en) 2016-06-15 2019-04-02 Gopro, Inc. Systems and methods for providing transcoded portions of a video
US9998769B1 (en) 2016-06-15 2018-06-12 Gopro, Inc. Systems and methods for transcoding media files
US9922682B1 (en) 2016-06-15 2018-03-20 Gopro, Inc. Systems and methods for organizing video files
US10045120B2 (en) 2016-06-20 2018-08-07 Gopro, Inc. Associating audio with three-dimensional objects in videos
US10185891B1 (en) 2016-07-08 2019-01-22 Gopro, Inc. Systems and methods for compact convolutional neural networks
US10469909B1 (en) 2016-07-14 2019-11-05 Gopro, Inc. Systems and methods for providing access to still images derived from a video
US10395119B1 (en) 2016-08-10 2019-08-27 Gopro, Inc. Systems and methods for determining activities performed during video capture
US9836853B1 (en) 2016-09-06 2017-12-05 Gopro, Inc. Three-dimensional convolutional neural networks for video highlight detection
US10282632B1 (en) 2016-09-21 2019-05-07 Gopro, Inc. Systems and methods for determining a sample frame order for analyzing a video
US10268898B1 (en) 2016-09-21 2019-04-23 Gopro, Inc. Systems and methods for determining a sample frame order for analyzing a video via segments
US10002641B1 (en) 2016-10-17 2018-06-19 Gopro, Inc. Systems and methods for determining highlight segment sets
US10284809B1 (en) 2016-11-07 2019-05-07 Gopro, Inc. Systems and methods for intelligently synchronizing events in visual content with musical features in audio content
US10262639B1 (en) 2016-11-08 2019-04-16 Gopro, Inc. Systems and methods for detecting musical features in audio content
US10482126B2 (en) * 2016-11-30 2019-11-19 Google Llc Determination of similarity between videos using shot duration correlation
US10534966B1 (en) 2017-02-02 2020-01-14 Gopro, Inc. Systems and methods for identifying activities and/or events represented in a video
US10339443B1 (en) 2017-02-24 2019-07-02 Gopro, Inc. Systems and methods for processing convolutional neural network operations using textures
US10127943B1 (en) 2017-03-02 2018-11-13 Gopro, Inc. Systems and methods for modifying videos based on music
US10185895B1 (en) 2017-03-23 2019-01-22 Gopro, Inc. Systems and methods for classifying activities captured within images
US10083718B1 (en) 2017-03-24 2018-09-25 Gopro, Inc. Systems and methods for editing videos based on motion
US10187690B1 (en) 2017-04-24 2019-01-22 Gopro, Inc. Systems and methods to detect and correlate user responses to media content
US10395122B1 (en) 2017-05-12 2019-08-27 Gopro, Inc. Systems and methods for identifying moments in videos
US10402698B1 (en) 2017-07-10 2019-09-03 Gopro, Inc. Systems and methods for identifying interesting moments within videos
US10614114B1 (en) 2017-07-10 2020-04-07 Gopro, Inc. Systems and methods for creating compilations based on hierarchical clustering
US10402656B1 (en) 2017-07-13 2019-09-03 Gopro, Inc. Systems and methods for accelerating video analysis
WO2019030776A1 (fr) * 2017-08-09 2019-02-14 Eswaran Kumar Dispositif robotique entraîné par intelligence artificielle (ia) capable de corréler des événements historiques avec des événements actuels pour l'indexation d'imagerie capturée dans un dispositif de type caméscope et la récupération
KR102413043B1 (ko) * 2017-09-22 2022-06-24 한국전자통신연구원 영상 컨텐츠의 샷 분할 방법 및 장치
KR102535411B1 (ko) 2017-11-16 2023-05-23 삼성전자주식회사 메트릭 학습 기반의 데이터 분류와 관련된 장치 및 그 방법
US11042988B2 (en) * 2018-06-28 2021-06-22 National Technology & Engineering Solutions Of Sandia, Llc Boundary detection evaluation
CN110826365B (zh) * 2018-08-09 2023-06-23 阿里巴巴集团控股有限公司 一种视频指纹生成方法和装置
KR101994592B1 (ko) * 2018-10-19 2019-06-28 인하대학교 산학협력단 비디오 콘텐츠의 메타데이터 자동 생성 방법 및 시스템
CN110619284B (zh) * 2019-08-28 2023-09-05 腾讯科技（深圳）有限公司 一种视频场景划分方法、装置、设备及介质
CN110717430A (zh) * 2019-09-27 2020-01-21 聚时科技(上海)有限公司 基于目标检测与rnn的长物体识别方法及识别系统
CN111031349B (zh) 2019-12-19 2021-12-17 三星电子（中国）研发中心 用于控制视频播放的方法及装置

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5821945A (en) 1995-02-03 1998-10-13 The Trustees Of Princeton University Method and apparatus for video browsing based on content and structure
JP3534368B2 (ja) * 1996-04-03 2004-06-07 株式会社東芝 動画像処理方法及び動画像処理装置
US6195458B1 (en) * 1997-07-29 2001-02-27 Eastman Kodak Company Method for content-based temporal segmentation of video
WO2001041451A1 (fr) * 1999-11-29 2001-06-07 Sony Corporation Procede et dispositif de traitement de signal audio-video
KR20010087552A (ko) * 2000-03-07 2001-09-21 구자홍 엠펙（ｍｐｅｇ）압축 비디오 환경에서 매크로 블록의시공간상의 분포를 이용한 디졸브/페이드 검출 방법
US6724933B1 (en) * 2000-07-28 2004-04-20 Microsoft Corporation Media segmentation system and related methods
US6810144B2 (en) * 2001-07-20 2004-10-26 Koninklijke Philips Electronics N.V. Methods of and system for detecting a cartoon in a video data stream
US7296231B2 (en) * 2001-08-09 2007-11-13 Eastman Kodak Company Video structuring by probabilistic merging of video segments
WO2007102511A1 (fr) 2006-03-09 2007-09-13 Pioneer Corporation dispositif de traitement d'image vidéo, méthode de traitement d'image vidéo, et programme de traitement d'image vidéo
DE102007063635A1 (de) 2007-03-22 2009-04-02 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Verfahren zur zeitlichen Segmentierung eines Videos in Videobildfolgen und zur Auswahl von Keyframes für das Auffinden von Bildinhalten unter Einbeziehung einer Subshot-Detektion
US20090290791A1 (en) 2008-05-20 2009-11-26 Holub Alex David Automatic tracking of people and bodies in video
US8364698B2 (en) * 2008-07-11 2013-01-29 Videosurf, Inc. Apparatus and software system for and method of performing a visual-relevance-rank subsequent search
US8364660B2 (en) * 2008-07-11 2013-01-29 Videosurf, Inc. Apparatus and software system for and method of performing a visual-relevance-rank subsequent search
KR102068790B1 (ko) 2009-07-16 2020-01-21 블루핀 랩스, 인코포레이티드 컴퓨터 실행 방법, 시스템 및 컴퓨터 판독 가능 매체
CN102333174A (zh) * 2011-09-02 2012-01-25 深圳市万兴软件有限公司 一种视频图像处理方法和装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2014055203A1 *

Also Published As

Publication number Publication date
CN104854600A (zh) 2015-08-19
JP2015536094A (ja) 2015-12-17
US8818037B2 (en) 2014-08-26
KR20150067160A (ko) 2015-06-17
WO2014055203A1 (fr) 2014-04-10
US20140093164A1 (en) 2014-04-03

Similar Documents

Publication Publication Date Title
US8818037B2 (en) Video scene detection
AU2018304058B2 (en) Identifying previously streamed portions of a media title to avoid repetitive playback
CN106663196B (zh) 用于识别主体的方法、系统和计算机可读存储介质
US7460689B1 (en) System and method of detecting, recognizing, and tracking moving targets
WO2014044158A1 (fr) Procédé et dispositif d'identification pour un objet cible dans une image
JP2005328105A (ja) 視覚的に代表するビデオサムネイルの生成
US9165180B2 (en) Illumination sensitive face recognition
WO2012037715A1 (fr) Identification d'une trame d'image clé à partir d'une séquence vidéo
CN110996183B (zh) 视频摘要的生成方法、装置、终端及存储介质
US10296539B2 (en) Image extraction system, image extraction method, image extraction program, and recording medium storing program
CN108780576B (zh) 使用对象边界框的视频片段中的重影去除的系统和方法
US20070061727A1 (en) Adaptive key frame extraction from video data
US9678991B2 (en) Apparatus and method for processing image
CN112752110B (zh) 视频呈现方法及装置、计算设备、存储介质
CN115720252A (zh) 用于在事件保留的情况下缩短视频的设备和方法
US11222429B2 (en) Object movement indication in a video
Bhaumik et al. Real-time storyboard generation in videos using a probability distribution based threshold
US11023738B2 (en) Information processing apparatus selecting highlight section from video, information processing method, and program
Srilakshmi et al. Shot boundary detection using structural similarity index
Hameed A novel framework of shot boundary detection for uncompressed videos
Seidl et al. A study of gradual transition detection in historic film material
JP5826560B2 (ja) ノイズ検知装置、再生装置、およびノイズ検知プログラム
CN114827473A (zh) 视频处理方法和装置

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20150401

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAX Request for extension of the european patent (deleted)
17Q First examination report despatched

Effective date: 20160726

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20161206