JP2017112448A - Video scene division device and video scene division program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video scene division device and a video scene division program capable of dividing a video content appropriately into scenes.SOLUTION: A video scene division device 1 includes a shot boundary detector 11 for detecting a shot boundary, i.e., a discontinuous point of the videos, based on the difference of image data between the frames of a video, a still image extraction unit 12 for extracting a plurality of still images for each shot divided by the shot boundary, a subject recognition unit 13 for recognizing the subject of each still image, a histogram generation unit 14 for generating a histogram indicating the frequency of appearance of a subject for respective shots, and a scene boundary determination unit 15 for determining a scene boundary, i.e., a discontinuous point of histogram, based on the similarity of the histogram.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an apparatus and a program for dividing video content into scenes.

  Conventionally, an application such as an image search has been realized by using a statistical technique and recognizing a subject in an image by software and adding metadata.

  By the way, video contents such as television programs are often expressed in a hierarchical structure as shown in FIG. Specifically, it is called a frame, a shot, a scene, and content (program video) from a smaller unit.

A shot has a short point of several seconds to several tens of seconds with a camera switching point at the time of shooting as a boundary. In a general program of about 1 hour, the number of shots increases to about 100 to 1000, so it is not easy to grasp the structure of the entire program from a sequence of shots. For example, in the case of a program that expresses the meaning of a video by combining a plurality of shots, such as a drama, the meaning of the video is often not understood only by a single shot. For this reason, a user of video search often feels that a shot is too fine as a unit of video division.
In addition, when presenting a search result in video search, a function of structurally showing the contents and reproducing a necessary video in a semantic unit is desired.

  In such a situation, for example, paying attention to the continuity of the color, pattern, or sound, a method of dividing the video using the point where the continuity is interrupted as a boundary has been proposed (see, for example, Patent Documents 1 and 2). .

Japanese Patent Laid-Open No. 2004-280669 JP 2008-5167 A

By the way, a scene is a section composed of a plurality of shots given meaning by a video editor, and often has the same place or time expressed in content. For this reason, the section indicated by the continuity of the video or audio signal used in the conventional method and the meaning section considered by the person are largely different, and are often not divided at the boundary desired by the user.
As described above, it is difficult to automatically detect a scene boundary that is a boundary of semantic video content.

  It is an object of the present invention to provide a video scene division apparatus and a video scene division program that can appropriately divide video content into scenes.

  A video scene division device according to the present invention includes a shot boundary detection unit that detects a shot boundary that is a discontinuous point of the video based on a difference in image data between video frames, and a shot divided by the shot boundary. A still image extraction unit that extracts a plurality of still images every time, a subject recognition unit that recognizes a subject for each still image, a histogram generation unit that generates a histogram indicating the appearance frequency of the subject for each of the shots, A scene boundary determining unit that determines a scene boundary that is a discontinuous point of the histogram based on the similarity of the histogram.

  The subject recognition unit may identify the plurality of subjects by clustering based on a predetermined feature amount included in the still image.

  The histogram generation unit normalizes a frequency obtained by normalizing the number of the still images in which the subject is recognized in the shot period, or a time in the shot period corresponding to the still image in which the subject is recognized. The normalized frequency distribution may be generated as the histogram.

  The histogram generation unit may generate the histogram for the group of subjects.

  The scene boundary determination unit may determine the scene boundary based on a partial histogram including only the predetermined number of subjects whose frequency in the histogram is higher.

  The scene boundary determination unit may adjust the determination result of the scene boundary based on an appearance pattern stored in advance regarding the time axis and position of the subject in the scene.

  The video scene division program according to the present invention causes a computer to function as the video scene division device.

  According to the present invention, video content can be appropriately divided into scenes.

It is a block diagram which shows the function structure of the video scene division | segmentation apparatus which concerns on embodiment. It is a figure which shows the specific example of the detection of the shot boundary which concerns on embodiment, and the extraction of a still image. It is the schematic which shows the procedure of the scene division | segmentation concerning embodiment. It is a flowchart which shows the process by the control part which concerns on embodiment. It is a figure which shows the example of the weighting to the object based on the program genre which concerns on embodiment. It is a figure which shows the example of the scene division | segmentation based on the knowledge of the video editing which concerns on embodiment. It is a figure which shows the image of the hierarchical unit of video content.

Hereinafter, an example of an embodiment of the present invention will be described.
FIG. 1 is a block diagram showing a functional configuration of a video scene dividing device 1 according to the present embodiment.

  The video scene dividing apparatus 1 includes a control unit 10 including a shot boundary detection unit 11, a still image extraction unit 12, a subject recognition unit 13, a histogram generation unit 14, and a scene boundary determination unit 15, and a storage unit 20. An information processing apparatus (computer) provided.

  The control unit 10 is a part that controls the entire video scene dividing device 1, and appropriately reads and executes various programs stored in the storage unit 20, thereby cooperating with the hardware described above. Various functions are realized. The control unit 10 may be a CPU (Central Processing Unit).

  The storage unit 20 stores various programs for causing the hardware group to function as the video scene dividing device 1, programs for causing the control unit 10 to execute various functions of the present embodiment, various data, and the like. The data stored in the storage unit 20 includes video data to be processed and scene division data after processing, as well as later-described determination criteria data related to scene division.

The shot boundary detection unit 11 detects a shot boundary, which is a discontinuous point at which continuously recorded videos are interrupted, based on the difference in image data between frames constituting the video data.
Specifically, the shot boundary detection unit 11 obtains a difference in image data from the immediately preceding frame for each frame. When the difference value exceeds the first threshold value, the shot boundary detection unit 11 detects a frame between frames before and after the first threshold value as a shot boundary. Note that the difference between the image data is appropriately set as a value for evaluating the degree of change in the image between frames, such as the total or average change amount of the pixel values included in the image data, or the change amount of the luminance histogram.

The still image extraction unit 12 extracts one or more still images for each shot divided by the shot boundary.
Specifically, the still image extraction unit 12 extracts, as a still image, a frame that exceeds the second threshold every time the accumulated difference of the image data exceeds the second threshold.

FIG. 2 is a diagram showing a specific example of shot boundary detection and still image extraction according to the present embodiment.
The shot boundary detection unit 11 compares the temporally continuous frames constituting the video content in order, and calculates the difference between the image data.

  When the calculated difference value exceeds the threshold (first threshold) X of the shot boundary, the shot boundary detection unit 11 detects the interval between the previous frame as a shot boundary.

  The still image extraction unit 12 extracts the frames before and after the shot boundary, that is, the first and last frames P1 and P2 in the shot as still images (thumbnail) representing the shot.

The still image extraction unit 12 accumulates, for example, the difference value between frames calculated by the shot boundary detection unit 11 for each shot, and the accumulated value is a threshold value (second threshold value) Y1, Y2, .., Each time it exceeds Y3, Y4,..., The frames P3, P4, P5, P6,.
The still image extraction unit 12 may extract still images at regular time intervals (for example, every 10 frames, every second, etc.).

  At this time, the still image extraction unit 12 stores the extracted still image in the storage unit 20 together with the shot number given in order from the top of the video and the time information for identifying the frame.

The subject recognition unit 13 recognizes a subject in the image for each extracted still image.
Specifically, the subject recognition unit 13 learns a plurality of specific subjects that are assumed in advance, and determines whether or not these subjects are included in the still image.

The subject to be learned in advance is appropriately selected depending on the content or field of the target video content. For example, in the case of a drama, a subject that frequently appears in video content to be processed is learned in advance, such as the face of a character.
At this time, it is desirable to learn a large number of subjects so that a wide range of subjects appearing in the video content can be recognized, but when the recognizable subjects are limited, it is assumed that the frequency of appearance in the program is high. The subject to be selected is selected.

The subject recognition unit 13 uses a technique that enables individual identification based on feature amounts in image data, such as the Bag-of-visual-words method, and the like (persons A, B, ...) is specified.
Alternatively, the subject recognition unit 13 performs identification based on temporary labels (A cluster, B cluster,...) Without specifying each of the plurality of subjects by clustering based on a predetermined feature amount included in the still image. Also good.

The histogram generation unit 14 counts the number of appearances for each subject, that is, the number of still images including the subject for each shot, and generates a histogram indicating the appearance frequency of the subject.
At this time, the histogram generation unit 14 expresses the reliability of appearance of each subject within the shot period, for example, as a frequency distribution normalized as in (1) or (2) below. As a result, the maximum frequency in the histogram for each shot is set to a constant value.

(1) The histogram generation unit 14 generates a frequency distribution obtained by normalizing the number of still images in which a subject is recognized during a shot period as a histogram.
(2) The histogram generation unit 14 generates a frequency distribution obtained by normalizing the time within the shot period corresponding to the still image in which the subject is recognized as a histogram.

  The histogram generation unit 14 may generate a histogram for a group of a plurality of subjects according to the type of video content. For example, in a drama, the characters are treated as a group such as a family or a circle, or in a travel program where the person is not the center of the subject, the people are grouped together and the “person” is the same as other subjects such as mountains and the sea. It can be treated as a level.

The scene boundary determination unit 15 determines a scene boundary that is a discontinuous point of a series of histograms based on the similarity of the generated histogram.
Specifically, the scene boundary determination unit 15 obtains the similarity between histograms generated for each shot in order to determine the continuity of subjects appearing in the scene. For example, when the similarity is equal to or lower than a certain level, It is determined that the interrupted scene has been switched.
Note that a technique such as histogram intersection may be used to determine the similarity between histograms. In addition, for example, a simple method may be used, such as determining a point at which a certain number of subjects at the same time have changed at the same time or at a certain rate or more as a scene division point.

FIG. 3 is a schematic diagram showing a procedure of scene division according to the present embodiment.
The video scene dividing device 1 divides the input video into shots 1 to 4 and extracts a plurality of still images (thumbnail images) from each shot.

Subsequently, the video scene dividing device 1 recognizes the subjects A to D from each still image, and generates a normalized histogram based on the number of appearances.
The video scene segmentation device 1 calculates the degree of similarity before and after the generated histogram in time series, and determines the boundary between the shot 3 and the shot 4 whose similarity is less than the threshold as the scene boundary.

Here, if determination of continuity is performed on a single subject, excessive division is likely to occur. Also, if a subject with a small number of recognitions (appearance count) is selected, it is often affected by erroneous detection by subject recognition processing.
Therefore, the scene boundary determination unit 15 determines a scene boundary based on a partial histogram including only a predetermined number of subjects whose frequencies in the histogram are higher. For example, a method of calculating similarity by paying attention to a predetermined number (for example, 3) of subjects appearing in the histogram or a certain ratio (for example, 50%) is adopted.

FIG. 4 is a flowchart showing processing by the control unit 10 according to the present embodiment.
In step S1, the shot boundary detection unit 11 detects a shot boundary from a series of frames included in the video content, and divides the video into a plurality of shots.

  In step S2, the still image extraction unit 12 extracts a plurality of still images (thumbnails) for each shot divided in step S1.

  In step S3, the subject recognition unit 13 recognizes the subject for each still image extracted in step S2.

  In step S4, the histogram generation unit 14 generates a histogram representing the appearance frequency of the subject recognized in step S3.

  In step S5, the scene boundary determination unit 15 sequentially selects the histogram generated in step S4 in time series.

  In step S6, the scene boundary determination unit 15 calculates the similarity between the histogram selected in step S5 and the histogram selected immediately before.

  In step S7, the scene boundary determination unit 15 determines whether or not the similarity calculated in step S6 is smaller than a predetermined threshold. If this determination is YES, the process proceeds to step S8, and if the determination is NO, the process proceeds to step S9.

  In step S8, the scene boundary determination unit 15 determines the shot boundary determined to have a low similarity in step S7 as a scene boundary.

  In step S9, the scene boundary determination unit 15 determines whether the histogram has been selected to the end and the video has ended. If this determination is YES, the process ends. If the determination is NO, the process returns to step S5.

<Modification>
As the continuity determination criterion based on the histogram similarity based on the above-described statistics, for example, the following determination criterion (A) or (B) may be used based on the theory of video content production / editing. Alternatively, the scene boundary determined by the above-described method may be adjusted according to these determination criteria.
Although a scene is a manually created structure, there is a certain theory for editing a program video. By using knowledge about the program video, errors in scene detection are reduced.

(A) Criteria for determining continuity considering program genre The histogram generation unit 14 and the scene boundary determination unit 15 may adjust the weighting of the subject based on the program genre.
For example, when a person is the center of a subject, such as a drama, and an individual is an important subject, and when a change between a person and another subject becomes a factor that divides a scene rather than a specific individual, such as a travel program Then, the weight of a person and a natural object is changed, and a criterion for determining continuity is provided separately.

  Specifically, in a drama or the like, after recognizing an individual, characters are handled as a set (AB group, BCD group, etc.), and a histogram that is a distribution for each set is used. On the other hand, in video content in which a person such as a journey is not the center of a subject, all the people are grouped together, and the boundary with other subjects (mountains, seas, etc.) is more weighted.

FIG. 5 is a diagram showing an example of weighting on the subject based on the program genre according to the present embodiment.
In the case of a drama (a), a boundary between a shot in which persons A and B appear and a shot in which persons C and D appear is determined as a scene boundary.
When the persons C and D are in the same group, the shot in which the persons C and D appear and the shot in which only the person D appears are determined as the same scene because the subjects are in the same group.

  Note that the scene boundary determination unit 15 may determine that a shot in which all of the subjects belonging to the group appear as the same scene, or a shot in which one or more or one of them appears as the same scene. You may judge. These determination criteria may be appropriately set depending on the program genre, the type of group, and the like.

In the case of travel (b), a transition from a shot of a landscape to a shot of a person E is determined as a scene boundary. Similarly, a transition from a shot of a person G as a subject to a shot of an animal as a subject is determined as a scene boundary.
In addition, a plurality of shots in which persons E, F, or G appear are determined to have the same group of subjects and are determined to be the same scene.

(B) Criteria for determining continuity using knowledge of video editing The scene boundary determination unit 15 adjusts the determination result of the scene boundary based on the appearance pattern stored in advance with respect to the time axis and position of the subject in the scene. Also good.
For example, in a program such as a drama where people's faces are shown alternately, the scene may be shredded when attention is paid to the individual's face. Therefore, by incorporating the knowledge of a conversation scene called a montage technique that alternately reflects two people, which is a general editing technique, a series of shots in which faces alternately become subjects are appropriately used as one conversation scene. Determined. Thereby, the excessive division | segmentation of a scene is suppressed.

FIG. 6 is a diagram illustrating an example of scene division based on the knowledge of video editing according to the present embodiment.
In the case of only a statistical method based on histogram similarity (a), a shot in which person A appears and a shot in which person B appears are determined as scene boundaries, and are divided into a plurality of scenes 1 to 4 in detail. Yes.

  When the feature that the characters in the conversation scene appear alternately is used as the criterion (b), a plurality of shots in which either the characters A or B appear alternately are determined as one scene, and the characters A and B appear. A scene boundary is determined between shots.

  Here, when the subject recognition unit 13 recognizes a person's face, the face position (for example, the center position) is preferably stored together with the shot number, frame time, person ID, and the like. Thereby, for example, the scene boundary determination unit 15 has the feature of the position where the person A and the person B are shown in the above-described dialog scene (FIG. 6) (the person A is on the screen left side, the person B is on the screen right side, etc.) Can be divided into scenes with high accuracy.

As described above, according to the present embodiment, the video scene dividing device 1 causes the appearance of a subject between shots based on the similarity of histograms indicating the appearance frequency of a still image subject extracted for each shot. Continuity is determined, and scene switching is determined at discontinuous points.
Therefore, the video scene dividing apparatus 1 can appropriately divide the video content into scenes that are breaks in semantic sections in a form closer to the semantic content using the recognition result of the subject representing the video content.
As a result, at the time of video search or reuse, display and playback can be performed in scene units, which are semantic units closer to human senses. In addition, when presenting video search results in units of content, it is easier to grasp the outline of the entire content by organizing and presenting it for each scene that is a semantic section. The next use becomes easy.

  Further, since the video scene dividing device 1 generates a histogram for each subject from a plurality of still images extracted from the shot period, noise due to erroneous recognition or oversight in subject recognition can be reduced, and the accuracy of scene division is improved. To do.

The video scene dividing device 1 detects a shot boundary when a difference between frames exceeds a first threshold value, and a still image (thumbnail) when a cumulative value of differences between frames exceeds a second threshold value for each shot. ).
Therefore, the video scene dividing device 1 can efficiently divide the video into shots based on simple rules and extract still images that characterize the contents of each shot.

  The video scene dividing device 1 can identify a plurality of subjects by clustering based on a predetermined feature amount included in a still image. Thereby, the video scene dividing device 1 can identify each unknown subject without learning in advance in order to identify the subject.

The video scene dividing device 1 normalized the frequency distribution obtained by normalizing the number of still images in which the subject is recognized in the shot period, or the time in the shot period corresponding to the still image in which the subject is recognized. A frequency distribution is generated as a histogram.
By these normalization methods, the video scene dividing device 1 expresses the reliability with which each subject appears within the shot period. Thereby, the maximum frequency in the histogram for each shot is set to a constant value, and comparison of histograms between shots can be performed more accurately.

  The video scene segmentation apparatus 1 can generate a histogram for a group of subjects so that a more appropriate histogram can be used according to the program genre, an appropriate scene boundary can be determined, and scene boundary overdivision can be performed. Is reduced.

  The video scene segmentation apparatus 1 determines a scene boundary by comparing partial histograms made up of only subjects with a higher frequency in the histogram. As a result, it is possible to reduce scenes with low appearance frequency or noise due to misrecognition and to divide the scene with high accuracy.

  The video scene dividing apparatus 1 can determine a scene boundary based on a judgment criterion using knowledge about editing of a program video based on an appearance pattern stored in advance with respect to a time axis and a position of a subject in a scene. This reduces the influence of subject recognition errors and excessive division of scene boundaries.

  As mentioned above, although embodiment of this invention was described, this invention is not restricted to embodiment mentioned above. Further, the effects described in the present embodiment are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the present embodiment.

  In the present embodiment, the configuration and operation of the video scene dividing device has been described. However, the present invention is not limited to this, and each of the components may be configured as a method or program for dividing a video into scenes. .

  Further, the present invention may be realized by recording a program for realizing the function of the video scene dividing device on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium. Good.

  The “computer system” here includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a hard disk built in the computer system.

  Furthermore, “computer-readable recording medium” means that a program is dynamically held for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It is also possible to include one that holds a program for a certain time, such as a volatile memory inside a computer system that becomes a server or client in that case. Further, the program may be for realizing a part of the above-described functions, and may be capable of realizing the above-described functions in combination with a program already recorded in the computer system. .

DESCRIPTION OF SYMBOLS 1 Image | video scene division | segmentation apparatus 10 Control part 11 Shot boundary detection part 12 Still image extraction part 13 Subject recognition part 14 Histogram generation part 15 Scene boundary determination part 20 Storage part

Claims (7)

A shot boundary detection unit that detects a shot boundary that is a discontinuous point of the video based on a difference in image data between frames of the video;
A still image extraction unit that extracts a plurality of still images for each shot divided by the shot boundary;
A subject recognition unit for recognizing the subject for each still image;
For each of the shots, a histogram generation unit that generates a histogram indicating the appearance frequency of the subject;
And a scene boundary determining unit that determines a scene boundary that is a discontinuous point of the histogram based on the similarity of the histogram.   The video scene segmentation device according to claim 1, wherein the subject recognition unit identifies a plurality of the subjects by clustering based on a predetermined feature amount included in the still image.   The histogram generation unit normalizes a frequency obtained by normalizing the number of the still images in which the subject is recognized in the shot period, or a time in the shot period corresponding to the still image in which the subject is recognized. The video scene dividing device according to claim 1, wherein the distribution of the converted frequency is generated as the histogram.   The video scene dividing device according to any one of claims 1 to 3, wherein the histogram generation unit generates the histogram for the group of subjects.   The video scene dividing device according to any one of claims 1 to 4, wherein the scene boundary determination unit determines the scene boundary based on a partial histogram including only the predetermined number of subjects having higher frequencies in the histogram. .   The video according to claim 1, wherein the scene boundary determination unit adjusts the determination result of the scene boundary based on an appearance pattern stored in advance with respect to a time axis and a position of the subject in the scene. Scene division device.   A video scene division program for causing a computer to function as the video scene division device according to any one of claims 1 to 6.

Images