JP2008042270A - Highlight extracting apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To extract important scenes of a video content picked up by a user by himself/herself. <P>SOLUTION: A light emission detector 12 detects a relative time in an input video image of a light emitter of a flash of a camera which is photographed in the input video image as an emission time of the flash. A highlight selector 14 selects the predetermined number of light emission times as a time in which a highlight exists from each light emission time detected by the light emission detector 12, on the basis of the number of light emission times included within a predetermined time before the light emission time, and selects a segment including the selected time as a highlight segment. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention automatically selects a part of video content as a highlight so that the video content can be viewed in a short time or the content of the video content can be easily grasped. Relates to the device.

  In recent years, with the widespread use of large-capacity digital storage such as DVDs (Digital Versatile Discs), hard disk drives (HDDs), and semiconductor memories, the amount of video content stored and managed at home has increased at an accelerated pace. ing. These storages can be accessed at random, and playback can be instantly started from a designated location, which greatly improves convenience over conventional tape media.

  Therefore, it is possible to easily realize a viewing method in which only a portion to be viewed is reproduced from a series of videos over a long time and the rest is skipped. On the other hand, if the user does not know where the scene that the user wants to see is recorded, the random access performance of such storage is not a merit.

  Therefore, Patent Document 1 discloses a technique for extracting points of interest in advance by analyzing audio information and image information with respect to content that has not yet been viewed by the user. In the technique disclosed in Patent Document 1, a numerical value set in advance in an index that defines an event that has occurred in a video stream is stored as a rule file, and the corresponding value is determined from the rule file based on the index in the input video information. Based on the index and the numerical value, generates the numerical video information that expresses the content of the video information by changing the numerical value, and uses the numerical value change of the generated numerical video information to convert the digest video from the video information. Extraction realizes digest generation.

As a result, the user can quickly find the scene he / she wants to watch for the broadcast program or news program of the broadcasted sports, and can watch a long program in a short time.
Japanese Patent No. 3176893

  On the other hand, content (hereinafter referred to as private content) recorded by a user with a video camera or the like recorded in a randomly accessible large-capacity digital storage such as an HDD is different from a broadcast program, Video and audio quality, shooting environment, etc. cannot be assumed in advance. In addition, there is a high possibility that useless images and miss shots are included, and the story is often lacking. In particular, when shooting an event such as a wedding ceremony or a presentation, one cut is often long, and when a photographed image is viewed later, only the photographing time is long and the image lacks excitement.

  For this reason, when viewing the recorded private content, it is difficult for the user to quickly reach the target important scene, and the advantage of the randomly accessible medium cannot be utilized. Therefore, development of a technique for enabling extraction of important scenes of private content has been desired.

  The present invention has been made in view of the above, and an object of the present invention is to provide a highlight extraction apparatus that can extract an important scene of private content.

  In order to achieve the above object, a highlight extracting apparatus according to the present invention extracts, as a highlight, a part that becomes an important scene in the video based on light emission of a camera flash reflected in the video. A light emission detecting means for detecting the presence or absence of flash emission for each frame of the input video, and detecting the relative time in the input video of the frame detected as having flash emission as the flash emission time; Storage means for storing the light emission time detected by the detection means, and the number of light emission times included in a predetermined time before the light emission time for each of the light emission times stored in the storage means. Within the predetermined time before the light emission time when the light emission time is higher than the score of the light emission time. If included, after canceling the score of the light emission time, select a predetermined number of the light emission times from the highest score as the time when the highlight exists, and the section including the selected time in the input video Highlight selection means for selecting as a highlight section is provided.

  Further, the highlight extracting apparatus of the present invention is a highlight extracting apparatus that extracts, as a highlight, a portion that becomes an important scene in the video based on light emission of a camera flash reflected in the video. A luminance histogram is generated for each frame of the video, and a correlation between the luminance histogram of the frame and the luminance histogram of the first predetermined number of frames before the frame is calculated as a first correlation, and the luminance histogram of the frame and the frame A correlation with a luminance histogram of a second predetermined number of frames larger than the first predetermined number from the frame is calculated as a second correlation, and the first correlation is smaller than a predetermined first threshold; If the second correlation is greater than a predetermined second threshold, the relative time in the input video of the frame is determined as flash emission. For each of the light emission times stored in the storage means, the light emission detection means for performing the detection operation for each frame for each frame, the storage means for storing the light emission time detected by the light emission detection means, and before the light emission time. The number of light emission times included in the predetermined time is calculated as the score of each light emission time, and the light emission time is included in the predetermined time before the light emission time having a score higher than the score of the light emission time. When the score of the light emission time is canceled, a predetermined number of the light emission times are selected as the time when the highlight exists in order from the highest score, and the section including the selected time is highlighted in the input video. Highlight selection means for selecting as a section is provided.

  The highlight extraction device of the present invention detects the relative time in the input image of the flash emission point of the camera's flash reflected in the input image as the flash emission time, and from each detected emission time, a predetermined time before the emission time is detected. Since a predetermined number of light emission times is selected as a time when a highlight exists and a section including the selected time is selected as a highlight period based on the number of light emission times included in the Can be extracted as highlights.

  Hereinafter, the best mode for carrying out the highlight extraction apparatus of the present invention will be described with reference to the drawings. In the present invention, it is assumed that the scene in which the flash of the camera is frequently emitted is an important scene in the private content.

  FIG. 1 is a block diagram showing a configuration of a highlight extracting apparatus according to an embodiment of the present invention. As shown in FIG. 1, the highlight extraction apparatus 1 according to the present embodiment includes a video storage unit 11 for storing an input video, and a flash time emission relative time in the input video of a flash emission point reflected in the input video. A light emission detection unit 12 that detects the time, a storage unit 13 that stores the light emission time detected by the light emission detection unit 12, and a highlight section that selects a highlight section in the input video based on the light emission time stored in the storage unit 13. And a light selection unit 14.

  The light emission detection unit 12 includes a buffer 12 a made up of a ring buffer having four elements, and an arithmetic processing unit 12 b that performs calculation and control of the light emission detection unit 12. The highlight selection unit 14 includes a memory 14 a that stores various types of information, and a calculation processing unit 14 b that performs calculation and control of the highlight selection unit 14.

  Next, the operation of the highlight extraction apparatus according to this embodiment will be described.

  The video storage unit 11 stores externally input video. The input video is a moving image composed of a series of still images. In the case of a video to which a compression encoding technique such as MPEG is applied, the decoded video is received as an input video.

  The light emission detection unit 12 receives an input video from the video storage unit 11, analyzes a still image constituting the input video, and detects a flash emission time. Hereinafter, a procedure for detecting the flash emission time in the light emission detection unit 12 will be described with reference to FIG. FIG. 2 is a flowchart showing a procedure for detecting the flash emission time in the light emission detector 12.

  First, when a new frame is received from the video storage unit 11 in step S110, in step S120, the arithmetic processing unit 12b of the light emission detection unit 12 creates a luminance histogram for the received frame. The luminance corresponds to the value of Y in the YUV color system. The RGB color system can be converted by the following (Equation 1).

Y = 0.299R + 0.587G + 0.114B (Formula 1)
The histogram is created by quantizing the brightness with an arbitrary width, calculating which class the brightness of each pixel of the input video belongs to, and adding the frequency of the corresponding class. If the image is sufficiently large, the image may be subsampled at an appropriate interval in order to shorten the processing time. An example of a luminance histogram is shown in FIG. In the luminance histogram shown in FIG. 3, the x-axis includes 64 classes, and the y-axis is the number of pixels (pixels) in each class.

  Next, in step S130, the arithmetic processing unit 12b stores the created histogram in the buffer 12a. The buffer 12a is a ring buffer having four elements, and a histogram can be stored in each element. The latest histogram is stored by overwriting the oldest buffer in the ring buffer.

In step S140, the arithmetic processing unit 12b calculates a correlation between the latest histogram in the buffer 12a and the histogram one frame before. The correlation is obtained by the following (Formula 2).

  Here, R is the correlation, N is the total number of classes in the histogram, and a (I) and b (I) are the frequencies of class I in each histogram.

Next, in step S150, the processing unit 12b, the correlation calculated in step S140, it is determined whether the first threshold value _{A 1} is less than. If the correlation is the first threshold value A ₁ or more (step S150: NO), the two frames can be determined to be consecutive, iconic. If the camera flash light yelling reflected in the video, the luminance distribution of the video changes abruptly, the correlation falls below a first threshold value A _1. Further, even when the back flush disappear normal image, the correlation falls below a first threshold value A _1.

Therefore, when the correlation is equal to or higher than the _first threshold A1 (step S150: NO), it is determined that no flash has been detected, and the process proceeds to step S190. If the correlation is the first threshold value _{A 1} is smaller than (Step S150: YES), since there is a possibility of flash, the process proceeds to step S160. Note that the first threshold A ₁ is determined to be optimal experimentally.

In step S160, the arithmetic processing unit 12b calculates the correlation between the latest histogram and the histogram three frames before. Then, in step S170, the arithmetic processing unit 12b compares the threshold value _{A 2} of the correlation and the second obtained in step S160.

  When the flash is reflected in the video, it is conceivable that the luminance distribution changes greatly during one to two frames and then returns to the previous luminance distribution. Therefore, in a frame having a small correlation with the luminance histogram of the previous frame and a large correlation with the luminance histogram of the previous three frames, the flash may be reflected in the immediately preceding frame.

Therefore, if the calculated correlation is greater than the second threshold value _{A 2} at the step S160 (step S170: YES), it is determined that the flash was not reflected on the previous frame, in step S180, the arithmetic processing unit 12b, The time is detected as the light emission time and stored in the storage unit 13. If the correlation calculated in step S160 is the second threshold value _{A 2} or less (step S170: NO), the flash as could be detected, the process proceeds to step S190.

Here, the time uses a common time code throughout the apparatus. As the time code, seconds, milliseconds, the number of frames, etc. can be considered. The second threshold value A _2, like the first threshold value A _1, determined experimentally optimum value.

  In step S190, the arithmetic processing unit 12b determines whether or not the processing for all the frames of the input video has been completed. If the processing has been completed (step S190: YES), the light emission time detection processing ends. If not completed (step S190: NO), the process returns to step S110, and the subsequent processing is repeated.

  The storage unit 13 holds the light emission time detected by the light emission detection unit 12 in step S180 of the flowchart shown in FIG. The time is stored as an array using the above time code. As the storage unit 13, a semiconductor memory or an HDD may be used. Moreover, a part of the same address space as the video storage unit 11 may be used, or another address space may be prepared.

  The highlight selection unit 14 selects a highlight section based on the light emission time held by the storage unit 13. Hereinafter, a procedure for selecting a highlight section in the highlight selection unit 14 will be described with reference to FIG. FIG. 4 is a flowchart showing a procedure for selecting a highlight section in the highlight selection unit 14.

  The light emission times detected by the light emission detection unit 12 are stored in the array list [] in the storage unit 13. First, in step S210, the arithmetic processing unit 14b initializes counter variables i and j. The initial value of the counter variable i is set to the minimum i where list [i] ≧ n. Here, n is an arbitrary natural number or real number, and represents milliseconds, seconds, or the number of frames according to the time code used. Hereinafter, for the sake of explanation, it is assumed that the time code is seconds. The initial value of the counter variable j is 0.

  Next, in step S220, the arithmetic processing unit 14b gives a score to each light emission time when the flash is detected, and stores the score as an array in the memory 14a. The memory 14a stores the score in the array point [], and stores the score corresponding to the time of list [i] in the point [i]. Here, the score is the number of times the flash is detected for n seconds before the emission time giving the score.

  Subsequently, in step S230, the arithmetic processing unit 14b adds 1 to the counter variable i. In step S240, the arithmetic processing unit 14b determines whether the counter variable i has reached the last element of the array list []. If the last element has not been reached (step S240: YES), the process proceeds to step S250. If the last element has been reached (step S240: NO), the process proceeds to step S260.

  In step S250, the arithmetic processing unit 14b updates the counter variable j. The counter variable j is the smallest j where list [j] is greater than or equal to list [i] −n. Thereafter, the process returns to step S220, and the subsequent processing is repeated.

  When scores are obtained at all the light emission times at which flashes are detected, in step S260, the arithmetic processing unit 14b sorts the array point [] storing the scores in descending order. In addition, the arithmetic processing unit 14b generates an array link [] that indicates what time each element of the array point [] is, and stores this in the memory 14a. For example, when the score at time list [p] is point [q], p = link [q].

  Next, in step S270, the arithmetic processing unit 14b integrates a plurality of overlapping light emission times. Times list [link [i]] and list [link [link [j] with time points [link [j]] with a certain score point [j] have higher score points [i] (ie j> i) than point [j] i]] − n, point [j] = 0.

  Next, in step S280, the arithmetic processing unit 14b acquires time list [link [k]] (0 ≦ k <m) from the top of the array link [] in descending order of the score, and the time when the highlight exists. And Here, m is an arbitrary natural number and represents the number of highlights acquired from a series of videos.

  Finally, in step S290, the arithmetic processing unit 14b determines a start point and an end point corresponding to each highlight in order to generate a highlight section including the time of each highlight extracted in step S280. The start point and end point may simply be a fixed time before or after the time when the highlight exists, or the cut point of the video closest to the time when the highlight exists is searched and set as the start point or end point. Good.

  Then, when the arithmetic processing unit 14b of the highlight selection unit 14 determines the start point and the end point corresponding to each highlight, the result is output to the external reproduction control device 2. The reproduction control device 2 reproduces the highlight video in the input video based on the output from the highlight selection unit 14.

  As described above, according to the present embodiment, the relative time in the input image of the flash emission point of the camera reflected in the input image is detected as the flash emission time, and from the detected emission time, Based on the number of light emission times included in the predetermined time, a predetermined number of light emission times are selected as the time when the highlight exists, and the section including the selected time is selected as the highlight period. It is possible to extract important scenes as highlights even for private content shot by a video camera. This makes it possible to quickly access and play back scenes where events are lively, and to view summary content of private content, for example, in content taken of events such as weddings and presentations.

  The present invention makes it possible to detect highlights of private contents with a simple process by assuming that scenes where the flash of the camera is frequently emitted are important scenes. Since complicated processing such as face recognition and voice recognition is not required, it can be applied to an embedded device such as a DVR (Digital Video Recorder) or a video camera, which cannot expect a high calculation capability.

  Embodiments in which the present invention is applied to a DVR and a video camera will be described below.

(Example 1)
FIG. 5 is a block diagram showing the configuration of a DVR to which the present invention is applied. As shown in FIG. 5, the DVR 30 is generated by an interface 31 that mediates input / output with an external device, an encoding / decoding circuit 32 that performs compression encoding of video and decoding of encoded data, and an encoding / decoding circuit 32. HDD 33 for storing the encoded data, a user interface 34 for receiving a user operation and outputting the result, and a control unit 35 for controlling each part of the apparatus.

  The control unit 35 includes a CPU (Central Processing Unit) 35a and a memory 35b. The CPU 35a executes the light emission time detection process shown in the flowchart of FIG. 2 and the highlight selection process shown in the flowchart of FIG. 4 according to the program stored in the memory 35b. In addition, an external storage 41 and a television monitor 42 are connected to the outside of the DVR 30 via an interface 31.

  Assume that a home video camera is connected as the external storage 41 in the DVR 30 having such a configuration. Under the control of the CPU 35a, the video reproduced by the video camera is input to the encoding / decoding circuit 32 via the interface 31, where it is converted into encoded data by compression encoding processing, and the generated encoded data is the HDD 33. Stored in

  Then, the CPU 35 a reads out the video from the encoding / decoding circuit 32 or the HDD 33 according to the program of the light emission time detection process, executes the light emission time detection process shown in the flowchart of FIG. 2, and stores the detected light emission time in the HDD 33. .

  When the user instructs playback of the highlight section via the user interface 34, the CPU 35a reads the light emission time from the HDD 33, and executes the highlight selection processing shown in the flowchart of FIG. 4 according to the highlight selection processing program. The highlight section to be reproduced is determined.

  Based on the determined highlight section, the CPU 35a reads the encoded video data of the section to be reproduced from the HDD 33, decodes the read encoded data, and outputs the decoded video via the interface 31. Thus, the encoding / decoding circuit 32 is controlled. The output video is displayed on a television monitor 42 connected to the outside of the DVR 30.

(Example 2)
FIG. 6 is a block diagram showing the configuration of a home video camera equipped with an HDD to which the present invention is applied. As shown in FIG. 6, the video camera 50 includes an image sensor 51, an image sensor driver circuit 52 that drives the image sensor 51, an encoding / decoding circuit 53 that performs compression encoding of video and decoding of encoded data, and encoding. A HDD 54 that stores the encoded data generated by the decoding circuit 53, a liquid crystal display 55 that displays video, a liquid crystal driver circuit 56 that drives the liquid crystal display 55, and a user that receives user operations and outputs the results. An interface 57 and a control unit 58 that controls each unit of the apparatus are provided.

  The control unit 58 includes a CPU 58a and a memory 58b. The CPU 58a executes the light emission time detection process shown in the flowchart of FIG. 2 and the highlight selection process shown in the flowchart of FIG. 4 according to the program stored in the memory 58b.

  In the video camera 50 having such a configuration, an image captured by the image sensor 51 is read into the control unit 58 via the image sensor driver circuit 52 under the control of the CPU 58a. The video is compressed and encoded by the encoding / decoding circuit 53, and the generated encoded data is stored in the HDD 54.

  At the same time, the CPU 58a reads out the video from the encoding / decoding circuit 53 or the HDD 54 according to the program of the light emission time detection process, executes the light emission detection process shown in the flowchart of FIG. 2, and stores the detected light emission time in the HDD 54. .

  When the user instructs playback of the highlight section via the user interface 57, the CPU 58a reads the light emission time from the HDD 54, and executes the highlight selection processing shown in the flowchart of FIG. 4 according to the highlight selection processing program. The highlight section to be reproduced is determined.

  Then, based on the determined highlight section, the CPU 58 a reads the encoded video data of the section to be reproduced from the HDD 54, decodes the read encoded data, and outputs the decoded video to the liquid crystal driver circuit 56. Thus, the encoding / decoding circuit 53 is controlled. Thereafter, the CPU 58a controls the liquid crystal driver circuit 56 so as to display the video in the highlight section on the liquid crystal display 55.

  In the second embodiment, the example in which the video information recorded in the HDD is analyzed is shown, but the recording medium may be an optical disk or a semiconductor memory. Further, since the same method can be applied to any medium connected to a network, it can also be applied to a video content service using communication.

It is a block diagram which shows the structure of the highlight extraction apparatus which concerns on one embodiment of this invention. It is a flowchart which shows the procedure which detects the light emission time of a flash. It is a figure which shows an example of the histogram of a brightness | luminance. It is a flowchart which shows the procedure which selects a highlight area. It is a block diagram which shows the structure of DVR to which this invention is applied. It is a block diagram which shows the structure of the video camera to which this invention is applied.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Highlight extraction apparatus 11 Image | video memory | storage part 12 Light emission detection part 12a Buffer 12b Arithmetic processing part 13 Storage part 14 Highlight selection part 14a Memory 14b Arithmetic processing part 30 DVR
31 Interface 32 Encode / Decode Circuit 33 HDD
34 User Interface 35 Control Unit 35a CPU
35b Memory 41 External storage 42 TV monitor 50 Video camera 51 Image sensor 52 Image sensor driver circuit 53 Encode / decode circuit 54 HDD
55 Liquid Crystal Display 56 Liquid Crystal Driver Circuit 57 User Interface 58 Control Unit 58a CPU
58b memory

Claims (2)

A highlight extraction device that extracts, as a highlight, a portion that becomes an important scene in the video based on light emission of a camera flash reflected in the video,
Flash detection means for detecting the presence or absence of flash emission for each frame of the input video and detecting the relative time in the input video of the frame detected as having flash emission as the flash emission time;
Storage means for storing the light emission time detected by the light emission detection means;
For each of the light emission times stored in the storage means, the number of light emission times included within a predetermined time before the light emission time is calculated as a score of each light emission time, and the light emission time is calculated based on the light emission time. When it is included in the predetermined time before the light emission time having a score higher than the score, after canceling the score of the light emission time, the time when the highlight exists for a predetermined number of the light emission times from the highest score And highlight selection means for selecting a section including the selected time as a highlight section in the input video. A highlight extraction device that extracts, as a highlight, a portion that becomes an important scene in the video based on light emission of a camera flash reflected in the video,
A luminance histogram is generated for each frame of the input video, and a correlation between the luminance histogram of the frame and the luminance histogram of the first predetermined number of frames before the frame is calculated as a first correlation, Calculating a correlation with a luminance histogram of a second predetermined number of frames larger than the first predetermined number from the frame as a second correlation, wherein the first correlation is smaller than a predetermined first threshold; And when the second correlation is larger than a predetermined second threshold, a light emission detecting means for performing an operation for each frame to detect a relative time in the input video of the frame as a light emission time of the flash;
Storage means for storing the light emission time detected by the light emission detection means;
For each of the light emission times stored in the storage means, the number of light emission times included within a predetermined time before the light emission time is calculated as a score of each light emission time, and the light emission time is calculated based on the light emission time. When it is included in the predetermined time before the light emission time having a score higher than the score, after canceling the score of the light emission time, the time when the highlight exists for a predetermined number of the light emission times from the highest score And highlight selection means for selecting a section including the selected time as a highlight section in the input video.

Images