JP2012105205A - Key frame extractor, key frame extraction program, key frame extraction method, imaging apparatus, and server device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract a key frame from moving image data accurately at a low cost.SOLUTION: A key frame extractor includes: a scene analysis part 510 for analyzing the moving image data and detecting scenes; a key frame extraction part 520 for extracting a frame image time-sequentially at the center or near the center as a key frame from the plurality of frame images corresponding to the scene having the longest scene length among the plurality of scenes detected by the scene analysis part 510; a feature amount analysis part 530 for normalizing the key frame, extracting the feature amount of an image, executing clustering processing, and generating a feature amount histogram; and a classification processing part 540 for performing machine learning and classification based on the feature amount histogram.

Description

  The present invention relates to a key frame extraction device, a key frame extraction program, a key frame extraction method, an imaging device, and a server device.

  A technique for extracting a key frame from a video clip (moving image data) is known (see, for example, Patent Document 1). In this patent document, a camera is provided with a camera motion sensor, a global motion is calculated to form a plurality of video segments, each segment is labeled according to a series of camera motion classes, and a key frame candidate is determined from the labeled segments. A technique for extracting the above is disclosed. This global motion is obtained by calculating camera work obtained from a camera motion sensor and video.

Special table 2009-539273 gazette

However, in the key frame candidate extraction method disclosed in Patent Document 1, it is necessary to provide a camera motion sensor in the camera. Further, in this extraction method, global motion is calculated, but the calculation load of this global motion is heavy and high speed processing capability is required. That is, the cost for realizing the key frame candidate extraction method disclosed in this document is high.
Therefore, the present invention has been made in view of the above circumstances, and it is not necessary to use a special sensor, and it is possible to extract key frames from moving image data with high accuracy and at low cost without requiring high arithmetic processing capability. An object of the present invention is to provide a key frame extraction device, a key frame extraction program, a key frame extraction method, an imaging device, and a server device.

[1] In order to solve the above-described problem, a key frame extraction device according to an aspect of the present invention includes a scene analysis unit that analyzes moving image data to detect a scene, and a scene of the scene detected by the scene analysis unit And a key frame extraction unit that extracts a key frame based on the length and the position of the image data in the time direction in the plurality of image data corresponding to the scene.
[2] In order to solve the above-described problem, a key frame extraction program according to one aspect of the present invention includes a scene analysis unit that detects a scene by analyzing moving image data, and a scene analysis unit that detects a computer. Based on the scene length of the scene and the position of the image data in the time direction in the plurality of image data corresponding to the scene, the key frame extracting unit is configured to extract a key frame.
[3] In order to solve the above problem, a key frame extraction method according to one aspect of the present invention includes a scene analysis step in which a scene analysis unit analyzes a moving image data to detect a scene, and a key frame extraction unit includes A key frame extraction step for extracting a key frame based on a scene length of the scene detected by the scene analysis unit in the scene analysis step and a position of image data in a time direction in a plurality of image data corresponding to the scene; It is characterized by having.
[4] In order to solve the above-described problem, an imaging apparatus according to an aspect of the present invention includes an imaging unit that captures and generates moving image data, and analyzes the moving image data generated by the imaging unit to generate a scene. A scene analysis unit to detect, and a key frame extraction unit to extract a key frame based on a scene length of the scene detected by the scene analysis unit and a position of image data in a time direction in a plurality of image data corresponding to the scene A thumbnail generation unit that generates reduced image data based on the key frame extracted by the key frame extraction unit; and a display unit that displays the reduced image data generated by the thumbnail generation unit.
[5] In order to solve the above-described problem, a server device according to one aspect of the present invention analyzes a moving image data storage unit that stores moving image data, and moving image data stored in the moving image data storage unit And extracting a key frame based on a scene analysis unit for detecting a scene, a scene length of the scene detected by the scene analysis unit, and positions of image data in a time direction in a plurality of image data corresponding to the scene A thumbnail generation unit that generates reduced image data based on the key frame extracted by the key frame extraction unit, and stores the reduced image data in the moving image data storage unit in association with the moving image data And.

  According to the present invention, it is not necessary to use a special sensor, and key frames can be extracted from moving image data with high accuracy and at low cost without requiring high calculation processing capability.

It is a block diagram showing the functional structure of the imaging device to which the key frame extracting device which is 1st Embodiment of this invention is applied. It is a block diagram showing the functional composition of the key frame extraction device in the embodiment. In the same embodiment, it is a schematic diagram for demonstrating spatiotemporal image data. In the same embodiment, it is the figure which represented typically a mode that a key frame extraction part extracts a key frame from normalized moving image data. In the embodiment, it is an example of an image of a key frame extracted by a key frame extraction unit. It is a flowchart showing the operation | movement procedure of the key frame extraction apparatus in the embodiment. In the embodiment, it is the figure which represented typically the list of the compression moving image data displayed on the display part. It is the figure which represented typically a mode that a key frame extraction part extracts three key frames from normalized moving image data in the 1st modification of 1st Embodiment. In the same modification, it is an example of the image of three key frames which the key frame extraction part extracted. In the modification, the histogram of each of the three key frame images in FIG. 9A and the integrated histogram generated by the clustering processing unit adding these three histograms for each class are schematically shown. FIG. In the modification, it is the figure which represented typically the list of the compression moving image data displayed on the display part. It is the figure which represented typically a mode that a key frame extraction part extracts three key frames from normalized moving image data in the 2nd modification of 1st Embodiment. In the same modification, it is an example of the image of three key frames which the key frame extraction part extracted. In the modified example, the histogram of each of the three key frame images in FIG. 13A and the integrated histogram generated by the clustering processing unit adding these three histograms for each class are schematically shown. FIG. It is the figure which represented typically a mode that a key frame extraction part extracts nine key frames from normalized moving image data in the 3rd modification of 1st Embodiment. It is a block diagram showing the whole structure of a network system including the server apparatus to which the key frame extraction apparatus which is 2nd Embodiment of this invention is applied.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
[First Embodiment]
FIG. 1 is a block diagram showing a functional configuration of an imaging apparatus to which a key frame extraction apparatus according to the first embodiment of the present invention is applied. As shown in the figure, the imaging apparatus 100 includes an imaging unit 110, a control unit 190, an operation unit 180, an image processing unit 140, a display unit 150, a storage unit 160, a buffer memory unit 130, and a communication. The unit 170 is connected via the bus 300.
In addition, a storage medium 200 is detachably attached to the imaging apparatus 100. Note that the storage medium 200 may be built in the imaging apparatus 100.

The imaging unit 110 is controlled by the control unit 190 based on the imaging conditions (for example, aperture value and exposure value) set by the control unit 190, and generates image data by imaging the light flux coming from the subject. In the present embodiment, the image data is frame image data. As will be described later, the imaging apparatus 100 handles still image data, which is photographic data, and moving image data, which is video (video) data. Still image data is one piece of image data. The moving image data is a plurality of image data. Further, the imaging unit 100 performs data compression processing of still image data and moving image data and stores the data in the storage medium 200. Still image data after data compression processing is compressed still image data, and moving image data after data compression processing is compressed moving image data. However, the imaging unit 100 may store still image data in the storage medium 200 without compression.
In the following description, a case where the imaging apparatus 100 handles moving image data will be described.
In the present embodiment, it will be described below that the data obtained by decoding the compressed moving image data is moving image data.

The imaging unit 110 includes an optical system 111, an imaging element 119, and an analog / digital (A / D) conversion unit 120 as functional configurations.
The optical system 111 has a lens group including an objective lens and a converging lens. The optical system 111 condenses the light flux coming from the subject and forms an image on the imaging surface of the image sensor 119.
The image pickup device 119 picks up an object image formed on the image pickup surface by photoelectric conversion and generates an image signal that is an analog signal. The image sensor 119 is realized by a solid-state image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, for example.
The A / D conversion unit 120 takes in the image signal generated by the image sensor 119 and converts it into image data that is digital data.

  The optical system 111 may be provided integrally with the imaging apparatus 100 or may be detachably attached to the imaging apparatus 100.

  The control unit 190 is realized by including a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The control unit 190 controls each unit via the bus 300 by reading the control program stored in the ROM into the RAM and executing each step. For example, the control unit 190 displays the image data output from the A / D conversion unit 120 on the display unit 150, or the data is compressed by the image processing unit 140 and stored in the storage medium 200. The control unit 190 also reads the compressed still image data and the compressed moving image data stored in the storage medium 200 into the buffer memory unit 130, decodes them by the image processing unit 140, and displays them on the display unit 150.

  Further, the control unit 190 includes a key frame extraction device 500 and a thumbnail generation unit 550 as functional configurations. The key frame extraction apparatus 500 takes in the moving image data read from the storage medium 200 to the buffer memory unit 130 by the control unit 190 and decoded by the image processing unit 140. Then, the key frame extracting apparatus 500 detects a scene from the captured moving image data. The key frame extraction apparatus 500 detects a scene by estimating the timing at which the scene changes (scene cut timing). Then, the key frame extraction apparatus 500 extracts a key frame based on the estimated scene cut timing, and outputs information for specifying the key frame (for example, frame identification information described later). Then, the key frame extraction device 500 classifies the key frames and outputs the classification data.

A scene in a moving image obtained by shooting is, for example, a section of video in which the same subject is expressed with temporal continuity, and the theme (theme) intended by the photographer is expressed in the scene. There are many. Separation of two scenes that are connected (scene cut) includes the start of imaging from the imaging stop state of the imaging apparatus 100, insertion of special effect images such as dissolves and wipes at the time of imaging, quick panning and zooming of the imaging apparatus 100, etc. This is caused by camera work. Moreover, a scene cut may occur due to editing after shooting.
The key frame is one or more representative image data of the moving image data. The key frame image is preferably an image representing the theme of the scene. The key frame extracting apparatus 500 according to the present embodiment extracts key frames in moving image data with high accuracy.

  The thumbnail generation unit 550 generates thumbnail image data (reduced image data) obtained by reducing the resolution of image data corresponding to the key frame based on the key frame extracted by the key frame extraction apparatus 500, and includes the key frame. It is stored in the storage medium 200 in association with moving image data.

The operation unit 180 has operation keys such as a power switch, a shutter button, a cross key, a confirmation button, a cancel button, and a menu button, and generates an operation key signal according to operation of the operation key by an operator such as a photographer. It supplies to the control part 190.
The image processing unit 140 executes image processing on the image data stored in the buffer memory unit 130 based on the image processing conditions stored in the storage unit 160. The image processing conditions are, for example, conditions relating to image data compression processing. For example, designation of data compression methods such as MPEG (Moving Picture Experts Group) -4, Motion JPEG (Joint Photographic Experts Group), MPEG-2, etc., data Specifies the degree of compression.
The buffer memory unit 130 is a storage unit that temporarily stores (buffers) data relating to various images such as still image data, moving image data, compressed still image data, and compressed moving image data. The buffer memory unit 130 includes, for example, image data output from the A / D conversion unit 120, various data received by the control unit 190 from the storage medium 200 via the communication unit 170, and data compression processing and decoding processing performed by the image processing unit 140. Various data generated when executing is stored.

The display unit 150 is realized by, for example, a liquid crystal display device, displays an image captured by the imaging unit 110, displays still image data or moving image data decoded by the image processing unit 140, and displays various menus. Or
The storage unit 160 stores information such as imaging conditions referred to by the control unit 190 and image processing conditions referred to by the image processing unit 140.
The communication unit 170 has a connection interface that can be connected to the storage medium 200. When the storage medium 200 is connected to the connection interface, the communication unit 170 writes, reads, or writes compressed still image data or compressed moving image data to the storage medium 200 based on the control of the control unit 190. Erase processing and the like are executed.

  The storage medium 200 is realized by a memory card, for example. The storage medium 200 stores compressed still image data and compressed moving image data. The storage medium 200 also stores thumbnail image data associated with moving image data.

Next, the configuration of the key frame extraction apparatus 500 will be described. FIG. 2 is a block diagram illustrating a functional configuration of the key frame extracting apparatus 500. As shown in the figure, the key frame extraction apparatus 500 includes a scene analysis unit 510, a key frame extraction unit 520, a feature amount analysis unit 530, and a classification processing unit 540.
When the moving image data read out from the storage medium 200 to the buffer memory unit 130 and decoded by the image processing unit 140 is captured by the control unit 190, the key frame extracting device 500 reads the moving image data into the key frame extracting device 500. The data are supplied to the scene analysis unit 510 and the feature amount analysis unit 530, respectively.

  The scene analysis unit 510 captures and analyzes the supplied moving image data, and detects a scene by estimating a scene cut timing. The scene analysis unit 510 includes a spatiotemporal image generation unit 511 and a scene detection unit 512 as functional configurations.

  The spatiotemporal image generation unit 511 normalizes the moving image data according to a predetermined attribute, and then changes the pixel value of the frame image over time from the normalized moving image data (normalized moving image data). Generate spatiotemporal image data to represent. The predetermined attributes are, for example, the aspect ratio, resolution, color space, and frame rate of the image data. Details of the process of generating the spatiotemporal image data by the spatiotemporal image generation unit 511 will be described later.

The scene detection unit 512 detects the scene by taking in the spatiotemporal image data generated by the spatiotemporal image generation unit 511 and analyzing the spatiotemporal image data to estimate the scene cut timing. The scene detection unit 512 extracts frame identification information of frame images corresponding to all estimated scene cut timings, and associates the frame identification information with the normalized moving image data and supplies them to the key frame extraction unit 520. . The frame identification information is information for specifying the frame images constituting the normalized moving image data, in other words, information for specifying the image data constituting the moving image data, for example, time stamp information. The time stamp information is represented by, for example, “hour: minute: second.frame number”.
Details of the process in which the scene detection unit 512 detects a scene will be described later.

  The key frame extraction unit 520 takes in the frame identification information and the normalized moving image data supplied from the scene detection unit 512 of the scene analysis unit 510, and extracts a frame image that becomes one key frame from the normalized moving image data. Then, the frame identification information corresponding to the extracted frame image is supplied to the thumbnail generation unit 550 of the control unit 190 and also to the feature amount analysis unit 530.

In the present embodiment, the key frame extraction unit 520 extracts key frames based on the following three assumptions relating to the characteristics of the moving image.
Assumption 1: The compositional change in the scene is small.
Assumption 2: The frame image near the scene cut is not an image with high theme characteristics.
Assumption 3: The moving image theme is included in a scene having a long scene length.

Assumption 3 and Assumption 1 are based on the fact that in a scene with a long scene length, it can be assumed that the photographer has maintained interest in the subject. Assumption 2 is that, in the vicinity of the end of the scene, the photographer's interest in the scene has diminished, is no longer interested, whether special video effects such as fade-out are operating, or just before the end of shooting. Based on the high probability. Assumption 2 is that, immediately after the start of the scene, the photographer is determining the composition of a new scene, whether a special video effect such as fade-in is operating, It is based on the high possibility of being in a situation such as.
A method by which the key frame extraction unit 520 extracts key frames will be described later.

The feature amount analysis unit 530 takes in the moving image data and the frame identification information supplied from the key frame extraction unit 520, respectively, and from the frame image corresponding to the frame identification information, an image feature amount (hereinafter simply referred to as “feature amount”). And the clustering process is executed.
The feature amount analysis unit 530 includes an image normalization unit 531, a feature amount extraction unit 532, and a clustering processing unit 533 as its functional configuration.

The image normalization unit 531 extracts image data corresponding to the frame identification information from the moving image data, normalizes the extracted image data according to a predetermined attribute, and generates normalized image data. The predetermined attributes are, for example, the aspect ratio, resolution, and color space of the frame image.
Specifically, for example, the image normalization unit 531 generates luminance image data by extracting only luminance components from image data that is a key frame. Next, the image normalization unit 531 trims the luminance image data into luminance image data in which the aspect ratio of the frame image is 4 to 3. For example, when trimming processing of luminance image data having an aspect ratio of 16: 9, 21: 9, 1: 1, etc. into 4: 3 luminance image data, rectangular image portions at both ends in the horizontal direction of the luminance image data before trimming And adjust so that the aspect ratio is 4 to 3. Note that a non-patterned (for example, black or gray) rectangular frame may be added to both ends in the horizontal direction of the luminance image data before trimming so as to match the aspect ratio. Next, the image normalization unit 531 converts the trimmed luminance image data into normalized image data having a resolution of 320 pixels in the horizontal direction × 240 pixels in the vertical direction by filtering processing or the like.

  The reason for providing the image normalization unit 531 is that if the aspect ratio of the frame image is different, the density of the subject in the key frame is different, and if the resolution is different, the information amount of the key frame is different. This is to prevent the evaluation of the feature value of the frame from being affected.

  The feature amount extraction unit 532 takes in the normalized image data generated by the image normalization unit 531 and extracts the feature amount from the normalized image data. For example, the feature amount extraction unit 532 checks the state of the luminance Gaussian distribution for each small region including a plurality of pixels in the normalized image data, and obtains a SIFT (Scale-Invariant Feature Transform) feature amount for each feature point.

  The clustering processing unit 533 takes in the feature amount of the normalized image data extracted by the feature amount extraction unit 532, executes a feature amount clustering process (Bag-of-words process) for each feature point, and performs a feature amount histogram ( Feature amount histogram). The clustering processing unit 533 generates the feature amount histogram by classifying the feature amounts of the normalized image data into K (for example, 1000) clusters by, for example, the K average method.

  The classification processing unit 540 analyzes the feature amount histogram generated by the clustering processing unit 533, classifies normalized image data, that is, key frames, and outputs classification data. For example, the classification processing unit 540 learns the classification of feature amounts by machine learning and outputs the classification result as classification data. The classification data includes identification information corresponding to the classification, keywords such as “tennis game”, “cookie making”, “soccer”, and the like that are determined in advance through learning. The classification processing unit 540 learns the classification of feature quantities using, for example, a support vector machine (Support Vector Machine) or a neural network.

  Next, processing in which the spatiotemporal image generation unit 511 generates spatiotemporal image data will be described. First, the spatiotemporal image generation unit 511 thins out image data of moving image data. For example, the spatiotemporal image generation unit 511 thins out the moving image data having a frame rate of 30 frames / second (fps; frame per second) so that the frame rate becomes 2 frames / second. However, this thinning process may be omitted. Next, the spatio-temporal image generation unit 511 generates luminance moving image data by extracting only luminance components from the thinned moving image data. Next, the spatio-temporal image generation unit 511 performs a trimming process on each image data of the luminance moving image data to image data in which the aspect ratio of the frame image is 4 to 3, for example. The trimming process is the same process as the process in the image normalization unit 531 described above. Next, the spatio-temporal image generation unit 511 converts the trimmed luminance moving image data into normalized moving image data whose resolution is, for example, 320 pixels in the horizontal direction × 240 pixels in the vertical direction by filtering processing or the like. Next, the spatiotemporal image generation unit 511 generates spatiotemporal image data from the normalized moving image data.

FIG. 3 is a schematic diagram for explaining the spatiotemporal image data. FIG. 4A is a diagram schematically showing a frame configuration of normalized moving image data that is the source of spatiotemporal image data. FIG. 6B is a diagram schematically showing the data configuration of the spatiotemporal image data generated by the spatiotemporal image generation unit 511 based on the normalized moving image data shown in FIG.
As shown in FIG. 5A, the normalized moving image data has the following attributes: the frame image has an aspect ratio of 4: 3, the resolution is 320 pixels in the horizontal direction × 240 pixels in the vertical direction, and the frame rate is 2 frames. / Sec. Here, an example in which the reproduction time is 351 seconds is shown. The horizontal lines in one frame image are represented as line 1, line 2,..., Line 30 from the upper end to the lower end of the frame image.

  The spatiotemporal image generation unit 511 generates spatiotemporal image data that is two-dimensional data by replacing the normalized moving image data with time (frame position) versus spatial coordinates as shown in FIG. In other words, the spatio-temporal image generation unit 511 generates spatio-temporal image data by arranging the pixel values from line 1 to line 30 in a line in the spatial coordinate axis direction for each frame image of the normalized moving image data. The spatiotemporal image data obtained from the normalized moving image data shown in FIG. 5A is one piece of image data for 702 frames in the time axis direction × 1200 pixels in the spatial coordinate axis direction.

Next, processing in which the scene detection unit 512 detects a scene will be described. The scene detection unit 512 detects an edge of the image by calculating, for example, a luminance gradient for the spatiotemporal image data. For example, the scene detection unit 512 detects the edge of the image from the spatiotemporal image data by the Canny method. The linear direction of the edge of the image detected here includes the vertical, horizontal, and diagonal directions.
Next, the scene detection unit 512 detects an edge that is a straight line parallel to the spatial coordinate axis from the detected edges of the image. For example, the scene detection unit 512 extracts only a linear component parallel to the spatial coordinate axis from the detected edge of the image by executing a Hough conversion process.

  Next, the scene detection unit 512 estimates, for each frame image, whether the frame image corresponds to a shot cut. Specifically, the scene detection unit 512 calculates the sum of the linear components in the spatial coordinate direction for each frame image (one pixel in the time direction), and if the sum is equal to or greater than a predetermined threshold, the frame image Is estimated to be a shot cut, and if the sum is less than a predetermined threshold, it is estimated that the frame image is not a shot cut. The estimated shot cut is a candidate for scene cut timing. The predetermined threshold value can be arbitrarily determined, and is defined by, for example, the ratio of the height in the spatial coordinate direction of the spatiotemporal image data. For example, the predetermined threshold value is 1/60 of the height in the spatiotemporal coordinate direction in the spatiotemporal image data (20 pixels if the height is 1200 pixels).

  Next, the scene detection unit 512 excludes a cut that is a mere shot and is not an effective scene from the linear component estimated to be a shot cut. This “cut that is a shot and is not an effective scene” corresponds to, for example, a special effect image such as dissolve or wipe. Specifically, in the linear component estimated to be a shot cut, the scene detection unit 512 uses the two when the time interval between two linear components adjacent in the time direction is shorter than a predetermined time. At least one of the linear components is excluded from the scene cut timing candidates. The predetermined time can be arbitrarily set, and is, for example, 2 seconds (in FIG. 3B, 4 pixels in the time direction).

  Next, the scene detection unit 512 outputs frame identification information and normalized moving image data of frame images corresponding to all the linear components remaining as scene cut timing candidates.

  Next, processing in which the key frame extraction unit 520 extracts key frames will be described with reference to FIG. FIG. 4 is a diagram schematically illustrating how the key frame extraction unit 520 extracts key frames from the normalized moving image data. The scene extraction unit 520 takes in the frame identification information output from the scene detection unit 512 of the scene analysis unit 510 and the normalized moving image data, and calculates the scene length of each scene in the normalized moving image data. This scene length is a time length or the number of frames. That is, based on the fact that the frame image corresponding to the frame identification information is the first frame image of the scene, the scene extraction unit 520 calculates the scene length of each scene. For example, in the case of the scene configuration shown in FIG. 5A, the frame image corresponding to the frame identification information is the first frame image of each scene, and the key frame extraction unit 520 displays the first scene (scene 1). To the fifth scene (scene 5) are counted as 10 frames, 6 frames, 15 frames, 12 frames, and 4 frames.

Next, the key frame extraction unit 520 detects the scene having the longest scene length (the number of frames is large) from the scenes according to the assumption 3 described above. For example, the key frame extraction unit 520 compares the number of frames of each scene to determine the third scene (scene 3: 15 frames) as the scene with the largest number of frames as shown in FIG. To detect.
Next, the key frame extraction unit 520 detects a frame image having a predetermined range of luminance values from the frame image corresponding to one detected scene from the normalized moving image data. Specifically, the key frame extraction unit 520 includes a low-luminance frame image having an average luminance value lower than the first threshold value from a frame image corresponding to one detected scene in the normalized moving image data, or A high-luminance frame image whose average luminance value is higher than a second threshold (second threshold> first threshold), or a low-luminance frame image and a high-luminance frame image are detected. FIG. 7B shows an example in which four low-luminance frame images and high-luminance frame images are detected from the 15-frame frame image corresponding to the third scene (scene 3).

Next, the key frame extraction unit 520 performs time-series central processing in accordance with Assumption 2 described above from the low-luminance frame image, the high-luminance frame image, or the frame image excluding the low-luminance frame image and the high-luminance frame image. Alternatively, one frame image near the center (for example, closest to the time series center) is extracted as a key frame. As described above, the reason for excluding the low-luminance frame image and the high-luminance frame image is that these frame images are generally difficult to see, and it is difficult for the feature amount analysis unit 530 in the subsequent stage to detect the feature amount. FIGS. 4C and 4D show an example in which the key frame extraction unit 520 extracts the sixth frame image from the left, which is a frame image centered in time series, from 11 frame images as a key frame. Show.
Next, the key frame extraction unit 520 supplies frame identification information corresponding to the extracted key frame to the thumbnail generation unit 550 of the control unit 190 and also to the feature amount analysis unit 530.

FIG. 5 is an example of a key frame image extracted by the key frame extraction unit 520. FIG. 5A shows a key frame image extracted by the key frame extraction device 500 using moving image data obtained by photographing the state of the tennis game by the imaging device 100. In this moving image data, a scene where two tennis players on the front side and the back side continue the rally as shown in FIG.
FIG. 5B shows a key frame image extracted by the key frame extraction device 500 using moving image data obtained by photographing the state of the girl making the cookie by the imaging device 100. In this moving image data, a scene with a composition in which the hand of the girl cannot be seen from the frame image as shown in FIG.

Next, main operations of the key frame extraction apparatus 500 according to the present embodiment will be described. FIG. 6 is a flowchart showing an operation procedure of the key frame extraction apparatus 500.
When the key frame extraction device 500 captures moving image data, in step S1, the spatiotemporal image generation unit 511 of the scene analysis unit 510 thins out the image data of the moving image data, and then extracts the moving image data after the thinning out to a predetermined value. Normalized moving image data normalized according to the attribute is generated. For example, as shown in FIG. 3A, the spatiotemporal image generation unit 511 has a resolution of 320 pixels in the horizontal direction × 240 pixels in the vertical direction, and a normalized moving image expressed by luminance information at a frame rate of 2 frames / second. Image data is generated.
Next, the spatiotemporal image generation unit 511 replaces the normalized moving image data with time (frame position) versus space coordinates to generate spatiotemporal image data that is two-dimensional data. For example, the spatio-temporal image generation unit 511 obtains spatio-temporal image data for 702 frames in the time direction × 1200 pixels in the spatial coordinate direction from the normalized moving image data shown in FIG. Generate.

In step S2, the scene detection unit 512 detects the scene by taking in the spatiotemporal image data generated by the spatiotemporal image generation unit 511 and analyzing the spatiotemporal image data to estimate the scene cut timing. . Specifically, the scene detection unit 512 detects the edge of the image by calculating, for example, a luminance gradient for the spatiotemporal image data.
Next, the scene detection unit 512 detects an edge that is a straight line in the spatial coordinate direction from the detected edges of the image.
Next, the scene detection unit 512 estimates, for each frame image, whether the frame image corresponds to a shot cut. Specifically, the scene detection unit 512 calculates the sum of the linear components in the spatial coordinate direction for each frame image. If the sum is equal to or greater than a predetermined threshold, the scene image is estimated to be a shot cut. If the sum is less than a predetermined threshold, it is estimated that the frame image is not a shot cut.
Next, the scene detection unit 512 excludes a cut that is a mere shot and is not an effective scene from the straight line component (scene cut timing candidate) estimated to be a shot cut. Specifically, in the linear component estimated to be a shot cut, the scene detection unit 512 uses the two when the time interval between two linear components adjacent in the time direction is shorter than a predetermined time. At least one of the linear components is excluded from the scene cut timing candidates.
Next, the scene detection unit 512 associates the frame identification information of the frame image corresponding to all the straight line components remaining as scene cut timing candidates and the normalized moving image data, and supplies them to the key frame extraction unit 520. .

Next, in step S3, the key frame extraction unit 520 takes in the frame identification information and the normalized moving image data supplied from the scene detection unit 512, and the frame image that becomes one key frame from the normalized moving image data. The frame identification information corresponding to is extracted and supplied to the feature amount analysis unit 530. Specifically, the scene extraction unit 520 calculates the scene length of each scene in the normalized moving image data based on the frame identification information and the normalized moving image data.
Next, the key frame extraction unit 520 detects a scene having the longest scene length from the scenes.
Next, the key frame extraction unit 520 detects a frame image having a predetermined range of luminance values from the frame image corresponding to one detected scene from the normalized moving image data. Specifically, the key frame extraction unit 520 includes a low-luminance frame image having an average luminance value lower than the first threshold value from a frame image corresponding to one detected scene in the normalized moving image data, or A high luminance frame image whose average luminance value is higher than the second threshold value, or a low luminance frame image and a high luminance frame image are detected.
Next, the key frame extraction unit 520 time-sequentially selects one of the low-luminance frame image, the high-luminance frame image, or the frame image excluding the low-luminance frame image and the high-luminance frame image in time series. A frame image is extracted as a key frame.
Next, the key frame extraction unit 520 supplies frame identification information corresponding to the extracted key frame to the thumbnail generation unit 550 of the control unit 190 and also to the feature amount analysis unit 530.

Next, in step S4, the image normalization unit 531 of the feature amount analysis unit 530 takes in the moving image data and the frame identification information supplied from the key frame extraction unit 520, and obtains image data corresponding to the frame identification information. Extract from video data.
Next, the image normalization unit 531 generates normalized image data obtained by normalizing the extracted image data according to a predetermined attribute. For example, the image normalization unit 531 generates normalized image data represented by luminance information with a resolution of 320 pixels in the horizontal direction × 240 pixels in the vertical direction.
Next, in step S5, the feature amount extraction unit 532 takes in the normalized image data generated by the image normalization unit 531 and extracts the feature amount from the normalized image data.
Next, in step S <b> 6, the clustering processing unit 533 performs a clustering process of the feature amounts of the normalized image data supplied from the feature amount extraction unit 532 to generate a feature amount histogram.

  In step S7, the classification processing unit 540 takes in the feature amount histogram generated by the clustering processing unit 533, analyzes the feature amount histogram, classifies normalized image data, that is, key frames, and outputs the classification result. Output as classification data.

Next, main operations of the imaging apparatus 100 to which the key frame extraction apparatus 500 is applied will be described. First, a process in which the imaging apparatus 100 generates thumbnail image data of key frames in compressed moving image data stored in the storage medium 200 based on the key frame extraction result of the key frame extraction apparatus 500 will be described.
When the key frame extraction apparatus 500 extracts a key frame and outputs frame identification information corresponding to the key frame, the thumbnail generation unit 550 of the control unit 190 takes in the frame identification information.
Next, the thumbnail generation unit 550 extracts and captures image data corresponding to the frame identification information from the storage medium 200 or the buffer memory unit 130.
Next, the thumbnail generation unit 550 generates thumbnail image data in which the resolution of the captured image data is reduced.
Next, the thumbnail generation unit 550 stores the generated thumbnail image data in the storage medium 200 in association with the corresponding compressed moving image data. The thumbnail image data may be stored in the header portion of the corresponding compressed moving image data.

Note that the thumbnail generation unit 550 also captures the classification data output by the key frame extraction device 500 and compresses the classification data (for example, keywords such as “tennis game” and “cookie making”) together with the thumbnail image data. The data may be stored in the storage medium 200 in association with the data.
Further, the control unit 190 may organize (for example, directory management) the compressed moving image data stored in the storage medium 200 based on the classification data.

Next, processing for displaying a list of compressed moving image data associated with thumbnail image data stored in the storage medium 200 on the display unit 150 will be described.
When the operation unit 180 outputs an operation signal for causing the display unit 150 to display a list of compressed moving image data stored in the storage medium 200 by the operation of the operation unit 180 by the operator, the control unit 190 performs the operation. Capture the signal.
Next, the control unit 190 reads the thumbnail image data associated with the compressed moving image data and the file name of the electronic file storing the compressed moving image data from the storage medium 200 and causes the display unit 150 to display them.

  FIG. 7 is a diagram schematically showing a list of compressed moving image data displayed on the display unit 150. In the drawing, a list of three compressed moving image data is displayed on the display screen 600 of the display unit 150, and is associated with the thumbnail images 601, 602, 603 and the thumbnail images 601, 602, 603, respectively. “201010103.mp4”, “201010105.mp4”, and “201011112.mp4”, which are the file names of the electronic files of the compressed moving image data, are displayed. In the figure, a cursor 604 is displayed on the thumbnail image 601. The cursor 604 can be moved on the thumbnail images 601, 602, and 603 by operating the cross key of the operation unit 180. When the confirm button is operated, the control unit 190 selects compressed moving image data associated with the thumbnail image data indicated by the cursor 604.

  As described above, in the key frame extraction apparatus 500 according to the first embodiment of the present invention, the scene analysis unit 510 detects a scene of moving image data, and the key frame extraction unit 520 starts from the scene having the longest scene length. One frame image at the center or near the center in time series is extracted as a key frame. By configuring in this way, the key frame extraction apparatus 500 relates to the characteristics of the moving image “The moving image theme is included in a scene having a long scene length” (assuming 3) and “the frame image near the scene cut”. Is not an image with high theme properties. ”(Assumption 2) is satisfied, and key frames can be extracted with high accuracy. In particular, since the key frame extraction apparatus 500 extracts key frames in consideration of the characteristics of a moving image, the calculation processing load is light and can be realized at low cost.

  In the key frame extraction apparatus 500, the feature amount analysis unit 530 normalizes the image data that is the key frame to extract the feature amount of the image, and the feature amount is subjected to clustering processing to represent the feature amount characteristic. Was generated. The key frame extracting apparatus 500 classifies the feature amount histogram by opportunity learning and outputs the classification data. With this configuration, the key frame extraction apparatus 500 can categorize image data, which is an extracted key frame, according to image characteristics. Therefore, the image capturing apparatus 100 including the key frame extraction apparatus 500 can manage the moving image data to be easily viewed and easily searched by applying the key frame and / or classification data.

[First Modification of First Embodiment]
In the first embodiment described above, the key frame extraction unit 520 extracts a frame image that becomes one key frame from the scene having the longest scene length in the normalized moving image data according to Assumption 3 and Assumption 2. In the first modification of the present embodiment, an example will be described in which the key frame extraction unit 520 extracts a plurality of frame images serving as key frames from a scene having the longest scene length in normalized moving image data. The key frame extraction unit 520 in the present modification extracts key frames in consideration of assumption 1 in addition to assumption 3 and assumption 2.

  In the present modification, the configuration different from the first embodiment is a key frame extraction unit 520, a clustering processing unit 533 mainly of a feature amount analysis unit 530, and a thumbnail generation unit 550. Since the overall functional configuration in the present modification is the same as that of the first embodiment, in this modification, a block diagram is omitted, and description is made using configurations having the same reference numerals as those in the first embodiment. To do. In this modification, only differences from the first embodiment will be described.

  The key frame extraction unit 520 takes in the frame identification information and the normalized moving image data supplied from the scene detection unit 512 of the scene analysis unit 510, and N frames (N is 2 or more) that become key frames from the normalized moving image data. And the frame identification information corresponding to the extracted N frame images is supplied to the thumbnail generation unit 550 of the control unit 190 and to the feature amount analysis unit 530.

  A process in which the key frame extraction unit 520 extracts N key frames will be described with reference to FIG. FIG. 8 is a diagram schematically illustrating how the key frame extraction unit 520 extracts N (N = 3) key frames from the normalized moving image data. FIGS. 4A and 4B are the same as FIGS. 4A and 4B in the first embodiment, and the processing by the key frame extraction unit 520 is also the same, and therefore corresponds to FIG. The subsequent processing will be described.

  The key frame extraction unit 520 includes N low-frequency frame images, high-luminance frame images, or frame images excluding the low-luminance frame images and the high-luminance frame images according to the above-described assumption 2 and assumption 1, and N frames at equal intervals. Are extracted as key frames. In this case, the key frame extraction unit 520 does not extract frame images corresponding to at least both ends of the scene according to Assumption 2. Specifically, the key frame extraction unit 520 determines the interval between the frame images to be extracted by calculating the following formula (1) or formula (2). In Expressions (1) and (2), D is a value indicating the interval between extracted frame images, F is the number of frame images in the scene, and N is the number of key frames to be extracted.

In order not to extract frame images corresponding to at least both ends of the scene, the key frame extraction unit 520 does not divide the number F of frame images in the scene by the number N of key frames to be extracted. Calculate the value D. On the other hand, when the number F of frame images in the scene is divisible by the number N of key frames to be extracted, the key frame extraction unit 520 calculates the value D using Expression (2).
When the value D is less than 2, the key frame extraction unit 520 connects the frame images corresponding to the scene with the next longest scene length and calculates Expression (1) or Expression (2).

  The key frame extraction unit 520 extracts frame images in the order of Dth counted from the end of the array of frame images, and then Dth counted from the position. In the example of FIGS. 8C and 8D, the key frame extraction unit 520 calculates D = 3 according to the equation (1), the third counting from the end of the 11th frame, the third counting from the position, Further, the third corresponding frame image from the position is extracted as a key frame.

  Next, the key frame extraction unit 520 supplies frame identification information corresponding to the extracted N key frames to the thumbnail generation unit 550 of the control unit 190 and also to the feature amount analysis unit 530.

FIG. 9 is an example of a plurality (N = 3) of key frame images extracted by the key frame extraction unit 520. FIG. 5A shows a key frame image extracted by the key frame extraction device 500 using moving image data obtained by photographing the state of the tennis game by the imaging device 100. In this moving image data, a scene shot at a substantially constant camera angle as shown in FIG. 5A is included as a scene having the longest time.
FIG. 5B shows a key frame image extracted by the key frame extraction device 500 using moving image data obtained by photographing the state of the girl making the cookie by the imaging device 100. Also in this moving image data, a scene taken at a substantially constant camera angle (a scene in which the girl's hand cannot be seen from the frame image) as shown in FIG. Included as
The images of the three key frames in FIGS. 4A and 4B are arranged in time series from the left side to the right side.

The feature amount analysis unit 530 takes in the moving image data and the N pieces of frame identification information supplied from the key frame extraction unit 520, and extracts the feature amount from the frame image corresponding to the frame identification information for each frame identification information. Then, the clustering process is executed.
For each frame identification information, the image normalization unit 531 extracts image data corresponding to the frame identification information from the moving image data, normalizes the extracted image data according to a predetermined attribute, and generates normalized image data. . Since the detailed processing of the image normalization unit 531 is the same as that of the first embodiment, the description thereof is omitted.
The feature amount extraction unit 532 takes in N normalized image data generated by the image normalization unit 531 and extracts feature amounts from the normalized image data for each normalized image data. Detailed processing of the feature amount extraction unit 532 is the same as that of the first embodiment, and a description thereof will be omitted.

  The clustering processing unit 533 takes in the feature amounts of the N normalized image data extracted by the feature amount extraction unit 532, and performs feature amount clustering processing (Bag-of-words processing) for each normalized image data and for each feature point. ) To generate a feature amount histogram. The clustering processing unit 533 generates the feature amount histogram by classifying the feature amounts of the normalized image data into K (for example, 1000) clusters by, for example, the K average method. Next, the clustering processing unit 533 adds the generated N feature amount histograms for each class to generate an integrated histogram. This integrated histogram is a histogram representing the characteristics of a moving image of a scene including a key frame.

  FIG. 10 schematically illustrates the feature amount histograms of the three key frame images in FIG. 9A and the integrated histogram generated by the clustering processing unit 533 adding the three feature amount histograms for each class. FIG. As shown in the figure, the characteristic distribution is emphasized by adding the feature amount histograms of images of three key frames having relatively high image characteristic similarity. In other words, the theme of the movie is emphasized.

  The classification processing unit 540 analyzes the integrated histogram generated by the clustering processing unit 533 and classifies normalized image data, that is, key frames. Since the detailed processing of the classification processing unit 540 is the same as that of the first embodiment, description thereof is omitted.

Next, main operations of the imaging apparatus 100 to which the key frame extraction apparatus 500 is applied will be described. First, a process in which the imaging apparatus 100 generates thumbnail image data of key frames in compressed moving image data stored in the storage medium 200 based on the key frame extraction result of the key frame extraction apparatus 500 will be described.
When the key frame extraction apparatus 500 extracts N key frames and outputs frame identification information corresponding to the key frames, the thumbnail generation unit 550 of the control unit 190 outputs frame identification information corresponding to the N key frames. take in.
Next, the thumbnail generation unit 550 extracts and captures image data corresponding to the N pieces of frame identification information from the storage medium 200 or the buffer memory unit 130.
Next, the thumbnail generation unit 550 generates thumbnail image data with reduced resolution for each of the N pieces of captured image data.
Next, the thumbnail generation unit 550 stores the generated N thumbnail image data in the storage medium 200 in association with the corresponding compressed moving image data. Note that N thumbnail image data may be stored in the header portion of the corresponding compressed moving image data.

Next, processing for displaying a list of compressed moving image data associated with N thumbnail image data stored in the storage medium 200 on the display unit 150 will be described.
When the operation unit 180 outputs an operation signal for causing the display unit 150 to display a list of compressed moving image data stored in the storage medium 200 by the operation of the operation unit 180 by the operator, the control unit 190 performs the operation. Capture the signal.
Next, the control unit 190 reads N thumbnail image data associated with the compressed moving image data and the file name of the electronic file in which the compressed moving image data is stored from the storage medium 200, and displays it on the display unit 150. Display. The control unit 190 realizes the display of the N thumbnail image data on the display unit 150 by, for example, one of the following two methods.

The first method is a method of displaying N thumbnail image data side by side on the display unit 150. The order in which the N thumbnail image data are arranged is, for example, time series.
The second method is a method of sequentially switching and displaying N thumbnail image data at the same coordinates on the display unit 150 at predetermined intervals. The time interval for displaying the thumbnail image data can be arbitrarily set, for example, “1 second”.

  FIG. 11 is a diagram schematically showing a list of compressed moving image data displayed on the display unit 150. FIG. 5A shows a display example by the first method, and FIG. 6B shows a display example by the second method. In FIG. 6A, a list of four compressed moving image data is displayed on the display screen 610 of the display unit 150, and thumbnail image data is displayed side by side for each compressed moving image data. ing. When attention is paid to the top of the list of four compressed moving image data, the three thumbnail images 611a, 611b, and 611c and the electronic file file of the compressed moving image data associated with the thumbnail images 611a, 611b, and 611c. The name “201010103.mp4” is displayed. In the figure, a cursor 612 is displayed surrounding thumbnail images 611a, 611b, and 611c. The cursor 612 can be moved up and down on the display screen 610 by operating the cross key of the operation unit 180. When the confirmation button is operated, the control unit 190 selects the compressed moving image data associated with the thumbnail image data indicated by the cursor 612.

  In FIG. 11B, a list of three compressed moving image data is displayed on the display screen 620 of the display unit 150, and thumbnail images 621, 622, and 623 for time division switching and thumbnails for time division switching. “201010103.mp4”, “2010101105.mp4”, and “201011112.mp4”, which are the file names of the electronic files of the compressed moving image data associated with the images 621, 622, and 623, are displayed. In each of the thumbnail images 621, 622, and 623 subjected to time division switching, the thumbnail image data is switched every predetermined time (for example, every second). In the figure, a cursor 624 is displayed on the thumbnail image 621 for time division switching. By operating the cross key of the operation unit 180, the cursor 624 can be moved on the time-division switching thumbnail images 621, 622, and 623. When the confirmation button is operated, the control unit 190 selects the compressed moving image data associated with the thumbnail image data for time division switching indicated by the cursor 624.

  As described above, in the key frame extraction apparatus 500 that is the first modification of the first embodiment of the present invention, the scene analysis unit 510 detects a scene of moving image data, and the key frame extraction unit 520 detects the scene length. A plurality of frame images are extracted as key frames from the longest scene without including frame images corresponding to at least both ends of the scene. By configuring in this way, the key frame extraction apparatus 500 relates to the characteristics of the moving image “The moving image theme is included in a scene having a long scene length” (assuming 3) and “the frame image near the scene cut”. Is not a high-theme image. (Assumption 2) In addition to “there is a small change in composition in the scene” (Assumption 1), the key frame that emphasizes the theme more accurately Can be extracted well.

[Second Modification of First Embodiment]
In the second modified example of the first embodiment, the key frame extraction unit 520 selects a plurality of scenes in the normalized moving image data in order of long scene length, and extracts one frame image that becomes a key frame for each scene. An example will be described. The key frame extraction unit 520 in this modification example extracts key frames in consideration of a plurality of themes in accordance with Assumption 3 and Assumption 2.

  In this modification, a configuration different from the first modification is a key frame extraction unit 520. Since the overall functional configuration in the present modification is the same as that of the first embodiment, in this modification, a block diagram is omitted, and description is made using configurations having the same reference numerals as those in the first embodiment. To do. In this modification, only differences from the first modification will be described.

  The key frame extraction unit 520 takes in the frame identification information and the normalized moving image data supplied from the scene detection unit 512 of the scene analysis unit 510, and M scenes in the normalized moving image data in the order of long scene length (M Is an integer greater than or equal to 2), and for each scene, one frame image serving as a key frame is extracted, and frame identification information corresponding to the extracted M frame images is obtained as a thumbnail generation unit 550 of the control unit 190. To the feature amount analysis unit 530.

  A process in which the key frame extraction unit 520 extracts M key frames will be described with reference to FIG. FIG. 12 is a diagram schematically illustrating how the key frame extraction unit 520 extracts M (M = 3) key frames from the normalized moving image data. FIG. 12A is the same as FIG. 4A in the first embodiment, and the processing by the key frame extraction unit 520 is also the same. Therefore, the processing after the processing corresponding to FIG.

The key frame extraction unit 520 detects M scenes in order from the scene having the longest scene length (the largest number of frames) according to Assumption 3 described above. For example, the key frame extraction unit 520 compares the number of frames of each scene, and in order from the largest number of frames, the third scene (scene 3: 15 frames) and the fourth are shown in FIG. The first scene (scene 4: 12 frames) and the first scene (scene 1: 10 frames) are detected.
Next, the key frame extraction unit 520 detects, for each detected scene, a frame image having a luminance value within a predetermined range from the frame image corresponding to the scene from the normalized moving image data. Specifically, the key frame extraction unit 520 includes a low-luminance frame image having an average luminance value lower than the first threshold value from a frame image corresponding to one detected scene in the normalized moving image data, or A high-luminance frame image whose average luminance value is higher than a second threshold (second threshold> first threshold), or a low-luminance frame image and a high-luminance frame image are detected. FIG. 5B shows four low-luminance frame images and high-luminance frame images detected from the frame image corresponding to the third scene (scene 3), and the frame corresponding to the fourth scene (scene 4). An example is shown in which five low-luminance frame images and high-luminance frame images are detected from the image, and three low-luminance frame images and high-luminance frame images are detected from the frame image corresponding to the first scene (scene 1). ing.

Next, for each detected scene, the key frame extraction unit 520 follows the assumption 2 described above from the low luminance frame image, the high luminance frame image, or the frame image excluding the low luminance frame image and the high luminance frame image. One frame image is extracted as a key frame in the center in the time series or near the center (for example, closest to the center of the time series). FIGS. 12C and 12D show the sixth frame from the left, which is a frame image centered in time series, from among the frame images corresponding to the third scene (scene 3) by the key frame extraction unit 520. An example in which an image is extracted as a key frame is shown. FIGS. 12C and 12D show that the key frame extraction unit 520 is the fourth frame from the left, which is the frame image centered in time series, among the frame images corresponding to the fourth scene (scene 4). In this example, the frame image is extracted as a key frame. FIGS. 12C and 12D show that the key frame extraction unit 520 is the fourth frame from the left, which is the frame image centered in time series, from the frame images corresponding to the first scene (scene 1). In this example, the frame image is extracted as a key frame.
Next, the key frame extraction unit 520 supplies frame identification information corresponding to the extracted M key frames to the thumbnail generation unit 550 of the control unit 190 and also to the feature amount analysis unit 530.

FIG. 13 shows an example of a plurality (M = 3) of key frame images extracted by the key frame extraction unit 520. FIG. 5A shows a key frame image extracted by the key frame extraction device 500 using moving image data obtained by photographing the state of the tennis game by the imaging device 100. The moving image data includes scenes with different compositions and subjects as long time scenes as shown in FIG.
FIG. 5B shows a key frame image extracted by the key frame extraction device 500 using moving image data obtained by photographing the state of the girl making the cookie by the imaging device 100. This moving image data also includes scenes with different compositions and subjects as scenes having a long time as shown in FIG. In the first embodiment and the first modified example, only a scene having a composition in which the hand of the girl cannot be seen from the frame image can be detected. However, according to the present modified example, a scene having a different composition (a scene in which a cookie appears). Etc.) can be detected.
The images of the three key frames in each of FIGS. 9A and 9B are arranged in order of increasing scene length from the left side to the right side.

  FIG. 14 schematically illustrates the feature amount histograms of the three key frame images in FIG. 13A and the integrated histogram generated by the clustering processing unit 533 adding the three feature amount histograms for each class. FIG. As shown in the figure, by adding the feature amount histograms of the images of three key frames having relatively different image characteristics, it is possible to obtain a distribution that covers the characteristic distribution of each scene. In other words, it is possible to grasp without having the theme of the video.

  As described above, in the key frame extraction apparatus 500 that is the second modification of the first embodiment of the present invention, the scene analysis unit 510 detects a scene of moving image data, and the key frame extraction unit 520 detects the scene length. A plurality of scenes are selected in descending order, and one frame image at the center or near the center is extracted as a key frame in time series for each scene. By configuring in this way, the key frame extraction apparatus 500 relates to the characteristics of the moving image “The moving image theme is included in a scene having a long scene length” (assuming 3) and “the frame image near the scene cut”. Is not an image with high theme properties ”(Assumption 2), and key frames considering a plurality of themes can be extracted with high accuracy.

[Third Modification of First Embodiment]
In the third modification example of the first embodiment, the key frame extraction unit 520 selects a plurality of scenes in the normalized moving image data in order of increasing scene length, and extracts a plurality of frame images that become key frames for each scene. An example will be described. The key frame extraction unit 520 in this modification example extracts key frames in consideration of a plurality of themes in accordance with Assumption 3 and Assumption 2.

  In this modification, a configuration different from the second modification is a key frame extraction unit 520. Since the overall functional configuration in the present modification is the same as that of the first embodiment, in this modification, a block diagram is omitted, and description is made using configurations having the same reference numerals as those in the first embodiment. To do. In this modification, only differences from the second modification will be described.

  The key frame extraction unit 520 takes in the frame identification information and the normalized moving image data supplied from the scene detection unit 512 of the scene analysis unit 510, and M scenes in the normalized moving image data in the order of long scene length (M Is an integer of 2 or more), and for each scene, N frame images (N is an integer of 2 or more) that are key frames are extracted, and frame identification information corresponding to these extracted M × N frame images Are supplied to the thumbnail generation unit 550 of the control unit 190 and to the feature amount analysis unit 530.

  A process in which the key frame extraction unit 520 extracts M × N key frames will be described with reference to FIG. FIG. 15 is a diagram schematically illustrating how the key frame extraction unit 520 extracts M × N (M × N = 3 × 3) key frames from the normalized moving image data. FIGS. 12A and 12B are the same as FIGS. 12A and 12B in the second modification, and the processing by the key frame extraction unit 520 is also the same, and therefore corresponds to FIG. The subsequent processing will be described.

For each detected scene, the key frame extraction unit 520 follows the above assumption 2 and assumption 1 from the low luminance frame image, the high luminance frame image, or the frame image excluding the low luminance frame image and the high luminance frame image. N frame images are extracted as key frames at equal intervals. The method for extracting the N key frames is the same as that in the first modification described above, and thus the description thereof is omitted here. In FIGS. 15C and 15D, the key frame extraction unit 520 calculates D = 3 by the expression (1) for the third scene (scene 3), and the third, counting from the end of the 11th frame, The third frame image counted from the position and the third frame image counted from the position are extracted as key frames. Further, in FIGS. 15C and 15D, the key frame extraction unit 520 calculates D = 2 for the fourth scene (scene 4) by the equation (1), and counts 2 from the end of the 7th frame. The frame images corresponding to the second, second from the position, and second from the position are extracted as key frames. Further, in FIGS. 15C and 15D, the key frame extraction unit 520 calculates D = 2 for the first scene (scene 1) by the equation (1), and counts 2 from the end of the 7th frame. The frame images corresponding to the second, second from the position, and second from the position are extracted as key frames.
Next, the key frame extraction unit 520 supplies frame identification information corresponding to the extracted M × N key frames to the thumbnail generation unit 550 of the control unit 190 and also to the feature amount analysis unit 530.

  As described above, in the key frame extraction device 500 which is the third modification of the first embodiment of the present invention, the scene analysis unit 510 detects a scene of moving image data, and the key frame extraction unit 520 detects the scene length. A plurality of scenes are selected in descending order, and for each scene, a plurality of frame images are extracted as key frames without including frame images corresponding to at least both ends of the scene. By configuring in this way, the key frame extraction apparatus 500 relates to the characteristics of the moving image “The moving image theme is included in a scene having a long scene length” (assuming 3) and “the frame image near the scene cut”. Is not a high-theme image. ”(Assumption 2) In addition to“ there is little change in composition in the scene ”(Assumption 1) Keyframes that take into account themes can be extracted with high accuracy.

[Second Embodiment]
FIG. 16 is a block diagram showing an overall configuration of a network system including a server device to which the key frame extraction device according to the second embodiment of the present invention is applied. As shown in the figure, the network system 700 has a configuration in which a server device 560 and information communication terminals 570-1 and 570-2 are connected via a network 580.
In the network system 700, the information communication terminals 570-1 and 570-2 used by the terminal user download the desired moving image data from the server device 560 storing a large number of moving image data to the terminal user. This is a system that allows browsing.

The server device 560 includes a moving image data storage unit 561, a key frame extraction device 500, a thumbnail generation unit 562, and a communication processing unit 563 as functional configurations.
The moving image data storage unit 561 is a storage device that stores moving image data, and is realized by, for example, a magnetic hard disk device.
The key frame extraction apparatus 500 is an apparatus according to any one of the first embodiment and the first to third modifications described above.

  The thumbnail generation unit 562 takes in the frame identification information and the classification data output from the key frame extraction device 500. Then, the thumbnail generation unit 562 extracts image data corresponding to the frame identification information from the moving image data storage unit 561, reduces the extracted image data, and generates thumbnail image data. Then, the thumbnail generation unit 562 associates the generated thumbnail image data with the corresponding moving image data. Then, the thumbnail generation unit 562 classifies and stores the moving image data associated with the thumbnail image data in the moving image data storage unit 561 based on the classification data.

  The communication processing unit 563 includes a network interface function, and controls communication with the information communication terminals 570-1 and 570-2 via the network 580. When receiving the download request from the information communication terminals 570-1 and 570-2, the communication processing unit 563 reads out the moving image data associated with the thumbnail image data from the moving image data storage unit 561 according to the download request. To the requesting information communication terminals 570-1 and 570-2.

The information communication terminals 570-1 and 570-2 are devices capable of communicating with a connection device for the network 580, and are realized by, for example, a computer device, a mobile phone, a smartphone, a personal digital assistant (PDA), or the like. Is done.
The information communication terminals 570-1 and 570-2 request and acquire a list of moving image data from the server device 560 and display it on a display device (not shown). The display content is such that thumbnail images are displayed in a list of moving image data as shown in the first embodiment and the respective modifications.
The network 580 is a computer network such as the Internet or a LAN (Local Area Network).

  According to the server device 560 according to the second embodiment of the present invention, the moving image data to be held is visually applied by applying the key frame and / or classification data provided by the key frame extraction device 500. It is easy to see and manage easily.

  In the first embodiment, the first to third modifications, and the second embodiment described above, the scene analysis unit 510 may be realized by applying a known scene detection technique. For example, the scene analysis unit 510 may detect a scene by a technique for detecting frame correlation by pattern matching. In addition, the scene analysis unit 510 may detect a scene by extracting and comparing all or some of the spatial frequencies of the image data.

  Moreover, you may make it implement | achieve a part of function of the key frame extraction apparatus which is embodiment mentioned above with a computer. In this case, the key frame extraction program for realizing the function is recorded on a computer-readable recording medium, and the key frame extraction program recorded on the recording medium is read by the computer system and executed. May be. Here, the “computer system” includes an OS (Operating System) and peripheral device hardware. The “computer-readable recording medium” refers to a portable recording medium such as a flexible disk, a magneto-optical disk, an optical disk, and a memory card, and a storage device such as a magnetic hard disk built in the computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, it may include a device that holds a program for a certain period of time, such as a volatile memory inside a computer system serving as a server device or a client. Further, the above program may be for realizing a part of the functions described above, or may be realized by a combination with the program already recorded in the computer system. .

  As mentioned above, although embodiment of this invention was explained in full detail with reference to drawings, the specific structure is not restricted to that embodiment, The design of the range which does not deviate from the summary of this invention, etc. are included.

DESCRIPTION OF SYMBOLS 100 Image pick-up device 110 Image pick-up part 111 Optical system 119 Image pick-up element 120 Analog / digital (A / D) conversion part 130 Buffer memory part 140 Image processing part 150 Display part 160 Storage part 170 Communication part 180 Operation part 190 Control part 200 Storage medium 300 Bus 500 Key frame extraction device 510 Scene analysis unit 511 Spatio-temporal image generation unit 512 Scene detection unit 520 Key frame extraction unit 530 Feature amount analysis unit 531 Image normalization unit 532 Feature amount extraction unit 533 Clustering processing unit 540 Classification processing unit 550 Thumbnail Generation unit 560 Server device 561 Moving image data storage unit 562 Thumbnail generation unit 563 Communication processing unit 570-1, 570-2 Information communication terminal 580 Network 700 Network system

Claims (12)

A scene analysis unit that analyzes moving image data and detects a scene;
A key frame extraction unit that extracts a key frame based on a scene length of the scene detected by the scene analysis unit and a position of image data in a time direction in a plurality of image data corresponding to the scene;
A key frame extraction device comprising: The key frame extraction unit
The center or near-center image data is extracted as a key frame in time series from a plurality of image data corresponding to a scene having the longest scene length among a plurality of scenes detected by the scene analysis unit. Item 2. A key frame extraction device according to Item 1. The key frame extraction unit
The plurality of image data is extracted as a key frame from a plurality of image data corresponding to the scene having the longest scene length without including image data corresponding to at least both ends of the scene. Key frame extractor. The key frame extraction unit
Among the plurality of scenes detected by the scene analysis unit, a plurality of scenes are selected in descending order of the scene length, and for each selected scene, from the plurality of image data corresponding to the scene, the center or the vicinity of the center is selected in time series. 2. The key frame extracting apparatus according to claim 1, wherein the image data is extracted as a key frame. The key frame extraction unit
5. The plurality of image data is extracted as a key frame for each selected scene from a plurality of image data corresponding to the scene without including image data corresponding to at least both ends of the scene. The key frame extraction device described. The key frame extraction unit
6. A plurality of image data having a predetermined range of luminance values is extracted from a plurality of image data corresponding to the scene, and key frames are extracted from the extracted plurality of image data. A key frame extraction device according to claim 1. A feature amount extraction unit that extracts a feature amount of an image from the key frame extracted by the key frame extraction unit;
A clustering processing unit that generates a feature amount histogram by performing clustering processing based on the feature amount of the image extracted by the feature amount extraction unit;
The key frame extraction device according to claim 1, further comprising: A feature amount extraction unit that extracts a feature amount of an image from each of a plurality of key frames extracted by the key frame extraction unit;
Based on the feature amount of the image extracted by the feature amount extraction unit, a clustering process is performed for each key frame to generate a plurality of feature amount histograms, and the plurality of feature amount histograms are added for each class to obtain an integrated histogram A clustering processing unit for generating
The key frame extraction device according to claim 1, further comprising: Computer
A scene analysis unit that analyzes moving image data and detects a scene;
A key frame extraction unit that extracts a key frame based on a scene length of the scene detected by the scene analysis unit and a position of image data in a time direction in a plurality of image data corresponding to the scene;
Key frame extraction program to function as A scene analysis step in which the scene analysis unit detects the scene by analyzing the moving image data;
The key frame extraction unit extracts a key frame based on the scene length of the scene detected by the scene analysis unit in the scene analysis step and the position of the image data in the time direction in the plurality of image data corresponding to the scene. A key frame extraction step;
A key frame extraction method comprising: An imaging unit for capturing and generating moving image data;
A scene analysis unit for analyzing the moving image data generated by the imaging unit and detecting a scene;
A key frame extraction unit that extracts a key frame based on a scene length of the scene detected by the scene analysis unit and a position of image data in a time direction in a plurality of image data corresponding to the scene;
A thumbnail generation unit that generates reduced image data based on the key frame extracted by the key frame extraction unit;
A display unit for displaying the reduced image data generated by the thumbnail generation unit;
An imaging apparatus comprising: A moving image data storage unit for storing moving image data;
A scene analysis unit for detecting a scene by analyzing the moving image data stored in the moving image data storage unit;
A key frame extraction unit that extracts a key frame based on a scene length of the scene detected by the scene analysis unit and a position of image data in a time direction in a plurality of image data corresponding to the scene;
A thumbnail generation unit that generates reduced image data based on the key frame extracted by the key frame extraction unit, and stores the reduced image data in the moving image data storage unit in association with the moving image data;
A server device comprising:

Images