JP2012160011A - Thumbnail extraction program and thumbnail extraction method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a thumbnail extraction program and a thumbnail extraction method that are capable of extracting a thumbnail suitable for three-dimensional video.SOLUTION: A thumbnail extraction program causes a computer to execute processing for: extracting feature points corresponding between a frame image pair constituting one frame of 3D video from the frame image pair as a feature point pair; calculating a depth amount of the feature point pair based on a distance between points of the feature point pair, a distance between lenses, and a focal length, and clustering the feature point pair using the depth amount; evaluating stereoscopic visibility of one or more frame image pairs from clustering result information of the one or more frame image pairs on the basis of a predetermined evaluation condition of stereoscopic visibility of a frame image pair based on clustering result information; and extracting a pair of the frame image with the most excellent evaluation result as a pair of the frame image for a thumbnail based on the evaluation result of stereoscopic visibility.

Description

  The present invention relates to a thumbnail extraction program and a thumbnail extraction method for extracting thumbnails from video.

  In recent years, a large amount of video has been accumulated in mobile phones and video cameras. There is a demand for searching for necessary images in a mobile phone, a video camera, or the like in which a large amount of images is stored in this way.

  Conventionally, when searching for a video, for example, a search method using metadata added to the video is used. However, there are many cases where the video cannot be narrowed down by a method of searching with metadata added to the video. After all, when searching for video, there are many cases where confirmation is made while playing back one by one.

  Conventionally, when searching for a video, a method of grasping the content of the video by displaying a thumbnail extracted from the video is also used. A thumbnail is an image that allows the user to grasp the content of the video. When thumbnails are used for video search, since the number of thumbnails per video cannot be increased, a better thumbnail extraction method is required (see, for example, Patent Document 1).

  Other thumbnail extraction methods include a method of extracting thumbnails from the first frame of a video or a frame after a predetermined number of seconds from the top of the video, or a thumbnail from a portion other than the CM in combination with a CM detection method. There are a method, a method of detecting a cut that is a video change and extracting the cut image as a thumbnail, and a method of extracting an up image of a face as a thumbnail in combination with a face detection method.

JP 2005-294904 A

  Currently, products that can handle three-dimensional (3D) images have begun to appear in mobile phones and video cameras. Therefore, it is conceivable that a large amount of 3D images will be accumulated in the future in mobile phones, video cameras and the like. In this way, it is considered that there is an increasing demand for searching for a necessary 3D video in a mobile phone or a video camera in which a large amount of 3D video is accumulated.

  However, it is considered that the problem in searching for a two-dimensional (2D) video as described above also occurs when searching for a 3D video. That is, even when searching for a 3D video, it is considered that a method of grasping the content of the video by displaying thumbnails extracted from the 3D video is used.

  However, the conventional thumbnail extraction method for 2D video has a problem that thumbnails suitable for 3D video cannot be extracted.

  An object of the present embodiment is to provide a thumbnail extraction program and a thumbnail extraction method capable of extracting thumbnails suitable for 3D video.

  In order to solve the above-described problem, the present embodiment extracts a feature point corresponding to a pair of frame images constituting one frame of 3D video as a feature point pair from the frame image pair, and calculates the depth amount of the feature point pair. The frame image pair is calculated based on a distance between points of the feature point pair, a predetermined inter-lens distance / focal length, and clustering the feature point pairs based on the depth amount, and based on predetermined clustering result information The stereoscopic visibility of the one or more frame image pairs is evaluated from the result information of the clustering of the one or more frame image pairs based on the evaluation condition of the stereoscopic visibility of the one or more frames, and the one or more frames Based on the evaluation result of the three-dimensional visibility of the image pair, the frame image pair having the best evaluation result is extracted as the frame image pair for the thumbnail. A thumbnail extraction program for executing the processing to computers.

  In addition, what applied the arbitrary combination of the component of this embodiment, expression, or a component to a method, an apparatus, a system, a computer program, a recording medium, a data structure, etc. is also effective as an aspect of this invention.

  According to the present embodiment, it is possible to provide a thumbnail extraction program and a thumbnail extraction method capable of extracting thumbnails suitable for 3D video.

It is a hardware block diagram of an example of PC. It is a functional block diagram of an example of the thumbnail extraction apparatus of a present Example. It is a flowchart of an example of the thumbnail extraction apparatus of a present Example. It is a block diagram of an example of a video information table. It is a block diagram of an example of a frame information table. It is an image figure of an example of two pictures which constitute one frame of 3D video. It is a block diagram of an example of a local feature-value table. It is an image figure of an example of the frame image which expressed the extracted feature point visually. It is a block diagram of an example of a corresponding point table. It is a block diagram of the other example of a corresponding point table. It is a block diagram of an example of a Z value histogram table. It is a block diagram of an example of a corresponding point histogram table. It is a block diagram of an example of a Z value cluster table. It is a block diagram of an example of a corresponding point cluster table. It is a block diagram of an example of a frame evaluation table. It is a schematic diagram of an example at the time of seeing a 3D camera from the top. It is an image figure of an example of the calculation method of Z value by parallax. It is a flowchart of an example of a process of step S3. It is an example histogram showing the value of each bin calculated. It is an image figure of an example of the frame picture which expressed the corresponding point contained in each cluster visually. It is a flowchart of an example of a process of step S4. It is a flowchart of an example of a process of step S5. It is a flowchart of an example of a process of step S28. It is a functional block diagram of the other example of the thumbnail extraction apparatus of a present Example. It is a flowchart of the other example of a process of step S5.

  Next, modes for carrying out the present invention will be described based on the following embodiments with reference to the drawings. The thumbnail extraction program of this embodiment is executed by a personal computer (PC) and various devices that handle 3D video such as a mobile phone and a video camera. In this embodiment, an example in which a thumbnail extraction program is executed on a PC will be described. The PC is configured by, for example, hardware shown in FIG. FIG. 1 is a hardware configuration diagram of an example of a PC.

  1 includes an input device 21, a display device 22, and a PC main body 23. The PC main body 23 includes a main storage device 31, an arithmetic processing device 32, an interface device 33, a recording medium reading device 34, and an auxiliary storage device 35 connected to each other via a bus 37. The input device 21 and the display device 22 are connected to the bus 37.

  The input device 21, the display device 22, the main storage device 31, the arithmetic processing device 32, the interface device 33, the recording medium reading device 34, and the auxiliary storage device 35 connected to each other via the bus 37 are managed by the arithmetic processing device 32. Data can be sent and received between each other. The arithmetic processing unit 32 is a central processing unit that controls operation of the entire PC 10.

  The interface device 33 receives data from a network or the like and passes the contents of the data to the arithmetic processing device 32. The interface device 33 transmits data to a network or the like in response to an instruction from the arithmetic processing device 32.

  The auxiliary storage device 35 stores a thumbnail extraction program that causes the PC 10 to execute at least processing in the thumbnail extraction device as part of a program that causes the PC 10 to perform the same function as the thumbnail extraction device. Then, when the arithmetic processing device 32 reads out and executes the thumbnail extraction program from the auxiliary storage device 35, the PC 10 functions as a thumbnail extraction device. The thumbnail extraction program may be stored in the main storage device 31 accessible to the arithmetic processing device 32. The input device 21 receives data input under the control of the arithmetic processing device 32. The thumbnail extraction program can be recorded in a recording medium 36 that can be read by the PC 10.

  Examples of the recording medium 36 include a magnetic recording medium, an optical disk, a magneto-optical recording medium, and a semiconductor memory. Magnetic recording media include HDDs, flexible disks (FD), magnetic tapes (MT) and the like. Examples of the optical disc include a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc-Read Only Memory), and a CD-R (Recordable) / RW (ReWriteable). Magneto-optical recording media include MO (Magneto-Optical disk).

  When distributing the thumbnail extraction program, it is conceivable to sell a portable recording medium 36 such as a DVD or CD-ROM in which the thumbnail extraction program is recorded. For example, the PC 10 executing the thumbnail extraction program reads the thumbnail extraction program from the recording medium 36 on which the recording medium reader 34 has recorded the thumbnail extraction program. The arithmetic processing device 32 stores the read thumbnail extraction program in the main storage device 31 or the auxiliary storage device 35.

  The PC 10 reads the thumbnail extraction program from the main storage device 31 or the auxiliary storage device 35 which is its own storage device, and executes processing according to the thumbnail extraction program. The arithmetic processing unit 32 implements various processes as described later according to the thumbnail extraction program.

  Hereinafter, at least the thumbnail extraction apparatus 40 in which the thumbnail extraction program is installed in the PC 10 will be described as an example.

  FIG. 2 is a functional block diagram of an example of the thumbnail extraction apparatus according to the present embodiment. 2 includes a frame image acquisition unit 41, a feature point extraction unit 42, a local feature extraction unit 43, a feature point pair extraction unit 44, a depth clustering unit 45, a thumbnail evaluation unit 46, a thumbnail extraction unit 47, and a video. An information table 51, a frame information table 52, a local feature table 53, a corresponding point table 54, a Z value histogram table 55, a corresponding point histogram table 56, a Z value cluster table 57, a corresponding point cluster table 58, and a frame evaluation table 59. .

  Here, an outline of the functional block of FIG. 2 will be described. The frame image acquisition unit 41 acquires two images constituting one frame of 3D video. The feature point extraction unit 42 extracts feature points from the image. The local feature extraction unit 43 extracts local feature amounts of feature points from the image. The feature point pair extraction unit 44 extracts a pair of feature points of two images constituting one frame of 3D video.

  The depth clustering unit 45 clusters the feature points by the depth distance calculated from the feature point pairs. The thumbnail evaluation unit 46 calculates a three-dimensional value (evaluation value) of the frame based on the clustering result. The thumbnail extracting unit 47 extracts a frame having the highest frame solidity value as a thumbnail frame.

  The video information table 51 stores 3D video information. The frame information table 52 associates and stores two images constituting one frame of 3D video. The local feature quantity table 53 stores the local feature quantities of feature points extracted from two images constituting one frame of 3D video.

  The corresponding point table 54 stores a pair of feature points extracted from two images constituting one frame of the 3D video as corresponding points, and stores a distance value of the corresponding points. The distance value is a distance between a pair of feature points when two images constituting one frame of 3D video are superimposed. The corresponding point table 54 may store the corresponding points and the depth distances (Z values) of the corresponding points in addition to the distance values of the corresponding points.

  The Z value histogram table 55 stores the number of corresponding points in a plurality of Z value ranges. The corresponding point histogram table 56 stores the corresponding points in association with the Z value range in which the corresponding points are stored.

  The Z value cluster table 57 stores a cluster that combines the peaks of the Z value histogram and the range of Z values before and after the peak. The corresponding point cluster table 58 stores the corresponding points in association with the clusters in which the corresponding points are stored. Further, the frame evaluation table 59 stores the solidity value (evaluation value) of each frame.

  FIG. 3 is a flowchart of an example of the thumbnail extraction apparatus according to this embodiment. In step S1, the frame image acquisition unit 41 acquires a 3D video image (frame image). Each frame of 3D video is composed of two images (frame images) viewed from a point slightly shifted in position. In most cases, the displacement of the position of the frame image in the 3D video is not so large. The frame images in the 3D video are considerably similar images.

  There are no particular restrictions on the 3D video of the acquisition source. The 3D video may be a TV video, a video captured live with a 3D camera, or a video stored in a file such as Blu-ray 3D. There is no particular restriction on the 3D video storage method. As a 3D video storage method, Side-by-Side (left and right images are stored side by side and stored as a single image), Frame alternative (right and left images are stored alternately), and the like can be used.

  For 3D video, video information such as the width and height of the video and the number of frames is made available. Such video information can be stored in the form of a video information table 51. FIG. 4 is a configuration diagram of an example of the video information table. The video information table 51 of FIG. 4 has video numbers, widths, heights, and frame numbers as data items.

  The video number is a 3D video identifier. The width is the width of the 3D video. The height is the height of the 3D video. The number of frames is the number of 3D video frames. Note that the number of frames may not be acquired with live video. In that case, it is possible not to use the number of frames. For example, if the number of frames cannot be acquired, processing can be performed without problems by extracting thumbnails from frames within a specified frame from the beginning.

  Since the subsequent processing is independent for each 3D video, the processing for one 3D video is described. The frame image acquisition unit 41 does not necessarily need to extract all the frames of the 3D video, and may extract a part of the frames of the 3D video. The frame image acquisition unit 41 extracts an image in combination with a cut detection method, a method for extracting images at regular intervals, a CM detection method, a face detection method, or another analysis method. And so on.

  The frame image acquisition unit 41 stores the acquired frame image in the frame information table 52. FIG. 5 is a configuration diagram of an example of the frame information table. The frame information table 52 in FIG. 5 has a frame number, a left image, and a right image as data items.

  The frame number is a frame identifier. The left image is an identifier of a left image (left frame image) constituting the frame. The right image is an identifier of the right image (right frame image) constituting the frame.

  FIG. 6 is an image diagram of an example of two images constituting one frame of 3D video. As shown in FIG. 6, the left frame image 100 and the right frame image 101 are quite similar images.

  In step S2 of FIG. 3, the feature point extraction unit 42 extracts feature points from the left frame image 100 and the right frame image 101 of one frame in the frame information table 52 acquired in step S1, respectively. Further, the local feature extraction unit 43 extracts the feature quantity of the extracted feature points as a local feature quantity.

  The feature point extraction method is not particularly limited. As simple feature point extraction methods, there are various edge detection methods such as a Sobel edge extraction method and a Canny edge extraction method, and corner detection methods such as a Harris corner method. The Sobel edge extraction method is described in, for example, “Mikio Takagi, Yoshihisa Shimoda: New Image Analysis Handbook, University of Tokyo Press, 2004”. The Canny edge extraction method is described in, for example, “CANNY J.,“ A Computational Approach to Edge Detection, ”IEEE Trans. On Pattern Analysis and Machine Intelligence, 8 (6), pp. 679-698, 1986”. The Harris corner method is described in, for example, “C. Harris, M. Stephens,“ A COMBINED CORNER AND EDGE DETECTOR, ”Proc. Of 4th Alvey vision Conference, pp. 147-151, 1988”.

  Further, the local feature amount extraction method is not particularly limited. It is conceivable that the local feature amount is extracted from an image near the feature point. As a local feature extraction method, a method such as a color histogram can be used. The method of the color histogram is described in, for example, “Mikio Takagi, Yoshihisa Shimoda: New Image Analysis Handbook, University of Tokyo Press, 2004”.

  Further, as a feature point extraction method and a local feature quantity extraction method, a method using SIFT (US Patent 6,711,293) or the like is also conceivable. SIFT is described in, for example, “D.G. Lowe,“ Object recognition from local scale-invariant features, ”Proc. Of IEEE Int. Conf. On. Computer Vision (ICCV) pp. 1150-1157, 1999.

  In SIFT, the difference (Difference of Gaussian) when Gaussian blurring is applied to an image is calculated, and the local maximum is extracted from the maximum point as a feature point. As a similar local feature extraction method, SURF or the like can be used. SURF is described in, for example, “BAY H.,“ SURF: Speeded up robust features, ”Proc. 9th ECCV, May 2006, Graz, Austria 1, 404-417, 2006”. The extracted feature quantity (local feature quantity) is in the form of a real value array (vector).

  The feature point extraction unit 42 and the local feature extraction unit 43 store the extracted feature points and local feature amounts in the local feature amount table 53. FIG. 7 is a configuration diagram of an example of the local feature table. The local feature quantity table 53 of FIG. 7 has a frame number, an image number, a feature quantity number, an X coordinate, a Y coordinate, and a feature quantity as data items.

  The frame number is a frame identifier. The image number is an identifier between the right frame image and the left frame image. The feature quantity number is an identifier of a feature point. The X coordinate is the X coordinate of the feature point. The Y coordinate is the Y coordinate of the feature point. The feature amount is a feature amount of a feature point.

  FIG. 8 is an image diagram of an example of a frame image that visually represents the extracted feature points. In FIG. 8, the feature points are represented by “◯”. As shown in FIG. 8, a plurality of feature points are extracted from the left frame image 100 and the right frame image 101. Note that the feature points represented on the left frame image 100 and the right frame image 101 correspond to the local feature table 53 of FIG.

  In step S3 of FIG. 3, the feature point pair extraction unit 44 refers to the local feature amount table 53 and extracts a feature point pair from the right frame image and the left frame image of the same frame as described later. The feature point pair extraction unit 44 stores the extracted feature point pairs in the corresponding point table 54 as corresponding points.

  FIG. 9 is a configuration diagram of an example of the corresponding point table. The corresponding point table 54 in FIG. 9 includes data items such as a frame number, a corresponding point number, a feature point number of the right frame image 101, a feature point number of the left frame image 100, and a distance value. The frame number is a frame identifier. The corresponding point number is an identifier of the corresponding point. The feature point number of the right frame image 101 is an identifier of the feature point of the right frame image 101. The feature point number of the left frame image 100 is an identifier of the feature point of the left frame image 100. The distance value includes a feature point identified by the feature point number of the right frame image 101 and a feature point identified by the feature point number of the left frame image 100 when the left frame image 100 and the right frame image 101 are superimposed. Is the distance between.

  FIG. 10 is a configuration diagram of another example of the corresponding point table. In the corresponding point table 54 of FIG. 10, a Z value is added to the data item of the corresponding point table 54 of FIG. The Z value is calculated from the X coordinate stored in the local feature table 53 and the distance value stored in the corresponding point table 54 as described later. The Z value represents the depth distance of the corresponding point.

  In step S4 in FIG. 3, the depth clustering unit 45 performs a clustering process described later based on the distance value stored in the corresponding point table 54 in FIG. 9 or the Z value stored in the corresponding point table 54 in FIG. Create a group of corresponding points of close depth distance. The depth clustering unit 45 displays the result of the clustering process as a Z value histogram table 55 shown in FIG. 11, a corresponding point histogram table 56 shown in FIG. 12, a Z value cluster table 57 shown in FIG. 13, and a corresponding point cluster table 58 shown in FIG. Save to.

  FIG. 11 is a configuration diagram of an example of a Z value histogram table. The Z value histogram table 55 in FIG. 11 has a frame number, a bin number, a start Z value, an end Z value, and the number as data items. The frame number is a frame identifier. The bin number is an identifier of a range of Z values (histogram bins). The starting Z value is the starting Z value of the bin. The end Z value is the end Z value of the bin. The number is the number of corresponding points in the bin.

  FIG. 12 is a configuration diagram of an example of the corresponding point histogram table. The corresponding point histogram table 56 of FIG. 12 has a frame number, a corresponding point number, and a bin number as data items. The frame number is a frame identifier. The corresponding point number is an identifier of the corresponding point. The bin number is an identifier of the bin.

  FIG. 13 is a configuration diagram of an example of the Z value cluster table. The Z value cluster table 57 of FIG. 13 has a frame number, a cluster number, a start Z value, an end Z value, and the number as data items. The frame number is a frame identifier. The cluster number is a cluster identifier. The starting Z value is the starting Z value of the cluster. The end Z value is the end Z value of the cluster. The number is the number of corresponding points in the cluster.

  FIG. 14 is a configuration diagram of an example of the corresponding point cluster table. The corresponding point cluster table 58 of FIG. 14 has a frame number, a corresponding point number, and a cluster number as data items. The frame number is a frame identifier. The corresponding point number is an identifier of the corresponding point. The cluster number is a cluster identifier.

  In step S5 of FIG. 3, the thumbnail evaluation unit 46 calculates a value (evaluation value) of the three-dimensionality of the frame based on the result of clustering, as will be described later. That is, the thumbnail evaluation unit 46 calculates an evaluation value as to whether the current frame is suitable as a thumbnail.

  In step S6, the thumbnail evaluation unit 46 stores the calculated three-dimensional value (evaluation value) of the frame in the frame evaluation table 59 shown in FIG. FIG. 15 is a configuration diagram of an example of a frame evaluation table. The frame evaluation table 59 in FIG. 15 has a frame number and an evaluation value as data items. The frame number is a frame identifier. The evaluation value is a value of the solidness of the frame.

  In step S7, the thumbnail evaluation unit 46 determines whether there is another frame to be analyzed (a value of solidness should be calculated). If it is determined that there are other frames to be analyzed, the thumbnail evaluation unit 46 requests the frame image acquisition unit 41 to acquire a frame image. Steps S1 to S6 are repeated for the number of frames to be analyzed.

  If it is determined that there are no other frames to be analyzed, the thumbnail evaluation unit 46 requests the thumbnail extraction unit 47 to extract thumbnail frames. In step S <b> 8, the thumbnail extracting unit 47 extracts the frame having the highest three-dimensionality value (evaluation value) of the frame in the frame evaluation table 59 as a thumbnail frame. Note that the thumbnail extraction unit 47 may perform weighting based on the thumbnail extraction position (first, last, etc. of 3D video) depending on the type of 3D video, for example.

  The details of the process of step S3 in FIG. 3 are as follows. The feature point pair extraction unit 44 associates the feature amounts extracted from the right frame image and the left frame image of the same frame in the local feature amount table 53, respectively.

When the feature quantity number set of the right frame image is R and the feature quantity number set of the left frame image is L, there is a problem of finding i and j that satisfy the following expression (1). Here, f _R (i) is the i-th feature amount of the right frame image. f _L (j) is the j-th feature amount of the left frame image. “dist” is a distance function between two feature quantities.

  That is, the feature point pair extraction unit 44 calculates the distance between the feature amounts for all pairs between the feature amount of the right frame image and the feature amount of the left frame image, and sets the minimum distance pair (i, j ) At the same time, the feature point pair extraction unit 44 also holds the distance between the feature amounts of the minimum pair (return value of the dist function). When the distance between the feature amounts is equal to or less than a predetermined distance threshold value, it is stored in the corresponding point table 54 as a corresponding point.

  Further, in Expression (1), all the matching between the feature amounts of the left frame image and the right frame image is performed. However, only a certain distance pair may be obtained for speeding up.

For example, the X coordinate of the i th feature point of the right frame image is x _R (i), the X coordinate of the j th feature point of the left frame image is x _L (j), and the i th feature point of the right frame image. Is set to y _R (i), the Y coordinate of the j-th feature point of the left frame image is set to y _L (j), and T _x and T _y are set as thresholds, the following equation (2) Only those that meet

In general, the lenses of a 3D image camera are arranged side by side. Accordingly, since the difference in the X coordinate is larger in the 3D image, a value larger than T _y is designated for T _x . As the distance function, a general technique such as Euclidean distance can be used. A distance function suitable for each local feature may be used.

When the Euclidean distance is used, the following equation (3) is obtained. Expression (3) is an example in which the values in the case of the number of dimensions i are v ₁ (i) and v ₂ (j), respectively, and the number of dimensions is n in the two vectors v ₁ and v ₂ . is there. The feature point pair extraction unit 44 stores the feature quantity number (i, j) of the extracted left and right frame images of the corresponding points and the distance value of the extracted corresponding points in the corresponding point table 54.

  When the corresponding point table shown in FIG. 10 is used, the feature point pair extraction unit 44 is stored as a corresponding point from the X coordinate stored in the local feature table 53 and the distance value stored in the corresponding point table 54. The Z value (depth distance) of the feature point is calculated. The Z value can be calculated using the following formula (4).

Here, f _C is a focal length which is a parameter of the camera at the time of capturing a 3D image. X _C is the distance between the two lenses of the cameras taking a 3D image. x _R is the X-coordinate of the right frame image of the corresponding points. x _L is the X coordinate in the left frame image of the corresponding point. It is assumed that the focal length has the same value for the two lenses. In addition, when each parameter in Formula (4) is illustrated, it will become as shown, for example in FIG. FIG. 16 is a schematic diagram of an example when the 3D camera is viewed from above.

  FIG. 17 is an image diagram illustrating an example of a Z value calculation method based on parallax. FIG. 17A shows an image of parallax between two lenses. FIG. 17B shows an image of corresponding points (feature point pairs) extracted from the right frame image and the left frame image in the same frame. In FIG. 17B, feature points are represented by “◯”, and paired feature points are visually represented by connecting them with lines. In FIG. 17B, pairs of feature points having a small Z value are connected by thin lines, and pairs of feature points having a large Z value are connected by thick lines.

  As shown in FIG. 17, the closer the position is to the lens, the more the left and right positions of the feature points are shifted between the left and right frame images. Therefore, the calculation of the depth distance can be performed using the shift of the left and right position of the feature point between the left and right frame images.

  In the processing after step S3 in FIG. 3, the absolute value of the Z value is not used, but the relative value is used. Therefore, a positive fixed value may be set for the focal length and the distance between the two lenses. Further, as can be seen from the above equation (4), the Z value and the distance value of the corresponding point are in an inversely proportional relationship. If the distance value of the corresponding point is used in the process of step S4, it is not necessary to perform the calculation of the Z value in the process of step S3. The feature point pair extraction unit 44 stores the calculated Z value in the corresponding point table 54.

  The process of step S3 in FIG. 3 can be performed, for example, according to the procedure of the flowchart shown in FIG. FIG. 18 is a flowchart of an example of the process in step S3.

  In step S11, the feature point pair extraction unit 44 selects one feature point of the left frame image stored in the local feature amount table 53, and acquires the feature amount of the feature point. In step S12, the feature point pair extraction unit 44 selects a feature point having a coordinate value close to that of the left frame image selected in step S11, and acquires a feature amount of the feature point.

  In step S13, the feature point pair extraction unit 44 calculates the distance value between the feature amounts of the two feature points selected in steps S11 and S12. In step S14, when the calculated distance value is the minimum value, the feature point pair extraction unit 44 stores the distance values of the feature values of the two feature points selected in steps S11 and S12 and the two feature points calculated in step S13. To do.

  In step S15, the feature point pair extraction unit 44 determines whether there are other feature points of the right frame image. If there are other feature points of the right frame image, the feature point pair extraction unit 44 repeats the processes of steps S12 to S14 for the other feature points of the right frame image. If there are no more feature points in the right frame image, the feature point pair extraction unit 44 calculates the difference between the X coordinates and the Z value of the two feature points having the minimum feature value distance value in step S16.

  In step S <b> 17, when the feature value distance value is equal to or smaller than the threshold value, the feature point pair extraction unit 44 stores the two feature points having the minimum feature value distance value as corresponding points in the corresponding point table 54. In step S18, the feature point pair extraction unit 44 determines whether there are other feature points of the left frame image.

  If there are other feature points of the left frame image, the feature point pair extraction unit 44 repeats the processes of steps S11 to S17 for the other feature points of the left frame image. If there are no more feature points in the left frame image, the feature point pair extraction unit 44 ends the process shown in FIG.

  The details of the process of step S4 in FIG. 3 are as follows. The depth clustering unit 45 performs a clustering process based on the Z value or the distance value obtained in the process of step S3, and creates a group of close depth distances.

  As a clustering method, a method using a histogram can be used. The depth clustering unit 45 defines a range of Z values (histogram bins) at specific intervals. The depth clustering unit 45 first initializes the value of each bin to 0. The depth clustering unit 45 refers to the corresponding point table 54 and calculates the number of corresponding points in the range of the Z value of each bin as the value of each bin. The depth clustering unit 45 stores the calculated bin values in the Z value histogram table 55.

  FIG. 19 is an example histogram showing the calculated value of each bin. The histogram in FIG. 19 corresponds to the image diagram shown in FIG. Note that the range of the Z value may be a fixed interval or may be changed by the absolute value of the Z value. According to the above equation (4), the Z value increases rapidly when the distance value of the corresponding point is close to zero. Therefore, the bin may be set to have a wider range of the portion with a large Z value. The depth clustering unit 45 stores, in the corresponding point histogram table 55, which corresponding points are stored in each histogram as necessary.

  The depth clustering unit 45 finds the peak of the histogram stored in the Z value histogram table 55 (bin having a locally large value), and combines it with the number of bins before and after that to form one cluster.

  When the number of bin numbers i in the histogram is h (i), the depth clustering unit 45 selects bins with bin numbers i that satisfy h (i)> h (i−1) and h (i)> h (i + 1). Let it be a peak. The depth clustering unit 45 combines the bin with the bin number i and the bins with bin numbers i−1 and i + 1 before and after that into one cluster. The depth clustering unit 45 sets the number of corresponding points in the cluster as c = h (i−1) + h (i) + h (i + 1). The depth clustering unit 45 stores the number of corresponding points in the cluster in the Z value cluster table 57. The depth clustering unit 45 stores in the corresponding point cluster table 58 which corresponding points are included in each cluster based on the corresponding point histogram table 56 as necessary.

  FIG. 20 is an image diagram of an example of a frame image that visually represents corresponding points included in each cluster. In FIG. 20, corresponding points included in the same cluster are surrounded by a line. The image diagram of FIG. 20 corresponds to the histogram shown in FIG.

  In order to improve accuracy in obtaining the histogram, a method of removing feature points that do not have an approximate Z value by comparing with the Z values of neighboring feature points is conceivable. For example, the depth clustering unit 45 acquires the X coordinate value and the Y coordinate value of the feature point from the corresponding point cluster table 58, the corresponding point table 54, and the local feature amount table 53.

  The coordinates of either the right frame image or the left frame image may be used, but the coordinate values of the same frame image are designated for all feature points. The depth clustering unit 45 extracts the difference between the X coordinate value and the Y coordinate value between the feature points, and if there is no other feature point whose distance is less than or equal to a predetermined threshold, the feature point is extracted from the cluster. exclude.

  Further, when the number of feature points constituting the cluster is small, the depth clustering unit 45 deletes the cluster itself. For example, the depth clustering unit 45 gives a threshold value in advance, and deletes the cluster when the number of feature points in the cluster is equal to or less than the threshold value.

  The process of step S4 in FIG. 3 can be performed, for example, according to the procedure of the flowchart shown in FIG. FIG. 21 is a flowchart of an example of the process in step S4.

  In step S21, the depth clustering unit 45 initializes the value of each bin of the Z value histogram table 55 with zero. In step S22, the depth clustering unit 45 refers to the corresponding point table 54, and acquires the corresponding point and its Z value. In step S23, the depth clustering unit 45 adds 1 to the bin value of the Z value histogram table 55 that matches the Z value.

  In step S24, the depth clustering unit 45 refers to the corresponding point table 54 and determines whether there is another corresponding point. If there is another corresponding point, the depth clustering unit 45 repeats the processes of steps S22 to S23. If there is no other corresponding point, the depth clustering unit 45 refers to the Z value histogram table 55 to obtain the peak of the histogram in step S25.

  In step S <b> 26, the depth clustering unit 45 collects corresponding points in the vicinity of the peak to create a cluster, and stores the cluster in the corresponding point cluster table 58. In step S27, the depth clustering unit 45 determines whether there is another histogram peak. If there are other histogram peaks, the depth clustering unit 45 repeats the processes of steps S25 to S26. If there is no other histogram peak, in step S28, the depth clustering unit 45 removes a feature point having no approximate feature point in the cluster, as will be described later.

  In step S29, the depth clustering unit 45 deletes the cluster when the number of feature points in the cluster is equal to or smaller than the threshold value. In step S30, the depth clustering unit 45 determines whether there is another cluster. If there are other clusters, the depth clustering unit 45 repeats the processes of steps S28 to S29. If there is no other cluster, the depth clustering unit 45 ends the process shown in FIG.

  The details of the process of step S5 in FIG. 3 are as follows. Based on the clustering result, the thumbnail evaluation unit 46 calculates whether or not the current frame is suitable as a thumbnail as an evaluation value.

  The following indices are used as frame evaluation values. The first index is whether or not there are a plurality of clusters included in a frame. Obviously, when a single cluster is included in a frame, it does not look three-dimensional. A frame looks more three-dimensional when it contains about three clusters. On the other hand, if the number of clusters in the frame is too large, the frame becomes cluttered as a whole and becomes difficult to see in three dimensions.

  The second index is whether or not the corresponding points in each cluster are collected. When the corresponding points in the cluster are scattered in the image, the frame becomes difficult to see three-dimensionally. The third index is whether or not the Z value distance between clusters is large. The frame looks more three-dimensional when the Z-value distance between clusters is large.

  The fourth index is whether the innermost cluster among the clusters is the background and whether the number of feature points included in the background is large. The frame looks three-dimensional as the number of feature points included in the background increases. In addition, the frame is easier to see in three dimensions as the background feature points gather at the top of the screen.

The thumbnail evaluation unit 46 performs scoring based on the above-described index, and extracts a three-dimensionality value. Regarding the number of clusters, using the evaluation function f _c of the weight w _c, the number of clusters in the argument. The weight w _c is a predetermined constant. For the evaluation function f _c may Equation (5) is utilized as follows.

  A method such as an average distance between clusters can be used to collect corresponding points in each cluster. When the image size is w, h, the cluster set is C, the number of corresponding points in the cluster is n, the X coordinate of the corresponding point i is x (i), and the y coordinate is y (i), the set of corresponding points The evaluation function fs can be expressed by the following equation (6).

The magnitude of the Z value distance between clusters is the magnitude of the distance of the Z value between clusters, where z _b is the innermost Z value of the cluster and z _f is the Z value of the frontmost cluster. The evaluation function f _z can be expressed by the following equation (7).

Regarding the background evaluation, when the innermost cluster set is C _b , the number of corresponding points in the cluster is n, the height of the image is h, and the y coordinate of the corresponding point i is y (i), The evaluation function f _b for evaluation can be expressed by the following equation (8).

  The frame evaluation function f is obtained in a form in which the above four evaluation functions are combined with weights, as expressed in Equation (9).

  The process of step S5 of FIG. 3 can be performed, for example, according to the procedure of the flowchart shown in FIG. FIG. 22 is a flowchart of an example of the process in step S5.

  In step S41, the thumbnail evaluation unit 46 calculates an evaluation value related to the number of clusters. In step S42, the thumbnail evaluation unit 46 calculates an evaluation value related to the cluster group. In step S43, the thumbnail evaluation unit 46 calculates an evaluation value related to the Z value difference of the cluster. In step S44, the thumbnail evaluation unit 46 calculates an evaluation value related to the cluster background. In step S45, the thumbnail evaluation unit 46 combines all the evaluation values calculated in steps S41 to S44 with weights to calculate an evaluation value.

  Note that the process of step S5 illustrated in FIG. 22 is an example, and any process may be used as long as the evaluation value is calculated using at least one combination of steps S41 to S44.

  Details of the processing in step S28 in FIG. 21 are as follows. FIG. 23 is a flowchart of an example of the process in step S28. In step S51, the depth clustering unit 45 refers to the corresponding point table 54 and the corresponding point cluster table 58, selects one of the feature points in the cluster, and acquires the position.

  In step S52, the depth clustering unit 45 selects one of the other feature points in the cluster and acquires the position. In step S53, the depth clustering unit 45 calculates the difference (distance) between the positions of the feature points selected in steps S51 and S52.

  In step S54, the depth clustering unit 45 determines whether the distance calculated in step S53 is equal to or less than a threshold value. If not below the threshold value, in step S55, the depth clustering unit 45 determines whether there is another feature point in the cluster. If there is another feature point in the cluster, the depth clustering unit 45 repeats the processing of steps S52 to S54. If there is no other feature point in the cluster, the depth clustering unit 45 deletes the feature point selected in step S51 from the cluster in step S56.

  In step S54, the depth clustering unit 45 determines whether there is another feature point in the cluster in step S57 after the processing in step S56. If there is another feature point in the cluster, the depth clustering unit 45 repeats the processes of steps S51 to S56. If there is no other feature point in the cluster, the depth clustering unit 45 ends the processing shown in FIG.

  FIG. 24 is a functional block diagram of another example of the thumbnail extracting apparatus of this embodiment. The thumbnail extraction device 40A in FIG. 24 adds background evaluation, which is another evaluation axis, to the thumbnail extraction device 40 in FIG. Specifically, in the thumbnail extraction device 40A of FIG. 24, a background evaluation unit 48 is added to the configuration of the thumbnail extraction device 40 of FIG. In particular, in the case of home video, it is effective to extract a portion having a characteristic background, such as a video taken at a travel destination, as a thumbnail.

  The thumbnail extraction device 40A in FIG. 24 will be described focusing on differences from the thumbnail extraction device 40 in FIG. The flowchart of the thumbnail extracting device 40A differs from the thumbnail extracting device 40 shown in FIG. 3 in the process of step S5.

  The process of step S5 of the thumbnail extraction device 40A can be performed, for example, according to the procedure of the flowchart shown in FIG. FIG. 25 is a flowchart of another example of the process in step S5.

  The processing from step S61 to S64 is the same as that from step S41 to S44 in FIG. In step S65, the background evaluation unit 48 evaluates a common background between frames.

Since the background is the innermost cluster, only the innermost cluster is used. The number of frames is F, the background cluster in frame i is Cb (i), the number is n (i), the feature quantity of the corresponding point j is f (i, j), and a predetermined distance threshold is T Then, the evaluation function f _f of the background evaluation common to the frames can be expressed by the following equation (10).

  In step S66, the thumbnail evaluation unit 46 combines all the evaluation values calculated in steps S61 to S65 with weights to calculate an evaluation value. The thumbnail evaluation unit 46 obtains the frame evaluation function f in a form in which the above five evaluation functions are combined with weights so as to be expressed by Expression (11).

  Note that the process of step S5 illustrated in FIG. 25 is an example, and any process may be used as long as the evaluation value is calculated using at least one combination of steps S61 to S65.

(Summary)
The thumbnail extracting devices 40 and 40A shown in the first and second embodiments need to extract thumbnails that look more stereoscopic when 3D video is targeted. In 3D video, there are many scenes that do not look very three-dimensional. The thumbnail extracting devices 40 and 40A shown in the first and second embodiments can select thumbnails that are more impressive and look good by extracting thumbnails from a stereoscopically visible scene.

  According to the thumbnail extraction devices 40 and 40A shown in the first and second embodiments, the depth of an image is obtained using the characteristics of the 3D video, and thumbnails that can be displayed in three dimensions can be selected based on the composition.

  The recommendation program in this embodiment can be provided not only by package software but also by a WEB service or the like.

The present invention may have the following configurations as described below.
(Appendix 1)
A feature point corresponding to a pair of frame images constituting one frame of 3D video is extracted from the frame image pair as a feature point pair,
Calculating a depth amount of the feature point pair based on a distance between points of the feature point pair, a predetermined inter-lens distance / focal length, and clustering the feature point pairs by the depth amount;
Based on the evaluation condition of the three-dimensional visibility of the frame image pair based on predetermined clustering result information, the three-dimensional image of the one or more frame image pairs is obtained from the result information of the clustering of the one or more frame image pairs. Appreciable visibility,
Thumbnail extraction for causing a computer to execute processing for extracting the frame image pair having the best evaluation result as the frame image pair for thumbnails based on the evaluation result of the three-dimensional visibility of the one or more frame image pairs program.
(Appendix 2)
The thumbnail extraction program according to appendix 1, wherein the process of evaluating the stereoscopic visibility of the one or more frame image pairs evaluates the stereoscopic visibility of the frame image pair based on the number of clusters.
(Appendix 3)
The process of evaluating the stereoscopic visibility of the one or more frame image pairs is performed by evaluating the stereoscopic visibility of the frame image pairs based on the degree of positional aggregation of feature points in the cluster. A thumbnail extraction program according to 1 or 2.
(Appendix 4)
The processing for evaluating the three-dimensional visibility of the one or more frame image pairs includes any one of appendices 1 to 3 for evaluating the three-dimensional visibility of the frame image pair based on a difference in depth between clusters. Crab thumbnail extraction program.
(Appendix 5)
The process of evaluating the stereoscopic visibility of the one or more frame image pairs is performed by evaluating the stereoscopic visibility of the frame image pair based on the number and arrangement of the innermost clusters. 3. The thumbnail extraction program according to any one of 3 above.
(Appendix 6)
The processing for evaluating the three-dimensional visibility of the one or more frame image pairs includes any one of appendices 1 to 5 for evaluating the three-dimensional visibility of the frame image pair based on a common background among a plurality of frames. Crab thumbnail extraction program.
(Appendix 7)
The process of extracting the feature point from the frame image pair as the feature point pair is as follows:
Extracting the frame image pair constituting one frame of the 3D video;
Extracting the feature points from the extracted frame image pair;
Extracting a local feature amount of the extracted feature point;
The thumbnail extraction program according to any one of supplementary notes 1 to 6, wherein the feature points corresponding to the frame image pairs are extracted as the feature point pairs using the extracted local feature amounts.
(Appendix 8)
A thumbnail extraction method executed by a computer,
A feature point corresponding to a pair of frame images constituting one frame of 3D video is extracted from the frame image pair as a feature point pair,
Calculating a depth amount of the feature point pair based on a distance between points of the feature point pair, a predetermined inter-lens distance / focal length, and clustering the feature point pairs by the depth amount;
Based on the evaluation condition of the three-dimensional visibility of the frame image pair based on predetermined clustering result information, the three-dimensional image of the one or more frame image pairs is obtained from the result information of the clustering of the one or more frame image pairs. Appreciable visibility,
A thumbnail extraction method, wherein the frame image pair having the best evaluation result is extracted as the frame image pair for thumbnails based on the evaluation result of the stereoscopic visibility of the one or more frame image pairs. .

10 PC
DESCRIPTION OF SYMBOLS 21 Input device 22 Display device 23 PC main body 31 Main memory device 32 Arithmetic processing device 33 Interface device 34 Recording medium reading device 35 Auxiliary storage device 36 Recording medium 37 Bus 40, 40A Thumbnail extraction device 41 Frame image acquisition unit 42 Feature point extraction unit 43 local feature extraction unit 44 feature point pair extraction unit 45 depth clustering unit 46 thumbnail evaluation unit 47 thumbnail extraction unit 48 background evaluation unit 51 video information table 52 frame information table 53 local feature table 54 corresponding point table 55 Z value histogram table 56 Corresponding point histogram table 57 Z-value cluster table 58 Corresponding point cluster table 59 Frame evaluation table 100 Left frame image 101 Right frame image

Claims (5)

A feature point corresponding to a pair of frame images constituting one frame of 3D video is extracted from the frame image pair as a feature point pair,
Calculating a depth amount of the feature point pair based on a distance between points of the feature point pair, a predetermined inter-lens distance / focal length, and clustering the feature point pairs by the depth amount;
Based on the evaluation condition of the three-dimensional visibility of the frame image pair based on predetermined clustering result information, the three-dimensional image of the one or more frame image pairs is obtained from the result information of the clustering of the one or more frame image pairs. Appreciable visibility,
Thumbnail extraction for causing a computer to execute processing for extracting the frame image pair having the best evaluation result as the frame image pair for thumbnails based on the evaluation result of the three-dimensional visibility of the one or more frame image pairs program. The thumbnail extraction program according to claim 1, wherein the process of evaluating the stereoscopic visibility of the one or more frame image pairs evaluates the stereoscopic visibility of the frame image pair based on the number of clusters. The process of evaluating the stereoscopic visibility of the one or more frame image pairs evaluates the stereoscopic visibility of the frame image pairs based on the degree of positional aggregation of feature points in the cluster. Item 3. A thumbnail extraction program according to item 1 or 2. The process of evaluating the stereoscopic visibility of the one or more frame image pairs evaluates the stereoscopic visibility of the frame image pair based on a difference in depth amount between clusters. A thumbnail extraction program according to any one of the above. A thumbnail extraction method executed by a computer,
A feature point corresponding to a pair of frame images constituting one frame of 3D video is extracted from the frame image pair as a feature point pair,
Calculating a depth amount of the feature point pair based on a distance between points of the feature point pair, a predetermined inter-lens distance / focal length, and clustering the feature point pairs by the depth amount;
Based on the evaluation condition of the three-dimensional visibility of the frame image pair based on predetermined clustering result information, the three-dimensional image of the one or more frame image pairs is obtained from the result information of the clustering of the one or more frame image pairs. Appreciable visibility,
A thumbnail extraction method, wherein the frame image pair having the best evaluation result is extracted as the frame image pair for thumbnails based on the evaluation result of the stereoscopic visibility of the one or more frame image pairs. .

Images