JP2013225731A - Image processing device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically select a favorable thumbnail image from a moving image.SOLUTION: An image processing device comprises: an extraction unit that extracts a plurality of images estimated to have a scene change from a moving image; a sorting unit that sorts the plurality of extracted images on the basis of feature amounts of the images; and a selection unit that sequentially performs face detection on the plurality of sorted images, and selects an image in which a face area is not detected.

Description

  The present invention relates to an image processing apparatus and a program for extracting thumbnail images.

  In recent years, on-demand services and the like include services that display thumbnail images to make it easier to grasp the program video to be distributed. When determining a thumbnail image, the image is often manually selected, and a large amount of video has to be processed, resulting in a very high work cost.

  As a technique for extracting thumbnail images without manual intervention, for example, there is a technique for extracting an image after a certain number of seconds from the start of a program (Patent Document 1). There is also a technique for selecting a thumbnail image based on a thumbnail image selected by the user first (Patent Document 2). Furthermore, there is a technique that uses information written on the Internet for a program as a measure of image selection (Patent Document 3).

JP 2003-274361 A JP 2005-294904 A JP 2007-166166 A

  However, in the conventional technique for extracting an image after a certain number of seconds, the extracted image is not necessarily suitable for a thumbnail image. Further, in the conventional technique in which the user first selects a thumbnail image or uses write information, the thumbnail image cannot be selected without human intervention.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus and program capable of automatically selecting a suitable thumbnail image from a moving image.

  An image processing apparatus according to an aspect of the present invention includes an extraction unit that extracts a plurality of images that are estimated to have undergone a scene change from a moving image, and extracts the plurality of extracted images based on feature values of the images. A sorting unit that rearranges and a selection unit that sequentially performs face detection on a plurality of rearranged images and selects an image in which no face area is detected.

  Further, the extraction unit may convert each pixel value in the image into an HSV color system, and determine a scene change using an S value representing saturation.

  The extraction unit may extract a background area in the image and determine a scene change using pixels of the background area.

  The sorting unit may generate an edge image from the extracted image, and rearrange the images in descending order of the edge amount obtained from the edge image.

  In addition, the sorting unit may rearrange the images in order of the color of the extracted images.

  According to another aspect of the present invention, there is provided a program for extracting, from a moving image, a plurality of images that are estimated to have undergone a scene change, and extracting the plurality of extracted images from the moving image. A sorting step of rearranging based on the amount and a selection step of sequentially performing face detection on the plurality of rearranged images and selecting an image in which no face area has been detected are executed.

  According to the present invention, a suitable thumbnail image can be automatically selected from a moving image.

1 is a block diagram illustrating an example of a schematic configuration of an image processing apparatus according to an embodiment. The block diagram which shows an example of the function of the image processing apparatus in an Example. The block diagram which shows an example of the function of an extraction part. The figure which shows an example of a background area | region. The block diagram which shows an example of the function of a sort part. The figure which shows an example of an edge image. The figure for demonstrating the sorting of a candidate image. The block diagram which shows an example of the function of a selection part. 6 is a flowchart illustrating an example of thumbnail image extraction processing of the image processing apparatus according to the embodiment. The flowchart which shows an example of the candidate image extraction process in an Example. The flowchart which shows an example of the rearrangement process of the candidate image in an Example. 6 is a flowchart illustrating an example of thumbnail image selection processing according to the embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. First, it is preferable that thumbnail images display images selected according to the following criteria.
・ Images with greatly changed scenes (highly likely to represent the main scene of the program)
・ Images that make it easy to grasp the program contents at a glance ・ Images that do not show the person's face up in consideration of the rights of the person appearing in the image In the examples shown below, images that meet the above criteria are automatically extracted from moving images The extracted image is used as a thumbnail image.

[Example]
<Configuration>
FIG. 1 is a block diagram illustrating an example of a schematic configuration of an image processing apparatus 10 according to an embodiment. As illustrated in FIG. 1, the image processing apparatus 10 includes a control unit 102, a main storage unit 104, an auxiliary storage unit 106, a drive device 108, a network I / F unit 110, an input unit 112, and a display unit 114. These components are connected to each other via a bus so as to be able to transmit and receive data.

  The control unit 102 is a CPU that controls each device, calculates data, and processes in a computer. The control unit 102 is an arithmetic device that executes a thumbnail image extraction processing program stored in the main storage unit 104 or the auxiliary storage unit 106. The control unit 102 receives data from the input unit 112 and the storage device, calculates and processes the data, and outputs the data to the display unit 114 and the storage device.

  The main storage unit 104 is a ROM (Read Only Memory), a RAM (Random Access Memory), or the like. The main storage unit 104 is a storage device that stores or temporarily stores programs and data such as OS (Operating System) and application software that are basic software executed by the control unit 102.

  The auxiliary storage unit 106 is an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software or the like.

  The drive device 108 reads the program from the recording medium 116, for example, a flexible disk, and installs it in the storage device.

  A predetermined program is stored in the recording medium 116, and the program stored in the recording medium 116 is installed in the image processing apparatus 10 via the drive device 108. The installed predetermined program can be executed by the image processing apparatus 10.

  The network I / F unit 110 has a communication function connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network) constructed by a data transmission path such as a wired and / or wireless line. This is an interface between the device and the image processing apparatus 10.

  The input unit 112 includes a keyboard having cursor keys, numeric input, various function keys, and the like, and a mouse and a slide pad for selecting keys on the display screen of the display unit 114. The input unit 112 is a user interface for a user to give an operation instruction to the control unit 102 or input data.

  The display unit 114 is configured by an LCD (Liquid Crystal Display) or the like, and performs display according to display data input from the control unit 102.

<Function>
Next, functions of the image processing apparatus 10 will be described. FIG. 2 is a block diagram illustrating an example of functions of the image processing apparatus 10 according to the embodiment. The image processing apparatus 10 illustrated in FIG. 2 includes at least an extraction unit 202, a sorting unit 204, and a selection unit 206.

  The extraction unit 202, the sorting unit 204, and the selection unit 206 can be realized by the control unit 102, the main storage unit 104 as a work memory, or the like, for example. The control unit 102 can execute the functions of the extraction unit 202, the sorting unit 204, and the selection unit 206 by executing a program.

  The extraction unit 202 extracts a plurality of images that are estimated to have changed scenes from the moving images. For example, the extraction unit 202 extracts an image that has undergone a scene change from the moving image. The moving image is, for example, a program video, and will be described below using the program video as an example.

  Thereby, the extraction unit 202 can extract an image in which the scene has changed greatly, that is, an image that is highly likely to represent the main scene of the program. Hereinafter, the image extracted by the extraction unit 202 is also referred to as a candidate image. The extraction unit 202 outputs the extracted candidate images to the sorting unit 204.

  The sorting unit 204 rearranges the plurality of candidate images extracted by the extracting unit 202 based on the feature amounts of the candidate images. For example, the amount of colors or the amount of edges may be used as the feature amount of the image. The sorting unit 204 outputs the rearranged candidate images to the selection unit 206.

  Thereby, the sort part 204 can rearrange the image which can grasp | ascertain the content of a program at a glance from candidate images so that it may become higher rank. That is, the sorted candidate images are arranged in an order that makes it easy to grasp the program contents at a glance.

  The selection unit 206 sequentially performs face detection on a plurality of candidate images rearranged by the sorting unit 204, and selects an image in which no face is detected. The selection unit 206 performs face detection on the candidate images in an order in which the program contents can be easily understood at a glance, and can select the candidate images as thumbnail images when no face is detected.

  Thereby, the selection unit 206 can finally select an image in which a human face is not recognizable so as to be recognized as a thumbnail image.

  With the above-described configuration, the image processing apparatus 10 according to the embodiment can automatically extract a suitable thumbnail image from a moving image without any manual input. Although not shown, the image processing apparatus 10 includes a storage unit that stores candidate images, sorted images, and thumbnail images.

<Each process>
Next, each process of the thumbnail image extraction process will be specifically described.

<< Extraction process >>
FIG. 3 is a block diagram illustrating an example of the function of the extraction unit 202. As illustrated in FIG. 3, the extraction unit 202 includes a background region extraction unit 302, a conversion unit 304, and a scene change determination unit 306. The extraction unit 202 may use a technique for extracting a scene change, which is a known technique, but extracts a scene change by a process described below.

  First, the extraction unit 202 extracts frame images from the input program video at predetermined frame intervals. Hereinafter, the extracted image is also referred to as a sampling image. Note that the extraction unit 202 is not necessarily required to sample.

  The background area extraction unit 302 extracts a background area from each sampling image. As the background area, a predetermined area may be set in advance as shown in FIG. FIG. 4 is a diagram illustrating an example of the background area. In the background area shown in FIG. 4, W represents the width of the image, and H represents the height of the image.

In the example illustrated in FIG. 4, the background region represents a region having a length of r ₁ W from both ends of the image in the W direction and a length of r ₂ H from the upper end of the H direction. The lengths of r ₁ W and r ₂ H only need to be set in the background region extraction unit 302.

  The background area extraction unit 302 may use a known technique for extracting a background area. For example, the background region extraction unit 302 may extract a background region by calculating a difference between a plurality of sampling images and deleting a moving object. The background area extraction unit 302 outputs the extracted background area to the conversion unit 304.

  The reason for using the background region is to detect a scene change without being affected by a change in a person or the like shown in the center.

  The conversion unit 304 converts the RGB value of each pixel included in the background area of the sampling image into the HSV color system. H represents hue, S represents saturation, and V represents lightness. The conversion unit 304 sums or calculates the average of the S values of all the pixels included in the background area, and uses this value as the background saturation feature amount. The conversion unit 304 outputs the obtained background saturation feature amount to the scene change determination unit 306 for each sampled image.

  The reason for using the saturation is that, for example, when the studio image is switched to the location image, the vividness and color of the image change greatly. Therefore, by using the saturation, it is possible to appropriately extract a scene change image suitable for the thumbnail. The conversion unit 304 does not necessarily have to use saturation, and may use, for example, a V value (brightness), a Y value (luminance) after color difference conversion, or the like.

  The scene change determination unit 306 calculates, for each sampling image, the difference between the background saturation feature amount and the previous sampling image in time. The scene change determination unit 306 estimates the top N sampling images having a large difference value as images having a scene change, and sets these images as candidate images. Here, the candidate image is, for example, an image later in time among the two sampled images for which the difference is calculated.

  In addition to selecting the top N images, the scene change determination unit 306 may use a sampled image having a difference that is equal to or greater than a predetermined threshold as a candidate image. For the predetermined threshold value, an appropriate value may be obtained by experiment or the like and set. The scene change determination unit 306 outputs the obtained candidate image to the sorting unit 204.

《Sort processing》
FIG. 5 is a block diagram illustrating an example of the function of the sorting unit 204. In the example illustrated in FIG. 5, the sorting unit 204 includes a feature amount calculating unit 402 and an image sorting unit 404.

  For example, the feature amount calculation unit 402 calculates the edge amount of the entire candidate image. The feature amount calculation unit 402 generates an edge image from the candidate image using a known technique. An edge image is an image obtained by extracting points (edge points) whose luminance change is larger than a threshold value.

  FIG. 6 is a diagram illustrating an example of an edge image. In the example shown in FIG. 6, edge extraction is performed from a full moon image.

  The feature amount calculation unit 402 adds the luminance values of all the pixels of the generated edge image, and uses this total value as the edge amount. The feature amount calculation unit 402 obtains an edge amount for each candidate image, and outputs the obtained edge amount and the candidate image to the image sorting unit 404.

  The reason for using the edge amount is that "an image of a scene seen from a bird's eye view" such as a studio or a landscape shot is preferable as a thumbnail image for grasping program contents at a glance, and such an image has a large edge amount. This is because there is a tendency.

  In addition to the edge amount, the feature amount calculation unit 402 may calculate the number of colors in the candidate image. This is because a larger number of colors may be preferable as a thumbnail image.

  The image sorting unit 404 rearranges the candidate images based on the feature amount acquired from the feature amount calculating unit 402 in the order in which the program contents can be easily grasped at a glance. The image sorting unit 404 sets the rearranged images as image 1, image 2,..., Image N, and outputs them to the selection unit 206.

  FIG. 7 is a diagram for explaining sorting of candidate images. In the example illustrated in FIG. 7, the image sorting unit 404 rearranges the five candidate images in descending order of the edge amount. As illustrated in FIG. 7, the image sorting unit 404 sets the candidate images as an image 1, an image 2,. Image 1 is an image of a cityscape and is suitable for a thumbnail image.

《Selection processing》
FIG. 8 is a block diagram illustrating an example of the function of the selection unit 206. In the example illustrated in FIG. 8, the selection unit 206 includes a face detection unit 502, a face determination unit 504, and a thumbnail image selection unit 506.

  The face detection unit 502 performs face detection processing in order from the image 1 on the rearranged candidate images. The face detection unit 502 may use, for example, a face area detection program included in the open source computer vision library “OpenCV” developed and released by Intel as a face detection technique. Note that the selection unit 206 may use other known face detection programs.

  The face determination unit 504 determines whether or not a face area exists as a result of face detection by the face detection unit 502. If there is no face area, face determination unit 504 outputs the image to thumbnail image selection unit 506. If a face area exists, the face determination unit 504 notifies the face detection unit 502 to that effect. Thereby, face detection is performed on the next image. Further, when a face is detected in all candidate images, the face determination unit 504 notifies the thumbnail image selection unit 506 to that effect.

  The thumbnail image selection unit 506 selects a candidate image determined by the face determination unit 504 as having no face area as a thumbnail image. In addition, the thumbnail image selection unit 506 may select the image 1 as a thumbnail image when the face determination unit 504 is notified that a face area has been detected in all candidate images. The selected thumbnail image is stored in the storage unit in association with the program video.

  By performing the above processing, the image processing apparatus 10 according to the embodiment can detect a face of a person who has a large scene change from each moving image, can easily grasp the program contents at a glance, and may have a problem in rights. Images that are not recognizable can be automatically extracted as thumbnail images.

<Operation>
Next, the operation of the image processing apparatus 10 in the embodiment will be described. FIG. 9 is a flowchart illustrating an example of the thumbnail image extraction process of the image processing apparatus 10 according to the embodiment.

  In step S101 illustrated in FIG. 9, the extraction unit 202 extracts a plurality of images that are estimated to have undergone a scene change from the moving images. This extraction process will be described later with reference to FIG.

  In step S102, the sorting unit 204 rearranges the plurality of images extracted by the extracting unit 202 based on the feature amount of the images. This rearrangement process will be described later with reference to FIG.

  In step S <b> 103, the selection unit 206 performs face detection processing in the order of images rearranged by the sorting unit 204 and selects an image in which no face area has been detected. This selection process will be described later with reference to FIG.

  Thereby, the image processing apparatus 10 can extract the selected image as a thumbnail image.

  FIG. 10 is a flowchart illustrating an example of candidate image extraction processing in the embodiment. In step S201 shown in FIG. 10, the extraction unit 202 samples the input moving image at a predetermined frame interval.

  In step S202, the background region extraction unit 302 selects one sampling image from a plurality of sampling images. For example, the background region extraction unit 302 selects in order from the earliest in time.

  In step S203, the background area extraction unit 302 extracts, for example, a preset area as a background area.

  In step S204, the conversion unit 304 converts the RGB value of each pixel in the background area into an HSV value. The conversion unit 304 sums up the S values of all the pixels included in the background area, and uses the total value as a feature amount of the background region (hereinafter also referred to as a background feature amount). Further, the conversion unit 304 may calculate the background feature amount using luminance or the like.

  In step S205, the scene change determination unit 306 calculates a difference value between the target background feature value and the background feature value of the immediately preceding sampling image. Since the scene change determination unit 306 cannot obtain a difference value for the first sampled image, the scene change determination unit 306 performs the next process without obtaining the difference value.

  In step S206, the extraction unit 202 determines whether processing has been performed for all the sampled images. If processing has been performed on all sampling images (step S206—YES), the process proceeds to step S207. If processing has not been performed on all sampling images (step S206—NO), the process returns to step S202, and the next sampling image is selected. .

  In step S207, the scene change determination unit 306 extracts the top N sampling images having a large difference value as candidate images.

  As a result, the extraction unit 202 can extract the top N images having a large scene change from the moving images as candidate images.

  FIG. 11 is a flowchart illustrating an example of candidate image rearrangement processing in the embodiment. In step S301 illustrated in FIG. 11, the feature amount calculation unit 402 substitutes 1 for a variable i for identifying a candidate image.

  In step S302, the feature amount calculation unit 402 calculates the feature amount of the candidate image i. Here, the edge amount and the number of colors may be used as the feature amount.

  In step S303, the feature amount calculation unit 402 determines whether i ≧ N. N represents the number of candidate images. If i ≧ N (step S303—YES), the process proceeds to step S305, and if i ≧ N (step S303—NO), the process proceeds to step S304.

  In step S304, the feature amount calculation unit 402 adds 1 to the variable i and calculates the feature amount for the next candidate image.

  In step S305, the image sorting unit 404 sorts the N candidate images in descending order of the feature amount. The image sorting unit 404 sets image 1, image 2,..., Image N from candidate images having a large amount of features.

  As a result, the image sorting unit 404 can rearrange the candidate images in an order in which the program contents can be easily grasped at a glance.

  FIG. 12 is a flowchart illustrating an example of the thumbnail image selection process in the embodiment. In step S401 illustrated in FIG. 12, the face detection unit 502 substitutes 1 for a variable i for identifying a candidate image.

  In step S402, the face detection unit 502 performs face detection processing on the candidate image i (sorted image i).

  In step S403, the face determination unit 504 determines whether there is a face area. If there is a face area (step S403-YES), the process proceeds to step S405, and if there is no face area (step S403-NO), the process proceeds to step S404.

  In step S404, the thumbnail image selection unit 506 selects the candidate image i as a thumbnail image.

  In step S405, the face detection unit 502 determines whether i ≧ N. If i ≧ N (step S405—YES), the process proceeds to step S407, and if i ≧ N (step S405—NO), the process proceeds to step S406.

  In step S406, the face detection unit 502 adds 1 to the variable i, returns to step S402, and performs face detection on the candidate image i.

  In step S407, the thumbnail image selection unit 506 selects the image 1 as a thumbnail image.

  Thereby, the image processing apparatus 10 can output the selected thumbnail image as a thumbnail image extraction result.

  As described above, according to the embodiment, a suitable thumbnail image can be automatically selected from a moving image. In addition, according to the embodiment, as a substitute thumbnail image of a program video distributed in an on-demand service or the like, a face of a person whose scene changes are large, the program contents can be easily grasped at a glance, and there is a possibility of causing a problem in rights. Images that are not recognizable can be automatically extracted. In addition, the cost of manually selecting thumbnail images is reduced, and the operation of the program distribution service is made more efficient.

  Note that the program executed by the image processing apparatus 10 has a module configuration including each unit described in the embodiment. As actual hardware, when the control unit 102 reads out and executes a program from the auxiliary storage unit 106, one or more of the above-described units are loaded onto the main storage unit 104, and one or more of the units are main. It is generated on the storage unit 104.

  As described above, the thumbnail image extraction processing described in the above-described embodiment may be realized as a program for causing a computer to execute. The thumbnail image extraction processing described above can be realized by installing this program from a server or the like and causing the computer to execute it.

  It is also possible to record the program on the recording medium 116 and cause the computer or portable terminal to read the recording medium 116 on which the program is recorded, thereby realizing the above-described processing.

  The recording medium 116 is a recording medium for recording information optically, electrically or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc., or an information electrically such as a ROM, flash memory, or the like. Various types of recording media such as a semiconductor memory for recording can be used.

  Although the embodiments have been described in detail above, the present invention is not limited to specific embodiments, and various modifications and changes other than the above-described embodiments are possible within the scope described in the claims.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 102 Control part 104 Main memory | storage part 106 Auxiliary memory | storage part 202 Extraction part 204 Sort part 206 Selection part 302 Background area extraction part 304 Conversion part 306 Scene change determination part 402 Feature-value calculation part 404 Image sort part 502 Face detection part 504 Face determination unit 506 Thumbnail image selection unit

Claims (6)

An extraction unit that extracts a plurality of images that are estimated to have changed from a moving image;
A sorting unit that rearranges the plurality of extracted images based on the feature amount of the images;
A selection unit that sequentially performs face detection on a plurality of rearranged images and selects an image in which no face area is detected;
An image processing apparatus comprising: The extraction unit includes:
The image processing apparatus according to claim 1, wherein each pixel value in the image is converted into an HSV color system, and a scene change is determined using an S value representing saturation. The extraction unit includes:
The image processing apparatus according to claim 1, wherein a background change in the image is extracted and a scene change is determined using pixels of the background area. The sorting unit
The image processing apparatus according to claim 1, wherein edge images are generated from the extracted images, and the images are rearranged in descending order of edge amounts obtained from the edge images. The sorting unit
The image processing apparatus according to any one of claims 1 to 3, wherein the images are rearranged in descending order of colors of the extracted images. On the computer,
An extraction step for extracting a plurality of images that are estimated to have undergone a scene change from the moving images;
A sorting step of rearranging the plurality of extracted images based on the feature amount of the images;
A selection step of sequentially performing face detection on the plurality of rearranged images and selecting an image in which no face area is detected;
A program for running