JP2008020944A - Image processing method, program, and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent the satisfaction of a user with a slide show from being degraded as much as possible in applying effects to any object other than the face extracted from an image. <P>SOLUTION: When it is decided that the face region extracted from an original image is a primary object, the threshold of object extraction is 5 as it is, and any other unnecessary object region is not extracted, and effects are not applied to any unnecessary object region. When the face region detected from the original image is not the primary object, the threshold of the object extraction is 4 decreased by one from 5 or a value less than that. Thus, it is possible to increase probability that the object region other than the face is extracted, and to increase the satisfaction of a user with a slide show in applying effects to even an object as the primary object intended by the photographer. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for generating a slide show based on still images.

  There is known software that automatically generates a slide show with a display effect (effect) such as a photo clip, as well as a conventional slide show that switches in a certain time. The effect zooms and crops a part of the image and displays the template image on top of it.

  For example, as in Patent Documents 1 and 2, there is a technique for extracting a face area from an image and creating a slide show using the information.

  On the other hand, in the case of an image having no face area such as a landscape photograph or an animal photograph, it may be desired to extract a subject such as a dog or a car. However, they have a uniform shape, like eyes in face extraction, and are not easy to specify, so the extraction accuracy tends to be low. As the extraction means, a method of extracting a region where the color has changed greatly or a region having many high frequency components can be considered.

To put it simply, a certain threshold is set for the amount of color change or the height of the frequency component, and when it exceeds the threshold, it is extracted as a target region.
JP 2005-182196 A JP 2005-354322 A

  The lower the object extraction threshold is set, the higher the possibility that an object will be extracted, but there is a higher possibility that an extra area or an erroneous area will be extracted as much. And if you apply effects to extraneous objects or the wrong area, the user will not be able to realize the original effect of the effect, but will feel the effect on the wrong object worthless, and the satisfaction of the slide show will decrease There is a fear.

  The present invention has been made in view of such problems, and its purpose is to correctly extract a region to be effected from an image regardless of whether or not it is a person and not to extract extraneous objects as much as possible. In this way, the user's satisfaction with the slide show is increased.

  The present invention is an image processing method for generating slide show data based on a still image, the step of extracting a face region that is a region where a human face exists from a still image, a step of setting object extraction accuracy, In accordance with the set object extraction accuracy, a main object is extracted based on at least one of the extracted face area and object area, and a step of extracting an object area that is an area where an object other than a face exists from a still image. Determining whether or not the main subject has been extracted, and setting a value obtained by reducing the currently set object extraction accuracy by a predetermined value as a new object extraction accuracy. ,including.

  In this method, when a main subject that is a human face or object is not extracted from a still image, the object extraction accuracy is reduced by a predetermined value, and this is set as a new object extraction accuracy. With a new object extraction accuracy set to be low, the probability of extracting an object increases, so the possibility that an object will be extracted becomes higher. In addition, since the extraction accuracy is lowered only when the main subject is not extracted from the still image, there is a possibility that the extraction accuracy becomes too low and unnecessary objects are extracted, or erroneous regions are extracted as object regions. It can be suppressed as much as possible. For this reason, there is little possibility that a slide show in which an effect is applied to an unnecessary object or an incorrect area is executed and the user is not satisfied.

  A program for causing a computer to execute the above image processing method is also included in the present invention.

  Further, a computer that executes the steps included in the image processing method and the step of generating slide show data that sequentially displays still images on which at least one of the extracted face area and object area is subjected to a predetermined effect; An image processing apparatus including a display control unit that sequentially displays still images on which effects have been implemented on a display device based on slide show data generated by a computer is also included in the present invention.

  In the present invention, when the main subject which is a human face or object is not extracted from the still image, the object extraction accuracy is reduced by a predetermined value, and this is set as a new object extraction accuracy. With a new object extraction accuracy set to be low, the probability of extracting an object increases, so the possibility that an object will be extracted becomes higher. In addition, since the extraction accuracy is lowered only when the main subject is not extracted from the still image, there is a possibility that the extraction accuracy becomes too low and unnecessary objects are extracted, or erroneous regions are extracted as object regions. It can be suppressed as much as possible. For this reason, there is little possibility that a slide show in which an effect is applied to an unnecessary object or an incorrect area is executed and the user is not satisfied.

  FIG. 1 is a block diagram of a slide show creation apparatus according to a preferred embodiment of the present invention. The slide show creation device includes a CPU 8, a slide show generation unit 1, a face region extraction unit 2, an object region extraction unit 9, a storage unit 4, a memory 5, a display control unit 3, a display device 6, and an input device 7. This apparatus is typically realized by a hard disk recorder, a personal computer, a digital camera, or the like.

  The input device 7 is a device for inputting still images and music data from the outside, and includes a data reading device for a portable recording medium such as a DVD or a flash memory, a network adapter, a serial port, and the like.

  The storage unit 4 is a storage medium such as a hard disk (HDD) that stores still images and music input from the input device 7. The storage unit 4 also stores a database storing programs and template management information, a program, a face area DB and an object area DB, which will be described later.

  The memory 5 temporarily stores a program for the CPU 8 and data necessary for program processing.

  The display control unit 3 generates a video signal to be output to the display device 6 constituted by a display or the like according to a command from the CPU 8 and outputs the video signal to the display device 6. The display device may be built in the slide show creation device or an external device.

  The slide show generation unit 1 stores one or more desired still images (for example, Exif format or JPEG format image files, hereinafter referred to as original images) selected by the user among the still images accumulated in the storage unit 4. Movie data in a format in which all or a part of a plurality of original images to which a predetermined effect is given are sequentially switched and reproduced based on an operation condition defined by a desired template selected from templates stored in the unit 4 Alternatively, moving image data (slide show) including sound is generated.

  The original image used for the slide show does not have to be all of the input still images. For example, an image in which neither an object area nor a face area described later can be extracted may be discarded. Alternatively, an image group in which the main subject, the object area, and the face area are estimated to overlap or an image group with high similarity may be used as the original image of the slide show, and the rest may be discarded.

  An example of specific template operating conditions is to display or specify the object area or face area (collectively expressed as a specific area) in the image to move horizontally and vertically (pan / tilt) at a predetermined speed. Enlarge or reduce the area (zoom in / zoom out), hide other areas (masking, see FIG. 2A), or compose an image that emphasizes the specific area (composite frame, FIG. 2 ( b)), there is a type of “effect” that creates a movement of the original image by rotating a specific area. A song name of “background music (BGM)” that is played back in synchronization with the playback of the slide show may be included in the operating conditions.

  The generated moving image data may be in a format independent of the template, such as an MP3 file, or may be converted into a format in which both a still image and a template are reproduced, such as an animation GIF. .

  Although details are omitted, the template can also define display states such as arrangement coordinates, sizes, colors, and the like of various objects such as documents, characters, and icons accompanying the reproduction of the original image.

  The face area extraction unit 2 extracts a face area in each original image, that is, an existing area of a human face portion. The face area extraction unit 2 outputs a signal indicating whether or not a face area has been extracted to the slide show generation unit 1. How the face area is extracted is arbitrary, but, for example, an area having a shape similar to the face contour pattern is extracted as the face area, or the presence part of the eyes, nose and mouth and the skin color that encompasses it For example, the contour of the entire face is estimated from the area, and the corresponding area is extracted as the face area.

  The object area extraction unit 9 extracts an existence area of an object other than an animal such as a dog or a cat, a vehicle such as an automobile, or other face area as an object area. How an object is extracted is arbitrary, but for example, a pixel region in which the color change compared to the surrounding pixel region exceeds a predetermined threshold or a pixel whose frequency component exceeds a predetermined threshold An area is extracted as an object area.

  If the face area to be extracted by the face area extraction unit 2 is based on the presence of specific parts such as eyes, nose and mouth, a relatively high extraction accuracy can be ensured. On the other hand, the object region to be extracted by the object region extraction unit 9 has a non-uniform nature of the object (for example, when the object is a dog, there are large differences in physical characteristics depending on the dog breed, It is difficult to use a specific part such as a mouth as a reference), and the threshold imaging that is the standard for object extraction is uniform, so it is not effective depending on the imaging conditions, and the extraction accuracy is low compared to face area extraction. Is generally true. Then, in order to extract as many objects as possible, it is necessary to set the extraction accuracy low.

  However, if faces have already been extracted, they are generally interpreted as the main subject in the original image, and as a result of object extraction with low accuracy, an extra effect is performed on the area that was mistakenly extracted as an object. This leads to user dissatisfaction, and there is a high possibility that processing itself for extracting an object with low accuracy is wasted. In order to cope with such a problem, the present embodiment relaxes the accuracy of object extraction necessary and minimum for object region extraction according to the result of extraction of the main subject.

  3 and 4 show an example of information stored in the face area DB and the object area DB included in the storage unit 4.

  As shown in FIG. 3, the face area DB includes an identification number (ID) of each image, the size of each image, one or a plurality of face areas extracted from each image by the face area extraction unit 2, and each face area. Are stored in association with the center coordinates of each face area and the size of each face area.

  On the other hand, as shown in FIG. 4, the object region DB includes an identification number (ID) of each image, the size of each image, one or a plurality of face regions extracted from each image by the object region extraction unit 9, The center coordinates of the object area and the size of each object area are stored in association with each other.

  The information shown in FIG. 3 or FIG. 4 does not necessarily have to be stored all at the same time, and only a part of the necessary information may be stored.

  FIG. 5 is a flowchart showing the flow of the slide show creation process. This process is executed by the CPU 8.

  In S1, a process for creating a face area DB and an object area DB, which will be described later, is performed.

  In S2, it is determined whether or not the following S3 (effect generation process) has been executed for all of the original images stored in the memory 5. If there is an original image for which S3 has not been executed, the process of S3 is performed on the original image. If it is determined that S3 has been executed for all the original images, this determination is terminated.

  In S3, data for generating a predetermined effect for each face area and each object area of each image is generated.

  In S4, the display order in the slide show of each original image to which the effect is set is determined in accordance with the input order of the original image, the time stamp order, the eye name or alphabetical order of the file name of the original image, or randomly.

  In S5, the slide show generation unit 1 is instructed to create slide show data for displaying still images with effects set in the face area and object area of each still image according to the display order determined in S4.

  FIG. 6 shows a flow of processing for creating the face area DB and the object area DB. This process is executed by the CPU 8.

  In S <b> 11, one or more still images input from the input device 7 are stored in the memory 5 as original images.

  In S12, it is determined whether or not the following S13 to S18 have been executed for all of the original images stored in the memory 5. If there is an original image for which S13 to S18 have not been executed, the processes of S13 to S18 are performed on the original image. If it is determined that S13 to S18 have been executed for all the original images, this process is terminated.

  In S13, the face area extraction unit 2 is instructed to extract a face area from each original image. The face area extraction unit 2 extracts a face area from the original image in response to a command from the CPU 8. If necessary, the extracted face area is associated with a number unique to the face area, the ID of the original image from which the face area is extracted, the coordinates of the face area in the original image, and the size of the face area. And stored in the face area DB (see FIG. 3).

  In S14, the object region extraction threshold is set to a predetermined initial value (for example, the highest level “5” in FIG. 7).

  In S15, the object area extraction unit 9 is instructed to extract an object area from each original image. The object area extraction unit 9 extracts an object area from the original image in response to a command from the CPU 8. The extracted object area is associated with a number unique to the object area, the ID of the original image from which the area is extracted, the coordinates of the object area in the original image, and the size of the object area, as necessary. And stored in the object region DB (see FIG. 4).

  In S16, it is determined whether or not at least one of the extracted face region and object region, that is, the specific region is a main subject in the original image. If any specific area is the main subject, the process returns to S12, and if any specific area is not the main subject, the process proceeds to S17.

  The determination of which specific area is the main subject is made by, for example, determining whether or not the specific area exists at the center position of the original image. If the specific area exists at the center position, the specific area is the main subject. Judge that there is. Alternatively, it is determined whether or not the specific area exists at the focal position stored in the tag information of the Exif file. If the specific area exists at the focal position, it is determined that the specific area is the main subject. Alternatively, since the main subject is not necessarily at the center of the original image, if the total area of the extracted specific regions occupies a predetermined ratio (for example, 50%) or more of the area of the original image, these It is determined that all the specific areas are the main subjects.

  In S17, it is determined whether or not the current object region extraction threshold is the lowest value (for example, the lowest level “1” in FIG. 7). If it is the minimum value, the process returns to S12, and if it is not the minimum value, the process proceeds to S18.

  In S18, a value (for example, “4”, which is one level lower than “5” in FIG. 7) that is lowered by one step from the current object region extraction threshold (color change threshold, frequency threshold, etc.) is set as a new threshold. And Then, the process returns to S15.

  The operations and effects of the above processing are as follows.

  FIG. 8 shows an example of an original image. FIG. 8A is an original image in which a person is a main subject, and FIG. 8B is an original image in which a person and other objects are main subjects. FIG. 8C illustrates an original image in which an object other than a person is the main subject.

  If face areas of face 1 to face 3 are detected from the original image as shown in FIG. 8A, and the central “face 2” of these face areas is determined as the main subject in S16, object extraction is performed. The threshold value remains 5, and no other extraneous object areas are extracted, and the effect is given to the face area as the main subject instead of the extra object areas (S3), and the user's satisfaction with the slide show may be reduced. Is small.

  If a face area called face 1 is detected from the original image as shown in FIG. 8B, but it is determined in S16 that the face area is not in the center and there is no main subject, the object extraction threshold value is 5. The value is reduced by one step from 4 or less. For this reason, the probability that the objects 1 to 4 that are object regions other than the face are extracted increases, and the effect is applied to both “face 1” and “object 4” that are the main subjects originally intended by the photographer. The satisfaction of the user with respect to the slide show is increased.

  Further, since no face area is detected from the original image as shown in FIG. 8C, the object extraction threshold is reduced by one step from 5 to 4 or less, and object areas other than the face are not detected. The probability of extraction increases, the probability that the object areas of objects 1 to 4 are extracted increases, an effect is given to “object 4” which is the main subject originally intended by the photographer, and the user's Satisfaction increases.

Schematic configuration diagram of slideshow creation device A figure showing an example of effects applied to a specific area The figure which shows information of face field DB notionally The figure which shows information of object field DB notionally Flow chart showing the flow of slide show generation processing Flow chart showing the flow of DB creation processing The figure which shows an example of the step of the threshold value of object area extraction Diagram showing an example of the original image

Explanation of symbols

1: slide show generation unit, 2: face area extraction unit, 3: display control unit, 6: display device, 9: object region extraction unit

Claims (3)

An image processing method for generating slide show data based on still images,
Extracting a face region, which is a region where a human face exists, from the still image;
Setting object extraction accuracy;
Extracting an object region that is an area where an object other than the face exists from the still image according to the set object extraction accuracy;
Determining whether a main subject has been extracted based on at least one of the extracted face region and object region;
Setting a value obtained by reducing the currently set object extraction accuracy by a predetermined value as a new object extraction accuracy in response to the determination that the main subject has not been extracted;
An image processing method including:   A program for causing a computer to execute the image processing method according to claim 1. The step included in the image processing method according to claim 1 and the step of generating slide show data for sequentially displaying still images on which at least one of the extracted face area and object area is subjected to a predetermined effect are executed. A computer,
A display control unit for sequentially displaying still images on which the effect has been performed on a display device based on slide show data generated by the computer;
An image processing apparatus.

Images