JP2008140107A - Image processor, image processing method, control program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain a composite image having a more natural finish without troubling a user. <P>SOLUTION: This image processor includes detecting an object included in an image, detecting a light source direction with respect to the object or a shading direction as the direction of shading to be generated by the rays of light, and selects the image of an object having the same or approximate light source direction or shading direction, and executes image composition by using the selected image of the object. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

The present invention relates to an image processing apparatus, an image processing method, a control program, and a recording medium, and in particular, an image processing apparatus, an image processing method for performing image composition using an object such as a human face,
The present invention relates to a control program and a recording medium.

2. Description of the Related Art Conventionally, there has been known a method of providing an interesting photo print by cutting a human face image from a plurality of images and synthesizing it with a predetermined template or decoration frame to generate a composite image.
For example, in the technique described in Patent Document 1, a face image of a person captured in image data such as a group photo is detected, the distance between the faces of adjacent persons is measured, and the distance between persons close to each other is measured. Group face images.
Then, by pasting the obtained group at the composition position and creating composite data, it is possible to simply generate composite image data in which face images of persons having a certain relationship are arranged and arranged It is.

JP 2006-121661 A

However, in the above-described prior art, the point that is taken into account when generating a composite image by cutting out a person's face image is only the relationship of forming a group such as a family, a friend, or a couple of persons. There was no point of view of performance, for example, generating a more natural composite image.
Therefore, there is a problem that a photograph having an unnatural and uncomfortable finish that can be clearly seen as a composite image is obtained.
Accordingly, an object of the present invention is to provide an image processing apparatus, an image processing method, and a control program capable of obtaining a composite image having a more natural finish without bothering the user.

In order to solve the above problems, an image processing apparatus includes an object detection unit that detects an object included in an image, and a direction detection unit that detects a shadow direction that is a direction of a light source with respect to the object or a shadow generated by light from the light source. An object selection unit that selects an image of an object having the same or approximate light source direction or shadow direction, and an image composition unit that performs image composition using the image of the selected object. It is a feature.
According to the above configuration, the object detection unit detects an object included in the image.
The direction detection unit detects a shadow direction that is a light source direction with respect to an object or a direction of a shadow generated by light from the light source.
As a result, the object selection unit selects an image of an object having the same or approximate light source direction or shadow direction, and the image composition unit performs image composition using the image of the selected object.
Therefore, the synthesized image includes only the image of the object having the same or approximate light source direction or shadow direction, and a synthesized image with less sense of incongruity can be obtained.

In this case, the image synthesis unit is configured to synthesize the image of the selected object with a predetermined background image, and the object selection unit is the same or approximate to the light source direction or the shadow direction as the background image. You may make it select the image of the object which has.
According to the above configuration, the image of the object to be synthesized and the background image have the same or approximate light source direction (or shadow direction), and a synthesized image with less discomfort for the viewer can be obtained.
A face image selection processing table that stores the detected object and the light source direction or the shadow direction in association with each other, and the object selection unit refers to the face image selection processing table to The image may be selected.
According to the above configuration, by referring to the face image selection processing table at the time of image composition,
An optimal object can be selected easily and quickly.

Further, the direction detection unit stores in advance a plurality of shadow templates representing the change state of the shadow of the object in a predetermined light source direction or a predetermined shadow direction, and the change state of the shadow most closely matches the image of the object. The light source direction or the shadow direction corresponding to the shadow template to be used is the light source direction or shadow direction of the image of the object.
According to the above configuration, the direction detection unit can reliably detect the light source direction or the shadow direction of the object image with a small amount of arithmetic processing.
Furthermore, the object may be a face.

Further, the image processing method is the same as or similar to an object detection process for detecting an object included in an image and a direction detection process for detecting a shadow direction that is a direction of a light source with respect to the object or a shadow generated by light from the light source. An object selection process for selecting an image of the object having the light source direction or the shadow direction, and an image synthesis process for performing image synthesis using the image of the selected object.
According to the above configuration, the synthesized image includes only the image of the object having the same or approximate light source direction or shadow direction, and a synthesized image with less sense of discomfort can be obtained.

Further, in a control program for controlling an image processing apparatus that performs processing of an input image by a computer, an object included in the image is detected, and a light source direction with respect to the object or a direction of a shadow caused by light from the light source A shadow direction is detected, an image of an object having the same or approximate light source direction or the shadow direction is selected, and image composition is performed using the image of the selected object.
According to the above configuration, the synthesized image includes only the image of the object having the same or approximate light source direction or shadow direction, and a synthesized image with less sense of discomfort can be obtained.
The control program may be recorded on a computer-readable recording medium.

Next, preferred embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a schematic configuration block diagram of an image processing apparatus according to an embodiment.
The image processing apparatus 10 includes a CPU 11 that controls the entire image processing apparatus, a ROM 12 that stores various control programs including an image processing program, and an RA that temporarily stores various data.
M13 and an interface unit 14 for performing various interface operations, and the CPU 11
The ROM 12, the RAM 13, and the interface unit 14 are connected to each other via a bus 15.

Further, the interface unit 14 is connected to various input devices 16 such as a keyboard for inputting various data by a user, a mouse, a scanner for scanning images, a digital camera for capturing images, and a digital video. ing. Further, an output device 17 such as a display or a printer is connected to the interface unit 14. Furthermore,
The interface unit 14 includes a magnetic recording device for recording on a magnetic recording medium such as a flexible disk and a hard disk, an optical disk recording device for recording on an optical recording medium such as an optical disk, and a memory for recording on a semiconductor recording medium such as a semiconductor memory card. An auxiliary storage device 18 such as a recording device is connected.
The input device 16 may be configured to input image data from a network (LAN, Internet, etc.) via a communication interface.

FIG. 2 is a functional block diagram of the image processing apparatus according to the embodiment.
The image processing apparatus 10 is roughly classified into an image input unit 21 for inputting image data from the outside,
From the image corresponding to the input image data, a face detection unit 22 for detecting a face image as an object, a face recognition unit 23 for recognizing which person's face image the detected face image is detected, are detected. A shadow determination unit 24 that determines a shadow direction that is a direction of a shadow generated in the face image, and a face image selection unit that selects an image having the same or approximate shadow direction as the shadow direction determined by the shadow determination unit 24 25, a background image storage unit 26 for storing a background image input via the image input unit 21, and a face image selection unit 25 for the background image read from the background image storage unit 26 based on a user instruction. And an image output unit 28 that outputs the combined image combined by the image combining unit 27.

Hereinafter, each function will be described in more detail.
The image input unit 21 has a function of capturing an image designated by the user, and supports various data formats or compression formats.
The face detection unit 22 has a function of detecting a face region by analyzing an image. In this case, a known technique may be used as a method of detecting a face from an image obtained by photographing a person. As an example of the face detection technique, a large number of face images may be collected in advance, and a face detector that identifies an image having a specific size other than the face image and the face based on the collected face images may be configured. . For example,
In order to express a face, image feature amounts generated from brightness and edges are used. The detector can be realized by a statistical method such as a multilayer neural network, a support vector machine, or discriminant analysis.
As other methods, the following are cited as references.
"Detection of face and estimation of face direction independent of background and face direction" IEICE Tech. Report, PRMU2001
-217, pp. 87-94 (2002-01), Yuichi Araki, Nobutaka Shimada, Yoshiaki Shirai The technique described in this document obtains the face part with probability, detects the face using the relaxation method, and further detects the part. Although the face direction is detected using the result, in the present embodiment, only the face detection method may be used among the techniques described in this document, and it is not necessary to perform the face direction detection. .

The face recognition unit 23 recognizes who the face (object) detected by the face detection unit 22 is. In this case, a publicly known technique may be used as a method for recognizing who the person's face image corresponds to. As an example, a method of collecting face images in advance and identifying an individual by a statistical method such as discriminant analysis based on the collected face images can be applied. Moreover, you may make it implement | achieve the discriminator which pinpoints an individual by learning using a neural network. Furthermore, as another technique, “person authentication device and person authentication method” disclosed in Japanese Patent Laid-Open No. 9-251534 can be cited. According to the technique disclosed in this patent document, it is possible to perform highly accurate human authentication by stably extracting feature points such as eyes and nose from an input image without being affected by changes in brightness. .

The shadow determination unit 24 has a function of determining the light source direction from the shadow (shadow) formed on the face when the light source direction changes. More specifically, as an example of processing for determining the light source direction from the reflected light of the object in the two-dimensional image, the face detection unit 22 cuts out the detected position, applies a linear function to the face image, What is necessary is just to determine a light source direction with the linear function to fit. In the following description, the configuration is described in which the light source direction is specified, but the shadow direction may be detected conversely.

The face image selection unit 25 selects a face image to be combined with a background (such as a frame) image designated by the user based on the recognition result of the face recognition unit 23 and the determination result of the shadow determination unit 24. As for the synthesis conditions in this case, generally, the following conditions are satisfied from the viewpoint of not impairing the naturalness of synthesis.
(1) Two or more same persons are not included in one composite image.
(2) The light source direction is only one direction.
Therefore, the face image selection unit 25 selects a combination of face images that meets the above conditions.

More specifically, for example, a table is created by associating the face recognition result with the light source direction result, and the face image that meets the condition is selected. Furthermore, when there are multiple combinations, it is possible to select a face image with the same shade density as much as possible, or to select a face image with the face direction as much as possible. It will be selected.
The background image storage unit 26 temporarily stores a background image designated by the user.
The image composition unit 27 determines a position at which the face image selected by the face image selection unit 25 is pasted on a background image such as a frame, and pastes the face image at a composition position set on the background image. Is generated. In this case, selection of the set combination position may be a method of inserting an image at a predetermined position such as a template, or a configuration in which a combination position (synthesis region) is manually selected by the user. Specifically, on the display constituting the output device, a plurality of elliptical areas are displayed as composition candidate positions, and a screen for allowing the user to select a composite position of the face image from the displayed plurality of elliptical areas is displayed. A configuration in which the user selects an elliptical area or drags and drops a face image on a desired elliptical area is also possible.

The image output unit 28 has a function of outputting and displaying the synthesized image synthesized by the image synthesizing unit 27 on a display so that the user can visually recognize it, or outputting it to a printer.
Next, the operation of the embodiment will be described.

FIG. 3 is a processing flowchart of the embodiment.
FIG. 4 is a process explanatory diagram of the embodiment.
First, the CPU 11 performs a composite image generation process described below on a plurality of images G1 to Gn input (designated) by the user via the input device 16 and the interface unit 14 (step S11).
Next, the CPU 11 sequentially selects one image Gi (i: 1) from the plurality of images G1 to Gn.
˜n) are read and function as the face detection unit 22 to detect a face image which is an object included in the image (step S12). In this case, the CPU 11 stores basic information such as an original image file name, a position (x and y coordinates) in the original image, and a size including the outline of the face for the detected face image. Become.
Subsequently, the CPU 11 recognizes who the face image detected in the process of step S12 is (step S13). In the example of FIG. 4, the CPU 11 captures the face images from person A to person A.
It is classified into four people of D.

More specifically, the CPU 11 functions as a face recognition unit based on face images of users registered in advance, identifies individuals corresponding to each face image, and uses the obtained recognition result as a person tag. Information is recorded in the RAM 13 or the auxiliary storage device 18 as information. The above description is for the case where the face image to be recognized is a face image of a person registered in advance, but when the face image of a person registered in advance cannot be specified The CPU 11 outputs a message to inform the user of that fact on the display constituting the output device, and performs processing such as prompting additional registration for the face image. Thereby, the registered person can be recognized to some extent from the next time.

Subsequently, the CPU 11 functions as a shadow determination unit for the face image obtained through face detection and face recognition, and estimates and determines the light source direction at the time of shooting in the image from which the face image is detected (step S14). Then, the CPU 11 records the determination result of the light source direction estimated and determined in the RAM 13 or the auxiliary storage device 18.
Here, the light source direction determination process will be described.
In the present embodiment, image data called a shadow template is used for the determination process of the light source direction.

FIG. 5 shows an example in which the light source direction is determined from the shadow formation state.
The face image F1 is in a backlit state, and in this case, the entire face image is covered with shadows.
Therefore, as the corresponding shadow template T1, an image in which one surface is in a low brightness state (dark part) is used.
The face image F2 has a light source direction on the right side of the drawing. In this case, the shadow image covers the left side of the face image in the drawing.
Therefore, as the corresponding shadow template T2, a gradation image is used in which the left side is in the low brightness state (dark part) and the right side is in the high brightness state (bright part) as viewed in the drawing.
The face image F3 has a light source direction on the left side of the drawing. In this case, the shadow image covers the right side of the face image in the drawing.

Therefore, as the corresponding shading template T3, a gradation image is used in which the right side is in a low brightness state (dark part) and the left side is in a high brightness state (bright part) as viewed in the drawing.
The face image F4 has a light source direction below the drawing, and in this case, on the face image drawing,
The upper side is covered with a shadow.
Therefore, as the corresponding shading template T4, a gradation image is used in which the upper side is in a low brightness state (dark part) and the lower side is in a high brightness state (bright part).
In the same manner, since the shadow is covered in the opposite direction of the face image with respect to the light source direction, the shadow template has a high lightness state (bright part) on the light source side and a low lightness state on the side away from the light source ( A gradation image that is a dark part) is used.

Therefore, the CPU 11 determines that a face image closer to the shadow template is a shadow template (optimum shadow template) corresponding to the face image, and determines the light source direction corresponding to the optimal shadow template of the face image. The light source direction.
Next, the CPU 11 selects a face image to be combined with a background image such as a photo frame (
Step S15).
Specifically, the CPU 11 performs synthesis based on the face image selection processing table in which the detection results of the face images obtained in steps S12, S13, and S14, the recognition results of each face image, and the determination results of the light source direction are recorded. In such a case, a combination of face images that is not unnatural is selected.

FIG. 6 is a diagram for explaining an example of the face image selection processing table.
The face image selection processing table 31 includes person tag data 32 for specifying a person, image file name data 33 for specifying an image file including a face image, and light source direction specifying data 34.
By referring to the person tag data 32 in the face image selection processing table 31, the CPU 11 can easily determine which image file name includes the person corresponding to the person tag data 32 (shown). Can be determined.
By the way, since one light source direction is always determined for each face image, the estimated light source direction is stored in the light source direction specifying data 34. In FIG. 6, for easy understanding, the mark “◯” indicates the estimated light source direction of the face image. Further, for easy understanding, in FIG. 6, the arrow of the light source direction and the shadow template are displayed in correspondence with each other, but actually, only data indicating that they correspond to these are stored. ing.

For example, it can be seen that the image file with the image file name N001 corresponding to the person tag A includes a face image taken by the person A receiving the light source from the left side.
With reference to the face image selection processing table 31, the CPU 11 selects a combination that has a selection condition (two or more same persons are not included and the light source direction is only one direction).
More specifically, when the light source direction of the composite image is the left direction, the person A and the image file name N0 included in the image file with the image file name N001, which are corresponding face images.
The face image of the person B included in the image file of 03 or the person C included in the image file of the image file name N005 is used as the selection result. The above description is for the case where face images in the same light source direction exist, but the light source direction approximates when there is no face image in the same light source direction or when the selection target is enlarged. It is also possible to include a face image in the selection result. In the case of FIG. 6, the face image corresponding to the light source direction in the upper left direction or the face image corresponding to the light source direction in the lower left direction, which is indicated by reference numeral A1, is selected. The selection of the selection target can be configured so that the user arbitrarily sets the selection target.

Then, the CPU 11 stores information for specifying the selected face image in the RAM 13.
Next, the CPU 11 combines the face image selected in step S15 with a background image such as a photo frame (step S16).
In this case, the position of the face image to be inserted into the background image is automatically arranged at a predetermined position, or the user selects and confirms the process. Along with this, CPU1
In 1, various types of image processing such as enlarging / reducing the size of the face image so as to fit the size of the inset area in the background image are performed simultaneously.

FIG. 7 is an explanatory diagram of an image composition instruction screen.
The CPU 11 displays a plurality of elliptical regions R1 to R3 as inset synthesis candidate positions on the display as the output device 17. Then, the face image to be fitted is displayed on the display, and the user's selection operation is prompted.
As a result, the user uses the mouse as the input device 16 to use the desired elliptical region R1.
The face images G1 to G3 are dragged and dropped one by one on -R3.
As a result, the CPU 11 adjusts the image size and the like, and outputs the synthesized image to the display or, if necessary, to the printer (step S17).

FIG. 8 is an explanatory diagram of the synthesis result.
When the composition is finished, the CPU 11 adds the face images G1 to G3 to the background image B as shown in FIG.
Is displayed or printed out.
As described above, according to the present embodiment, a synthesized image having a more natural finish can be obtained without bothering the user and without a sense of incongruity between the light source directions of the background image and the face image.

In other words, shadows may occur due to local unevenness such as the nose when illuminated from the side of the face with directional lighting, but when creating a composite image, images with various illumination directions When combined, the result is a loss of natural feeling, but according to the present embodiment, a local shadow due to a change in lighting conditions is determined from the face image, and then the face image to be combined is selected to generate a combined image By doing so, an image with a more natural feeling can be generated.

In the above description, the case of a face image has been described as the object to be synthesized. However, the present invention can be similarly applied to an object such as a whole body (whole) image of a person, a vehicle, a building, or the like.

In the above description, the case of determining the light source direction has been described, but it is also possible to determine to determine the shadow direction that is the direction of the shadow caused by the light from the light source.
In the above description, the case where the control program is stored in the ROM in advance has been described. However, the control program can be recorded on a recording medium such as a flexible disk, a magnetic disk, an optical disk, or a semiconductor memory card.

Further, a network interface may be provided as a communication interface, and the control program may be downloaded and executed via the network.

1 is a schematic configuration block diagram of an image processing apparatus according to an embodiment. It is a functional block diagram of the image processing apparatus of an embodiment. It is a processing flowchart of an embodiment. It is processing explanatory drawing of embodiment. It is determination processing explanatory drawing of a light source direction. It is a figure for demonstrating an example of a face image selection process table. It is explanatory drawing of an image composition instruction | indication screen. It is explanatory drawing of a synthetic result.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Image processing apparatus, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Interface part, 15 ... Bus, 16 ... Input device, 17 ... Output device, 18 ... Auxiliary storage device, 21 ... Image input part, 22 ... face detection unit, 23 ... face recognition unit, 24 ... shadow determination unit, 25 ... face image selection unit,
26 ... Background image storage unit, 27 ... Image composition unit, 28 ... Image output unit, 31 ... Face image selection processing table, 32 ... Person tag data, 33 ... Image file name data, 34 ... Light source direction specifying data, FG ... Composition Image, T1 to T4 ... shadow template.

Claims (8)

An object detection unit for detecting an object included in the image;
A direction detection unit for detecting a shadow direction which is a light source direction with respect to the object or a shadow direction caused by light from the light source;
An object selection unit that selects an image of an object having the same or approximate light source direction or shadow direction;
An image composition unit that performs image composition using an image of the selected object;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The image combining unit combines the image of the selected object with a predetermined background image;
The object selection unit selects an image of an object having the light source direction or the shadow direction that is the same as or similar to the background image.
An image processing apparatus. The image processing apparatus according to claim 1 or 2,
A face image selection processing table for storing the detected object in association with the light source direction or the shadow direction;
The image processing apparatus, wherein the object selection unit selects an image of an object with reference to the face image selection processing table. 4. The image processing apparatus according to claim 1, wherein the direction detection unit stores in advance a plurality of shadow templates representing a change state of the shadow of the object in a predetermined light source direction or a predetermined shadow direction. An image processing apparatus, wherein the light source direction or the shadow direction corresponding to the shadow template that most closely matches the shadow change state with respect to the object image is set as the light source direction or the shadow direction of the object image. . The image processing apparatus according to any one of claims 1 to 4,
An image processing apparatus, wherein the object is a face. An object detection process for detecting objects included in the image;
A direction detection process for detecting a shadow direction which is a light source direction with respect to the object or a shadow direction caused by light from the light source;
An object selection process for selecting an image of an object having the same or approximate light source direction or shadow direction;
An image composition process for performing image composition using an image of the selected object;
An image processing method comprising: In a control program for controlling an image processing apparatus for processing an input image by a computer,
Lets you detect objects in the image,
Detecting a shadow direction which is a light source direction with respect to the object or a shadow direction caused by light from the light source;
Selecting an image of an object having the same or approximate light source direction or shadow direction;
Image synthesis is performed using the image of the selected object.
A control program characterized by that. 8. A computer-readable recording medium on which the control program according to claim 7 is recorded.