JP4498040B2 - Image processing method and apparatus - Google Patents

Description

  The present invention relates to an image processing method and apparatus for correcting an image including a human skin region photographed by a photographing apparatus in an oblique light environment.

2. Description of the Related Art Conventionally, a technique for detecting a light source position direction using a light source detection unit and instructing a recommended shooting direction when shooting a subject using a shooting device is known (see, for example, Patent Document 1). It is also known to perform exposure correction on the density of the entire image and development that optimizes the density of a person who is considered to be the main subject at the film camera development station. It has been.
Japanese Patent Laid-Open No. 08-160507

  In general, when the subject is a person, it is often the case that the subject is not accepted as the quality of the photographed image when the tendency of the oblique light state is strong. However, cameras as described above that detect the light source and indicate the shooting direction are not widely used in the current market, and the subject is shot regardless of whether it is in a slanted state, so that the human image is appropriately displayed. Shooting may not be possible.

  In addition, there is a case where a certain degree of improvement can be seen for an already shot image by exposure correction centered on a person. However, even when the exposure correction is performed on the luminance distribution in the oblique light environment and the average value is optimized, the actual human skin area is partially blown out or crushed, and the correction is not performed. There are many cases where this is not done properly.

  The present invention has been made in consideration of such circumstances, and can appropriately correct the image of the central subject included in the photographed image without losing its quality. An object of the present invention is to provide an image processing method and apparatus.

In order to solve the above problems, an image processing method according to the present invention includes:
An acquisition step of acquiring an image captured by the imaging device;
An extraction step of extracting the skin area of the subject from the image;
An analysis step of analyzing the image data of the skin region;
And a determination step of determining whether or not the subject has been photographed in an oblique state based on the analysis result of the image data.

In order to solve the above problems, an image processing apparatus according to the present invention includes:
Acquisition means for acquiring an image captured by the imaging device;
Extraction means for extracting a skin area of the subject from the image;
Analyzing means for analyzing the image data of the skin region;
And determining means for determining whether or not the subject has been photographed in an oblique state based on the analysis result of the image data.

  According to the present invention, it is possible to appropriately correct an image of the central subject included in the photographed image without impairing the quality of the subject. In other words, for a human image obtained by photographing a person in an oblique light state, conventionally, even if the luminance distribution of the skin area of the person who is the subject is analyzed, the exposure correction performed using the average luminance value of the area is the target. Although it has been impossible to suppress the occurrence of whiteout or crushing in the area, according to the present invention, the light reception state from the light source, more specifically, by analyzing the image data and taking the determination of oblique light into account, Therefore, it is possible to provide higher-quality image correction.

  Hereinafter, an image processing method using an image processing apparatus according to an embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 11 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. As shown in FIG. 11, the image processing apparatus according to the present embodiment is achieved using a configuration similar to a so-called personal computer.

  Specifically, the image processing apparatus according to the present embodiment includes an input / output port (I / O port) 51 through which data is input / output, a mouse 52 that is a pointing device, and a keyboard 53 that inputs numerical values, characters, and the like. A monitor 54 for performing various displays, a controller 55 for controlling the operation of the entire apparatus, a memory card drive 56 for reading image data recorded on the memory card, and data and programs are stored. A hard disk 57 and a bus 58 for connecting the above-described units to each other are provided.

  As the monitor 54, for example, various display devices such as a liquid crystal monitor, a CRT monitor, an EL display, a plasma display, and a television receiver can be used. Or you may use what is called a touch panel provided with each function of the mouse | mouth 52, the keyboard 53, and the monitor 54. FIG.

  The control unit 55 performs various processes on the image data read by the input / output port 51 or the memory card drive 56 based on the input from the mouse 52 or the keyboard 53.

  In this embodiment, a personal computer will be described as an example of an apparatus that implements an image processing apparatus. However, any apparatus capable of processing image data may be a multifunction printer (MFP) or photo processor that handles images. A similar embodiment is conceivable even for a built-in device in a device other than a personal computer such as a direct printer (FDP).

  FIG. 1 is a flowchart for explaining the overall flow of image processing in an image processing apparatus according to an embodiment of the present invention. First, the image data read out by the input / output port 51 and the memory card drive 56 of the image processing apparatus shown in FIG. (Step S101).

  Next, the image data acquired in step S101 is expanded, and image-related data necessary for subsequent processing is acquired during the series of operations (step S102). The related data includes data necessary for feature extraction of the target image area.

  Next, feature region extraction processing is performed on the developed image data (step S103). For example, when the definition of the registered feature area is the main person skin area in the image, a processing method for detecting a person's face area in an image including a moving image is known as an entrance / exit management or monitoring system. ing. Then, the human skin or face area is subjected to feature determination, and the matched area is extracted.

  Further, the characteristic difference between the light source state for the photographed person and the photographing state level under the oblique light condition is determined from the characteristics such as the luminance distribution state of the human skin region extracted in step S103 (step S104). Here, the oblique light means that the direction of the light source with respect to the subject is an oblique direction. For example, in the case of an undulating subject such as a person's face, shadows occur in a slanting situation, which is not a good photographing situation. For this reason, it is generally considered preferable to shoot under a uniform light source such as cloudy when taking a person as a subject, and human face shoots under oblique light conditions where extreme luminance distribution occurs except for special applications Therefore, it is not possible to obtain a preferable shooting result. Therefore, in this step, the oblique light level is determined using the feature value of the luminance distribution of the human skin region extracted as the subject as a threshold value. Details of the processing will be described later.

  Next, based on the results detected and determined in step S103 and step S104, a correction instruction due to the feature amount based on the extracted data is set for the initial image correction (step S105). In the present embodiment, as the setting contents, for example, for normal image correction (contrast, white balance, exposure correction, sharpness, saturation, etc.) processing, only the instruction amount for items that require a correction amount other than the initial setting Create

  Then, it is confirmed whether or not an instruction is given in step S105. If there is an instruction, the initial set value for the image correction effect quantity is replaced with a form that reflects the instruction quantity and image correction is executed (step S106).

  As described above, in the present embodiment, when the oblique light determination level is higher than the target value for optimizing the exposure correction amount to the skin area detected by the oblique light determination level, the target value is relaxed and detected. It is possible to prevent image correction such that the skin area is overexposed or crushed.

  FIG. 12 is a flowchart for explaining details of the extraction region feature determination processing in step S104. First, in step S103, it is determined whether or not feature amount data including presence / absence information of a human skin region is generated in the image (step S1201). This processing is performed when it is detected that the image has a feature quantity that is considered to improve the image quality by setting a different correction initial setting intensity for the image correction initial setting usage form. A value different from the initial set amount for image correction is indicated.

  As a result, when the human skin region does not exist (No), the oblique light determination process is terminated. On the other hand, if a result indicating that a human skin area exists in the image is obtained (Yes), the process proceeds to step S1202.

  Next, the processing after step S1202 will be described using the sample images shown in FIGS. FIG. 2 is a sample image A in which the face of a person as a subject is photographed slightly dark. As a photographing environment of the sample image A shown in FIG. 2, the photographing is performed under a uniform light source such as a light cloud. FIG. 6 is a sample image B taken with a composition in which the person who is the subject is arranged in the center of the image. Note that the light source of the sample image B shown in FIG. 6 is an environment in which direct sunlight is irradiated from the diagonal left of the subject (that is, an oblique light environment).

  Then, information necessary for the oblique light determination is acquired from the result extracted in step S103 (step S1202). In the feature determination process, at least the luminance average value of the extracted skin region and the luminance data for each pixel are acquired. It should be noted that feature amount image data in which not only luminance data but also other saturation data and hue data are considered with appropriate weights may be used. That is, not only luminance data but also other image data (image data of a sample image) itself may be used. Then, data analysis processing for performing oblique light determination is performed from the data acquired in step S1202 (step S1203). FIG. 13 is a flowchart for explaining in detail the analysis processing of acquired data in step S1203.

  First, a luminance histogram of the human skin region extracted and acquired in step S103 is created (step S1301). FIG. 5 is a diagram showing the human skin region extracted and acquired in step S103 for the sample image A shown in FIG. The area shown in FIG. 5 may be determined after grouping by detecting the detected human skin area in units of 8 × 8 pixels, and the area may be distinguished by adding a color.

  FIG. 4 is a diagram representing the luminance average value of the portion compared with the human skin region extracted in step S103 and the luminance distribution of each pixel as a luminance histogram. The horizontal axis in FIG. 4 represents luminance, which is represented by 8 bits with “0” at the left end and “255” at the right end. The vertical axis represents the pixel distribution. As can be seen from FIGS. 2 and 4, the luminance distribution of the human skin region is distributed in a well-balanced manner around the vicinity of 124 and close to the normal distribution.

  Next, a description will be given using the sample image B shown in FIG. FIG. 9 is a diagram showing the human skin region extracted and acquired in step S103 for the sample image B shown in FIG. The area shown in FIG. 9 may be determined after grouping by detecting the detected human skin area in units of 8 × 8 pixels. For example, the area may be distinguished by adding a color.

  FIG. 8 is a diagram showing a luminance histogram of the luminance average value of the portion compared with the human skin region extracted in step S103 and the luminance distribution of each pixel. The horizontal axis in FIG. 8 is the luminance, and is represented by 8 bits with “0” at the left end and “255” at the right end. The vertical axis represents the pixel distribution. As can be seen from FIGS. 6 and 8, the luminance distribution of the human skin region has an average luminance of 133 in the human skin region, but the distribution is divided into a high luminance portion and a low luminance portion. .

  After creating the brightness histogram in this way, the brightness average value of the detected human skin area (extraction area) is calculated (step S1302). Then, for example, luminance data regions that are 25% or more and 30% or more away from the luminance average value of the detected human skin region are detected, and the area ratio between the detection region and the entire human skin region is calculated (step S1303). .

  As described above, the image analysis processing in the present embodiment is for determining whether the illumination of the light source to the human skin region is in a slanted state, so that the luminance is calculated based on the distribution of the calculated luminance histogram. An average, a luminance region of 25% or more or 25% or less from the luminance average, and a luminance region of 30% or more or 30% or less from the luminance average are calculated, respectively, and an area ratio in the target human skin region is calculated.

  After analyzing the acquired data in this way, it is determined whether or not an area that is 30% or more out of the calculated luminance average occupies an area ratio of 30% or more of all target human skin areas. (Step S1204). As a result, when the determination is met (Yes), the process proceeds to step S1205.

  In step S1205, it is determined that the person in the image that is the subject was shot under strong ambient light, and “level 1” having the most certainty is set in the “slope intensity level” that is the determination memory area. . Then, after the setting is completed, the processing (step S104) is terminated and the process proceeds to image correction.

  On the other hand, if the determination does not match in step S1204 (No), the process proceeds to step S1206. In step S1206, it is determined whether or not an area that is 25% or more out of the brightness average calculated in step S1204 occupies an area ratio of 25% or more of all target human skin areas. As a result, if the determination matches (Yes), the process proceeds to step S1207.

  In step S1207, it is determined that the person in the image that is the subject is photographed under a weak oblique light environment light, and the “oblique light level” that is the determination memory area is determined to be a certain oblique light state. Level 2 ”is set. Then, after the setting is completed, the present process (step S104) is terminated and the process proceeds to image correction. On the other hand, if the determination in step S1206 does not match the determination (No), the process ends (step S104) and the process proceeds to image correction.

  In this embodiment, in order to lighten the detection determination process, as described above, a threshold value based on the ratio of the average luminance of the region and the ratio from the average value is set as the determination of the luminance distribution. Judgment was made. However, the determination method is not limited to this, and for example, it can be performed based on a difference between high luminance and low luminance, or distribution determination using standard deviation is also conceivable.

  Next, an embodiment will be described in which the oblique light determination performed on the human skin region in the image in step S104 is reflected in the image correction processing after step S105. In image correction processing, correction items such as 1) white balance, 2) contrast, 3) exposure, 4) saturation, and 5) sharpness are generally considered as items.

  In the present embodiment, 3) the contents of exposure correction will be described. First, an example of the exposure correction operation will be described.

  FIG. 15 is a diagram showing an example of an exposure correction amount determination table used in the image processing apparatus according to an embodiment of the present invention. In FIG. 15, the horizontal axis represents the luminance average (AVE) of the input image in the range from 0 to 255, and the vertical axis represents the target value (NAVE) of the luminance average value after conversion from 0 to 255. It is expressed as a range.

  Here, the value of NAVE is obtained for each luminance average (AVE) of the input image from the conversion table of FIG. 15, and the table shown in FIG. 16 is created from the result. FIG. 16 is a diagram showing an example of a lookup table for exposure correction according to an embodiment of the present invention.

  The table creation procedure shown in FIG. 16 first plots the luminance average (AVE = A) value of the input image between 0 and 255 on the horizontal axis. Subsequently, the target value (NAVE = NA) of the luminance average value after conversion is plotted between 0 and 255 on the vertical axis. Then, the intersection points are searched for vertically from the plot points, and the conversion table is created by connecting the points (0, 0) and (255, 255) as shown in FIG. By using this conversion table, it is possible to convert the luminance value of the entire image into a desired value.

  3 and 7 detect the human skin region by performing the exposure correction processing according to the present embodiment on the sample images A and B shown in FIGS. 2 and 6, respectively, and perform image correction using the luminance average value thereof. It is a figure which shows the result of a process.

  First, FIG. 3 will be described. As a result of human skin area detection for the sample image A shown in FIG. 2, when the average brightness is acquired as a value of “124” and the brightness distribution of the area is calculated, The region has a luminance of 86 or less or a luminance of 162 or more. Further, the luminance that is 25% or more away is an area having a luminance of 93 or less or a luminance of 155 or more.

  As a result, from the calculated luminance histogram, the partial area having a luminance of 86 or less is 0%, and the partial area having a luminance of 162 or more is 2.5%, and “level 1” is not applicable in the “obliqueness level” determination. Furthermore, the partial area with a luminance of 93 or less is 0%, and the partial area with a luminance of 155 or more is 4.8%. Therefore, in the sample image A shown in FIG. 2, it is determined that the human skin is not in the obliquely lighted state as a result of detecting human skin, and exposure correction processing is performed on the detected brightness average value of the skin area to the target brightness.

  Next, FIG. 14 will be described. FIG. 14 is a diagram showing a target luminance value setting graph used for image exposure correction in one embodiment of the present invention. As described above, when the human skin area is detected in the exposure correction using the tables of FIGS. 15 and 16, the exposure correction target value is changed using the graph shown in FIG. The exposure correction target value is set to NAVE as described with reference to FIG. 15, but when a human skin area is detected in the image, the target set value is set according to a straight line connected with circles. Specifically, when the detected average luminance value of the skin area is 60, NAVE is set to a value 1.8 times the average luminance. When the detected average luminance value of the skin region is 180, NAVE is set to a value that is 1.2 times the average luminance.

  Here, in the sample image A shown in FIG. 2, since the average luminance of the detected human skin region is 124, NAVE is about 1.4 of the average luminance value. The image shown in FIG. 3 is obtained as a result of the comprehensive image correction including the exposure correction target value using the NAVE set as described above.

  FIG. 17 is a diagram showing a luminance histogram of the human skin region in the image shown in FIG. From FIG. 17, it can be confirmed that the average luminance of the human skin area has increased to 165 and is in a good state.

  Next, the image in FIG. 7 will be described. As a result of human skin area detection of the sample image B shown in FIG. 6, when the average luminance is acquired as a value of 133 and the luminance distribution of the area is calculated, the luminance is 93 or less or The region has a luminance of 173 or higher. Further, the luminance that is 25% or more away is an area having a luminance of 100 or less or a luminance of 167 or more.

  From the calculated luminance histogram, the partial area with the luminance of 93 or less is 42.4%, the partial area with the luminance of 173 or more is 25.2%, and “level 1” corresponds to the “obliqueness level” determination. Therefore, the sample image B shown in FIG. 6 is an image including human skin and is determined to be in a severe oblique light state.

  Then, by using the detected luminance average value of the skin region and the target luminance value setting graph shown in FIG. 14, the oblique light level 1 in which an exposure target value of about 1.35 times is represented by a triangle mark in the normal case. Therefore, the multiplication rate is about 1.2 times. Thereafter, processing equivalent to the above-described procedure is executed using the sample image of FIG. FIG. 7 shows an image obtained as a result of image correction using the average value of the human skin region when the oblique light determination is not used.

  FIG. 10 is a diagram showing a luminance histogram of the human skin region after the correction process using the average luminance according to the embodiment of the present invention. In FIG. 10, the horizontal axis represents luminance, and is represented by 8 bits with “0” at the left end and “255” at the right end. The vertical axis represents pixel distribution. As is apparent from a comparison between FIG. 10 and FIG. 8, it can be seen that the corrected image has a stronger contrast than the luminance distribution in the human skin region of the original image shown in FIG. In particular, in the high luminance region, the skin itself is close to a whiteout state as the luminance value rises to around 230.

  As described above, by using the oblique light determination according to the present embodiment and reflecting it in exposure correction, it is possible to perform optimal correction to the human skin area so as not to cause deterioration in quality in the human skin area such as whiteout. It was. In this embodiment, the correspondence to the high luminance part in the exposure correction is shown as an example, but by using the determination based on the luminance distribution analysis on the human skin, the optimization to the low luminance region is possible. Needless to say, it is possible to specify a target for each item of image correction such as contrast correction.

  Furthermore, in the present embodiment, when image data after shooting is used as an object for observation or printing in an information processing apparatus such as a personal computer, the image that has been corrected for each image is used. When a person or the like is included as a subject, an image more expected by the user can be provided by performing correction with emphasis on the area. Further, it is possible to provide higher-quality image correction by taking into account the light reception state from the light source in this important region by determining oblique light from the luminance distribution.

  Although the embodiment has been described in detail above, the present invention can take an embodiment as, for example, a system, apparatus, method, program, or storage medium (recording medium). The present invention may be applied to a system composed of a single device or an apparatus composed of a single device.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowchart shown in the drawing) that realizes the functions of the above-described embodiment is directly or remotely supplied to the system or apparatus, and the computer of the system or apparatus Is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

  As a recording medium for supplying the program, for example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, a client computer browser is used to connect to an Internet homepage, and the computer program of the present invention itself or a compressed file including an automatic installation function is downloaded from the homepage to a recording medium such as a hard disk. Can also be supplied. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, the present invention includes a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of the processes.

  Furthermore, after the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU or the like provided in the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

6 is a flowchart for explaining the overall flow of image processing in the image processing apparatus according to the embodiment of the present invention; This is a sample image A in which the face of the person who is the subject is photographed slightly dark. It is a figure which shows the result of the image correction process which performed the exposure correction process which concerns on this embodiment with respect to the sample image A shown in FIG. 2, detected a human skin area | region, and used the luminance average value. It is the figure which represented the luminance average value of the part contrasted with the human skin area | region extracted by step S103, and the luminance distribution of each pixel with the luminance histogram. It is a figure which shows the person skin area | region extracted and acquired by step S103 with respect to the sample image A shown in FIG. It is a sample image B photographed with a composition in which a person as a subject is arranged in the center of the image. It is a figure which shows the result of the image correction process which performed the exposure correction process which concerns on this embodiment with respect to the sample image B shown in FIG. 6, detected a human skin area | region, and used the luminance average value. It is the figure which represented the luminance average value of the part contrasted with the human skin area | region extracted by step S103, and the luminance distribution of each pixel with the luminance histogram. It is a figure which shows the person skin area | region extracted and acquired by step S103 with respect to the sample image B shown in FIG. It is a figure which shows the brightness | luminance histogram of the person skin area | region after the correction process by the average brightness | luminance which concerns on one Embodiment of this invention. 1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. It is a flowchart for demonstrating the detail of the feature determination process of the extraction area | region in step S104. It is a flowchart for demonstrating in detail the analysis process of the acquisition data in step S1203. It is a figure which shows the graph for a target luminance value setting used for image exposure correction | amendment in one Embodiment of this invention. It is a figure which shows an example of the exposure correction amount determination table used with the image processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the look-up table for exposure correction | amendment which concerns on one Embodiment of this invention. It is a figure which shows the brightness | luminance histogram of the person skin area | region in the image shown in FIG.

Claims (10)

An image processing method executed by each of an acquisition unit, an extraction unit, an analysis unit, and a determination unit of an image processing apparatus,
An acquisition step in which the acquisition means acquires an image captured by an imaging device;
An extracting step in which the extracting means extracts a skin area of a subject from the image;
An analyzing step in which the analyzing means analyzes the image data of the skin region;
The determination means, the object is closed and a determination step of determining whether taken with oblique lighting state based on the analysis result of the image data,
The analysis step calculates a region where the difference between the luminance value of each skin region of the subject and the luminance average value of the skin region is a predetermined value or more, and calculates the area ratio of the region in the skin region;
The image processing method characterized in that the determination step determines a degree of a slanting state when the image is taken according to the area ratio . Change means, and a changing step of, based on the degree of oblique state of the image determined by the determining step, changes the correction target value at the time of correcting the image,
The image processing method according to claim 1 , further comprising: a correcting unit that corrects the image according to the correction target value changed by the changing step. The extraction step, the image processing method according to claim 1 or 2, characterized in that extracts a face region of a person from the image as a skin region of the subject. The changing step changes an exposure correction target value when correcting the exposure of the image,
The image processing method according to claim 2 , wherein the correction step corrects the exposure of the image according to the exposure correction target value changed by the changing step. Acquisition means for acquiring an image captured by the imaging device;
Extraction means for extracting a skin area of the subject from the image;
Analyzing means for analyzing the image data of the skin region;
Determination means for determining whether or not the subject was photographed in a slanting state based on the analysis result of the image data ,
The analysis means calculates a region where a difference between a luminance value of each skin region of the subject and a luminance average value of the skin region is a predetermined value or more, and calculates an area ratio of the region in the skin region;
The determining means, the image processing apparatus the image according to the area ratio is equal to or you determine the extent of the oblique state when taken. Changing means for changing a correction target value when correcting the image based on the degree of the oblique light state of the image determined by the determining means;
The image processing apparatus according to claim 5 , further comprising: a correcting unit that corrects the image according to the correction target value changed by the changing unit. The image processing apparatus according to claim 5 , wherein the extraction unit extracts a human face area from the image as the skin area of the subject. The changing means changes an exposure correction target value when correcting the exposure of the image,
The image processing apparatus according to claim 6 , wherein the correction unit corrects exposure of the image according to the exposure correction target value changed by the changing unit. On the computer,
An acquisition procedure for acquiring an image captured by the imaging device;
An extraction procedure for extracting the skin area of the subject from the image;
An analysis procedure for analyzing the image data of the skin region;
A determination procedure for determining whether or not the subject has been photographed in an oblique light state based on the analysis result of the image data ;
The analysis procedure calculates a region where the difference between each luminance value of the skin region of the subject and the average luminance value of the skin region is a predetermined value or more, and calculates an area ratio of the region in the skin region;
The determination procedure, a program for you determine the extent of the oblique state when the image is captured in accordance with the area ratio. A computer-readable recording medium storing the program according to claim 9 .

Images