JP2001069334A - Method and device for processing photograph and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract an area of a pupil for automatically correcting the area of the pupil without requiring skill by allowing a correcting process to include a step which executes each extracting processing corresponding to the kind of a color state of the pupil to generate a candidate of a pupil image at the time of designating the area including the pupil and specifies the pupil image form the respective displayed candidates of the pupil image. SOLUTION: A correcting process includes a step which executes each extracting processing corresponding to the kind of a color state of the pupil to generate a candidate of a pupil image at the time of designating an area including the pupil and specifies the pupil image from the respective displayed candidates of the pupil image. In this photograph processing device 1, when the area including the pupil is designated, a candidate preparing means 55 successively executes each extracting processing for extracting the candidate of the pupil corresponding to the color state of the pupil and prepares the candidates of the pupil image to display the respective candidates of the pupil image on an image display part 51. When the area including the pupil is designated and the area to be corrected is decided, a boundary area correction arithmetic means 56 varies the degree of mixing original data from the center of the corrected area to the boundary of the area in the corrected area.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a photographic processing method, a photographic processing apparatus, and a storage medium capable of automatically correcting a digital image. More specifically, in digital image output to a silver halide photographic processing exposure machine, an image including a pupil is converted into image data of a color-separated digital image, a pupil image is extracted from the image data, and the pupil is identified. The present invention relates to a photographic processing method, a photographic processing device, and a storage medium for correcting an image to a natural color.

[0002]

2. Description of the Related Art When a person is photographed in color, when a flash light source such as a flash is used when the periphery of a subject is dark or the like, the pupil is photographed in red. This may result in a natural image. In order to avoid this, there are shooting methods that do not bring the optical axis of the shooting lens close to the optical axis of the flash,
There has been proposed a recording method or the like in which shooting conditions are recorded on a film or the like and correction is performed using the recording.

There is also a method in which a captured image is converted into a digital image, color-separated, a region to be corrected is extracted using each color data, and correction is performed by an image processing technique. That is, for example, using a so-called image processing software such as retouching software, a captured image is displayed on a display screen, and red eye portions are corrected pixel by pixel with a mouse or the like.

In correcting an image in this way, it is necessary for an operator to manually specify a peripheral portion including a pupil by operating a mouse or a keyboard. Then, a true pupil region is extracted from the roughly designated region, and pupil color conversion is performed.

When extracting the true pupil area by automatically or mechanically processing the data of the digital image and extracting the pupil area, the shape of the pupil area and the pupil area and its surroundings may vary depending on various situations at the time of photographing. Since the color balance with the region is various, the data processing algorithm may erroneously extract the original pupil region.

In other words, the background color of the face cannot be simply processed unexpectedly because the following factors are related to each other. First, the outer corner of the eye generally looks black due to the effect of eyelashes, but the ground is reddish, so that red can be detected from the outer corner of the eye. Eyebrows have different races and individuals, and there are also black, gold, and white. The color of the skin also varies depending on the race or individual, and the color of the skin may be reddish even in the case of a black person or in the case of shooting in a place with a strong shadow.

[0007] In this case, the red tint sometimes has a considerably high density and saturation. Also, hair may be on eyes and eyebrows. In addition, the shadows around the eyes tend to be quite dark, especially for those with deeply carved features. Since various elements are involved as described above, even if a pupil is extracted from a pixel value by digital data processing, there is a problem that there are various limitations in uniform processing.

[0008] Even when the original pupil region can be extracted, an unnatural part peculiar to digital data processing is inevitably generated at the boundary between the extracted pupil region and the original image. There is also.

[0009]

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems, and requires no skill to automatically correct a pupil region in a displayed image. An object of the present invention is to provide a photographic processing method, a photographic processing device, and a storage medium that can be automatically extracted.

[0010]

Means for Solving the Problems As a result of the present inventors' earnest tests and researches to solve the above-mentioned problems, extracting the pupil by focusing on the shape of the pupil results in obtaining a stable extraction result. Because it is difficult to focus on colors and densities, and their properties are naturally not the same, instead of processing in a unified manner, pupil extraction is performed by using different colors and densities for each case. That is what we decided to do.

That is, if the shape of the pupil is analyzed in detail, for example, a circular shape (the pupil does not cover the eyelid),
Semicircular (the upper half of the pupil is hidden, like a so-called upper eye), track type (laughing face, etc. Sometimes,
(It looks like a fan or a triangle.) It is not only presenting various shapes, but also changing completely depending on the facial expression, posture, and line of sight.In all of these cases, the conventional simple identification algorithm is effective. It is not something that can cope with.

As a means taken by the present inventors based on the above findings, the photographic processing method according to claim 1 of the present invention converts an image including a pupil into image data of a color-separated digital image. In a photographic processing method including a correction step of specifying a pupil image from data and correcting a color or the like with respect to the pupil image, when a region including the pupil is designated, each extraction process corresponding to the type of the pupil color state is performed. The correction step includes a candidate specifying step of sequentially specifying a pupil image from each of the pupil image candidates generated and displayed.

According to this method, the pupil portion can be detected regardless of the shape of the pupil, even if the designated area includes the eyelids and white eyes, or has the influence of the skin, eyelashes, and the like.

Further, in the photographic processing method according to the present invention, a threshold value is determined by a discriminant analysis method for each of the extraction processes.
Since the pupil image candidates are obtained from the image data according to the threshold value, the optimal threshold value can be uniquely and dynamically determined.

According to a third aspect of the present invention, in the photographic processing apparatus, a display means 51 for displaying a digital image, image data of the digital image, a storage means 54 for storing the image data, and an arbitrary range in the image are designated. In a photo processing device that includes a designating unit 52 and a converting unit 53 that performs a conversion process for converting the image data into an image, and corrects a color or the like for a pupil image to produce a photograph, an area including a pupil is designated. In addition, candidate extraction means 55 for sequentially performing each extraction process for extracting a pupil image candidate corresponding to the pupil color state to create a pupil image candidate and displaying each pupil image candidate is further provided. It is characterized by.

According to the apparatus, when performing pupil color correction, a plurality of pupil image candidates can be easily and objectively obtained, and it is only necessary to select an optimum pupil portion from the optimum pupil image candidates.

According to a fourth aspect of the present invention, there is provided a storage medium, wherein a computer converts a digital image, extracts a digital image, creates a candidate image,
A storage medium storing a control program for performing color correction and the like to perform color correction of a pupil image. When a region including the pupil is designated, each extraction process corresponding to the type of the pupil color state is sequentially performed. A pupil image candidate is created and displayed, and a pupil image is identified from each pupil image candidate displayed.

According to the storage medium, the specific step can be executed by using a storage medium suitable for the model of the photographic processing apparatus.

[0019]

Embodiments of the present invention will be described below with reference to the accompanying drawings.

In the present embodiment, the pupil of a person's face image is treated as an image processing object, and its color, density, and the like are corrected. Note that, among the eyes of a person, a so-called pupil and an iris are combined to form a pupil. In addition, the present embodiment also includes a case of correcting the color of the pupil, such as the color of the blue eye, the gold eye, and the like, in addition to the red eye described in the section of the related art.

FIG. 1 is an explanatory view schematically showing the configuration of a photographic processing apparatus having an automatic correction system for specifying and correcting a pupil image according to the present invention (hereinafter referred to as an "automatic pupil image correction system"). FIG. 2 is a block diagram conceptually showing mainly an image data processing system.

The photographic processing apparatus according to the present invention is similar to a conventionally known photographic processing apparatus using a silver halide photographic method except that it has an image processing section 50 described in detail below.
Developing, fixing, rinsing and drying various chemical or physical processes such as drying and the like automatically in a series of steps processing device 61, and each pixel of a color image such as a color negative blue (B), green (G) A color scanner device 62 for performing color separation of three primary colors of red and red (R) and reading out the light intensities of blue, green and red for each pixel as a color signal in order for each color; A photographic processing unit 60 including an exposure device unit 63 for scanning and exposing the photographic photosensitive material to the photographic material, and a control unit 70 connected to a power supply or the like (not shown) for controlling the operation of the entire photographic processing device. I have.

Then, after converting the color signal into a digital signal for each pixel of the color image, an image to be corrected is extracted and various image processing is performed in a correction step of correcting the image as described later. Exposure data is determined based on the image data subjected to the image processing, and the exposure data is scanned and exposed on the photographic material 40. At this time, the photographic material 40
Is pulled out of the magazine 42, transported to the exposure point 41 by the rollers 43 and the like, and is exposed in the exposure device section 51. Thereafter, normal photographic processing is performed to obtain a color print or the like. The operation of the entire photographic processing apparatus 1 as described above is controlled by the operation unit 70.

The processing device 61 and the color scanner device 6
2. Since the exposure device 51 and the like are equivalent to a conventionally known photographic processing device, a description thereof will be omitted, and the image processing unit 50 having the automatic pupil image correction system according to the present embodiment will be described below.

The image processing unit 50 includes an image display unit 51 as display means for displaying a digital image, a mouse as designating means 52 for designating and extracting an arbitrary area in the displayed image, A conversion unit 53, which is an arithmetic unit for performing an arithmetic process, an image conversion process, and the like; a storage unit, which is a storage unit 54 that stores image data and the like; Each extraction process for extracting a corresponding pupil candidate is sequentially performed to create a pupil image candidate and display each pupil image candidate, and a region including the pupil is designated, and a correction region to be corrected is determined. When the correction is performed, the correction area is provided with boundary area correction calculation means 56 for changing the degree of mixing of the original data from the center of the correction area toward the boundary of the correction area. Controlled by part 70 Is controlled.

The features of the present photographic processing apparatus will be clarified in the following description of the specific procedure of the pupil image candidate and the procedure of the boundary region correction. Is easy to objectively (or mechanically) multiple pupil image candidates simply by specifying an area about the size of glasses
It is only necessary to select the optimum pupil part from the pupil image candidates, so that skill is not required, and only by selecting the optimum pupil image candidate, the pupil image can be suitably and easily specified and corrected. To make a photo.

Further, according to the present photographic processing apparatus, a corrected image having a natural feeling that the corrected image is continuous with the original image can be obtained in an automatic processing, and a highly skilled craftsmanship can be obtained. A parameter such as a threshold value can be adjusted without a necessary operation, and an optimal corrected image can be suitably and extremely easily determined to produce a color photograph.

First, the image display section 51 displays image data obtained by reading image information recorded on a photographic material such as a negative film F as digital data by a color scanner 62 or the like. It is composed of a liquid crystal display device, a plasma display device and the like. Further, an interface 57 (keyboard or the like) capable of inputting / outputting additional instructions and information related to statistical calculation processing and image conversion processing for the target image is provided in association with the image display unit 51.

The mouse as the designating means 52 has a function of designating an arbitrary point or area of the image on the image display unit 51 and a function of acting on the image display unit 51 as described later. Things. Other devices having a similar function may be used, and a so-called trackball or the like may be used.

The conversion means 53 is provided with various thresholds and LUTs.
(lucky table), statistical processing (e.g., discriminant analysis (described later)), and image processing such as color correction for specified red-eye etc. , Toning (color, density), gradation, scaling, rotation, inversion, projection, shading, synthesis,
Various image processing such as division can be performed, and is constituted by various logic circuits, arithmetic circuits, registers, and the like.

The storage means 54 stores predetermined data relating to the threshold value and the LUT according to the present embodiment, data in the middle of the arithmetic processing of the image data and the like, and a pupil color correction called a candidate specifying step and a boundary area correcting step which will be described later. It stores a computer program or the like based on an algorithm related to the present invention, and is configured by a RAM and a ROM at a low device cost. As described above, since the image data and the like are stored in the storage unit 54, the pupil color automatic correction according to the present embodiment can be performed by high-speed automatic processing at low device cost by using a normal storage unit. it can.

Then, candidate creating means 55 according to the present invention
Performs an extraction process for sequentially extracting pupil candidates corresponding to the color state of the pupil when a region including the pupil is designated, creates a pupil image candidate, and displays each pupil image candidate on the image display unit 51 When the region including the pupil is specified and the correction region to be corrected is determined,
In the correction area, the degree of mixing the original data from the center of the correction area toward the boundary of the correction area is changed.
Both have various logic circuits and arithmetic circuits for performing arithmetic processing for generating various threshold values and LUTs, statistical arithmetic processing, image conversion processing such as color correction, and the like. It is composed of an arithmetic circuit, a register and the like.

A procedure in a specific stage for specifying a pupil image
(Procedure) and the computer program enabling the procedure in the boundary area correction step to be executed by a computer include a floppy disk and a CD-R (compact disc-recordabl).
e), can be stored and stored in a storage medium such as a MO (magnet optical disc), and can be used mounted on the storage means 54, and can be used separately from the photo processing device according to the present embodiment. it can.

In this case, the specific step and the boundary area correcting step can be executed by using a storage medium suitable for the type of the photographic processing apparatus. Further, the storage medium storing the specific step and the boundary area correction step can be widely applied to other models and the like.

In the following, referring to the flowcharts shown in FIGS. 3 to 13, in the correction step of the automatic pupil color correction system performed by the image processing unit 50 according to the present embodiment, a specifying step of specifying a pupil image candidate The procedure for implementing will be described. The procedure at this specific stage is mainly processed by the candidate creating means 55.

The pupil color states targeted by the present invention include so-called red eyes, black eyes, blue eyes, and gold eyes. Since the respective color characteristics are different, the characteristics will be described first. The principle of the extraction method set by reflecting the characteristics will be described. Among them, red eyes and gold eyes are related to shooting conditions, such as reflecting external light such as a strobe, and those in which the entire pupil or pupil looks red are called red eyes, and the entire pupil or pupil is golden and has high brightness. What looks white is called gold. In addition, these colors include those with red, blue, and green tints. In the following description, extracting a pupil image (candidate) is simply referred to as extracting a pupil.

First, the color of the iris is black or brown with a high density depending on race or individual difference (both are combined as a black eye, and among the black systems, eyes which are not red or gold eyes are black eyes), Alternatively, they are roughly classified into light gray, blue, or blue-green (three are combined as blue eyes, and among the blue eyes, eyes that are not red-eye or gold-eye are blue eyes). The present invention is directed to correcting these pupil colors in addition to the problem of toning related to shooting conditions such as red eye, as well as the problem of toning related to natural conditions such as iris color. And

The features of the eyes of each color tendency will be described below. However, in the following description, as is well known, a catch light refers to a reflected light reflected on a part of the eyeball. This image is reflected in brightness, position, size,
Various things are recognized in terms of shape and the like.

The characteristics of red eyes are as follows.
That is, (i) the pupil is often divided into three regions: a red region of the pupil, a catchlight, and an iris. (ii) In the red region, the R value is high and the GB value is low. (iii) The catchlight is achromatic or slightly reddish, and has high RGB throughout. (iv) The iris is generally achromatic or slightly reddish, and RG
The B value is low overall. (v) Saturation tends to be relatively higher than flesh color. However,
Dark areas and black skin tend to be more saturated.

From the above features (i) to (v), the red eye has a high R value and high saturation, and can be separated from other regions by the tendency that the hue is red and the saturation is high. .

Next, the features of blue eyes are as follows.
That is, (i) the pupil is often divided into three: a pupil, a catchlight, and an iris. (ii) The pupil RGB values are low overall. (iii) The catchlight is nearly achromatic and has high RGB values as a whole. (iv) The iris has a high G value or B value and a low R value. (v) Since the blue eyes are white, the periphery of the eyes is white or fair skin (RG
(The B value is high). From the above features (i) to (v), the blue eye can be separated from other regions except for the catchlight due to the tendency that the R value is generally lower than that of the surroundings.

Next, the features of the iris are as follows.
That is, (i) the pupil is often divided into the pupil, the catchlight, and the iris. (ii) The RGB values of the pupil and iris are low as a whole. (iii) The catchlight is almost achromatic and the RGB values are high overall. From the above features (i) to (iii), the G value has high contrast with the white eye and skin, the image quality is stable, and the blue eye can be separated using the G value.

Next, the features of the gold eyes are as follows.
That is, (i) the pupil is often divided into a pupil and an iris. (ii) The pupil has various cases such as when the pupil is white, reddish, bluish, or greenish, and any one of the RGB values shows a considerably high value. (iii) The RGB values of the iris are lower than the pupil. From the features (i) to (iii) described above, any of the gold values of the gold values has a high value, so that the gold values can be separated.

The following algorithms (1) to (8) are configured to extract a pupil based on the knowledge about each color state such as red eye and blue eye described above. At this time, since red and blue are not uniformly extracted from digital image data depending on the relationship between the color of the pupil and the color of the background of the face, etc. It should be noted that the flow is different for each.

In addition to the above-described eye characteristics of each color,
In general, depending on the resolution, the radial structure of the iris can be identified.If the image is small, on the contrary, the pupil and the iris cannot be identified. The algorithms (1) to (7) were configured taking into account that it is difficult to determine the region.

In the algorithm for extracting a pupil according to the present invention, the pupil region is first designated by a mouse or the like. Specify at least one pupil within the area (regardless of pupil color)
It is assumed that the size of the pupil is, for example, 4% or more of the square area circumscribing the circle.

Hereinafter, algorithms (1) to (7) in a specific stage for specifying a pupil image candidate, which is an overall flow for extracting a pupil according to the present invention, will be described. Such an algorithm is (1)
Image size adjustment, (2) RGB decomposition, (3) Red eye extraction using R saturation image, (4) Blue eye extraction using R image, (5) Black eye extraction using G image, (6) Gold eye extraction using RGB maximum value image ,
(7) Image size adjustment and each step up to (7) are the overall flow.

Then, in the overall flows (1) to (7),
Important and characteristic data processing methods commonly used in each stage include (A) threshold determination by discriminant analysis, (B) candidate search, (C) feature calculation, (D) candidate search 2, (E) closing -A data processing method called opening is used.
Further, these overall flows (1) to (7) and the data processing methods (A) to (E) include subroutines (a) to (h) described later. Therefore, the overall flow, important data processing methods (hereinafter, referred to as processing methods), and subroutines will be described in this order.

First, FIG. 3 shows the overall flow, in which a candidate image represented by a bit image and
An RGB color image is input to the entire flow. Hereinafter, in the flowchart, the candidate image is described as cnd, and the RGB color image is described as rgb. In the following, symbols that are not otherwise specified are based on conventional usage.

(1) “Image Size Adjustment” Step (Steps 1000 and 1)
100, denoted as resize in FIG. 3) is the candidate image cnd and RG
With respect to the B color image rgb, when the designated image (bit image) is larger than a predetermined threshold (for example, 150),
The reduction is performed by linear interpolation, and the candidate image is substituted by securing an image area of the same size as the reduced image. On the other hand, if it is equal to or less than the predetermined threshold, the specified image is used as it is.

(2) In "RGB separation" (step 2000), the specified image is separated into images for each of R, G, and B colors for all pixels. The color-separated images are shown below.
Described as r, g, b.

(3) "Red eye extraction by R saturation image" stage
In (step 3000 and FIG. 4), the “maximum filter ((b) of the subroutine described later)” for R, and the “minimum filter ((a) of the subroutine)” for G and B Will be applied. As is well known, the maximum value filter is a process for thickening a bright portion, and the minimum value filter is a process for thinning a bright portion. An image in which the saturation is emphasized by these processes is generated, and a red saturation image is generated by the following equation (Formula 1) (red saturation image creation).

That is, s is the saturation value of the pixel of interest, r, g, b
Is the smaller of the RGB values, g, and b of the pixel of interest.
min, the larger value is expressed as max, and PERM
Is a threshold value of the R value to be determined as chromatic, for example, 50
Is set. At this time, the logical sum of the bit operation (logical operation) is written as “&”, and the PERM is (PERM <r) &
For && (max <r),

S = 100 × (r−min) / (r + 1), s = 0 in other cases.

A threshold value is determined for this red saturation image by a well-known “discrimination analysis method”, and judgment conditions 1 and 2 are set using the threshold value, and a pupil image A “candidate search” for searching for a candidate is performed. In FIG. 4, red is an R saturation image, and bin1 and bin2 are 2
Value image, min, max, dlt are the minimum and maximum values of R saturation value,
Each represents an increment. The detailed flow of “candidate search” is shown in FIG. 8 (described later).

In the "red eye extraction using R-saturation image" stage, the optimal threshold can be uniquely and dynamically determined by using the discriminant analysis method without any subjective parameter. There is an advantage that the threshold can be determined.

(4) “Blue Eye Extraction by R Image” Step (Step 4
000 and FIG. 5), the R image is subjected to the “minimum value filter”, and the threshold value is determined by the “discrimination analysis method” and the “candidate value” is determined by the determination conditions 1 and 2 as in step 3000. Search "is performed.

(5) “Eye extraction by G image” (step 500)
0 and FIG. 6), the G image is subjected to the “minimum filter”, the threshold is determined by the “discrimination analysis method”, and the “candidate search” is performed by the determination conditions 1 and 2. Will be Here, g means the G image.

(6) "Gold extraction by RGB maximum value image"
(Step 6000, Fig. 7) In the step, "RGB maximum value filter"
After performing a process of replacing the pixel value of interest by the maximum value of the RGB value of the pixel by an operation, performing a “maximum value filter” and determining a determination condition, a “candidate search” is performed,
When a candidate pupil image exists, the candidate pupil image is registered. The “RGB maximum value filter” replaces the value of the pixel of interest with the maximum value of RGB. Here, cth means an RGB maximum value image.

Next, (7) “Image Size Adjustment” (Step 700)
0, 7100, which is denoted as resize in FIG. 3). Here, when the candidate image is larger than a predetermined threshold, the candidate image is reduced by linear interpolation, and the candidate image is substituted by securing an image area of the same size as the reduced image. On the other hand, if it is not more than the predetermined threshold, the candidate image is used as it is.

Thus, through the steps (1) to (7),
A pupil image candidate is registered. Next, important data processing methods
(A) to (E) will be described.

First, the processing method (A) "determination of threshold value by discriminant analysis" in the above steps (3) to (6) (for example, step 3250)
Will be described. Prior to the discriminant analysis method, a histogram hist of the value of each pixel is created. For this histogram hist, when separating the histogram into two groups with a certain value, moving the value to be separated over the entire histogram
The value at which the ratio between the inter-group variance and the within-group variance becomes maximum (that is, dynamically) is set as the threshold th. This ratio is an evaluation function for measuring good separation, and the threshold th is determined so as to maximize this evaluation function. This well-known statistical method is a discriminant analysis method. Here, an optimum threshold th for dividing the histogram into two parts, the object and the background, is dynamically determined.

Next, the processing method (B) “candidate search” in the above steps (3) to (6) (steps 3400, 3500, 4400, 4500, etc.
FIG. 8) will be described. In the "candidate search", for red eyes (or gold eyes, the parentheses in this paragraph are replaced at the same time), input an R saturation image (or RGB maximum value image) gray, and set th as the minimum saturation After substituting the value min and performing the binarization calculation, “labeling” and “feature calculation”,
“Candidate search 2” is performed, and for the candidate image, a determination is made of th ≦ max with respect to the maximum value of the saturation (or the maximum value of the RGB maximum values) max.

Here, in the binarization calculation, the coordinates of the pixel are x,
When expressed as y, for the R saturation image (or RGB maximum value image) gray, when gray [x, y] <th, bin [x, y] = 0 and gr
When ay [x, y] ≧ th, bin [x, y] = 1. In the flowchart, gray is an R saturation image, an RGB maximum value image,
Represents an R image or a G image, and lbl is a labeling image (described later)
Represents

On the other hand, for blue eyes (or black eyes, the parentheses in this paragraph are replaced at the same time), substantially the same algorithm as the algorithm in FIG. 8 is applied. The difference from the algorithm of FIG. 8 is that after inputting the R image (or G image), the maximum value max of the saturation is substituted as th, and the binarization calculation is performed when gray [x, y] ≧ th. , Bin [x, y] = 0, and gra
When y [x, y] <th, bin [x, y] = 1.
After performing “labeling” and “feature calculation”, “candidate search 2” is performed. However, it is different from the case of red-eye or gold-eye in that the determination of min ≦ th is performed in step 7810 instead of th ≦ max.

Here, the "labeling" is a process for distinguishing a plurality of clusters when there are some clusters. And cut out the separated object. The algorithm is described in the subroutine section.

Next, the processing method (C) “feature calculation” in the processing method (B) “candidate search” will be described. In the "feature calculation" (for example, step 7450), among the feature conditions described later, when the feature condition is satisfied, the circumscribed rectangle and radius of the label are calculated, and when the feature condition is satisfied, the density of the label is calculated. It is sequentially determined whether or not the condition is satisfied.

Here, the characteristic condition is determined as dust determination, Pixel ≦ pixel, the characteristic condition is also determined as dust determination, Smin ≦ cw × ch, and the characteristic condition is determined as position and size. Stuff and rmin
≤Rmin, and rmean≤Rmean, and rmax≤Rmax and si
ze ≦ Smax.

Where cw and ch are the width, height, and rmi of the circumscribed rectangle.
n, rmax, rmean are the minimum, maximum, and average distances from the center of the label, respectively, Rmin, Rmean, and Rmax are the minimum, average, and maximum radius thresholds of the label, nlab, respectively.
el is the number of labels, density is the density of the label, Smin, Smax
Are the threshold of the label circumscribed rectangle minimum size, the threshold of the label circumscribed rectangle maximum size, Pixel is the threshold of the number of label pixels, and pixel is the number of label pixels.

Next, the processing method (D) “candidate search 2” in the processing method (B) “candidate search” (for example, step 7470,
In FIG. 9 and FIG. 10), the feature condition,
After determining whether or not the above condition is satisfied, binarization, hole filling, and structural element size calculation are performed, and a “closing / opening” (step 7560, described later) process is performed, so-called dust removal is performed. Then, the feature of the candidate label is calculated again, and it is sequentially determined whether or not the feature condition is satisfied. Among these, when the feature condition is satisfied, it is determined that there is a candidate.

Here, the calculation of the structure element size σ is performed by cw ≦
When ch, σ = cw / 2 + 1, and when ch <cw, σ = ch / 2
+1 is calculated. In the feature recalculation, the aspect ratio of the circumscribed rectangle is expressed as rate, and when cw ≦ ch, rate = 10
When 0 × cw / ch and ch <cw, rate = 100 × ch / cw is calculated.

The feature condition is related to the next candidate condition and is “there is a next candidate label with the smallest rmin”. The feature condition is rm in the feature determination before shape correction.
ean ≦ Rmean and Density / 5 ≦ density, and the feature condition is Smin ≦ cw × ch and Rate <rate and Density ≦ de
nsity. Where Density is the label density threshold, Rate
Is a threshold value of the label circumscribed rectangle aspect ratio.

For eyes other than the red eye, "opening closing" (FIG. 10, step 7660) is performed instead of "closing opening" while performing the same algorithm as for red eyes.

Processing method (D) The above-mentioned processing method (E) “closing opening” in “candidate search 2”,
"Opening and Closing" is performed after "Image Expansion" (described later) is performed on the binary image, and then "Create Structural Element"
This is an arithmetic process performed after (described later).

First, "closing" performs dilation and erosion in this order, and "opening" performs erosion and dilation in this order. “Closing” and “opening” are arithmetic processing performed as an image processing technique for smoothly converting one image to another image.

These "closing opening"
Or, the operation called "opening and closing" is a so-called morphology (morphology) operation, the processing defined by the Minkovski sum is a relation, and the meaning as an image processing technique is:
This corresponds to “shifted overlap (or expansion)”. When this process is performed, small holes or cove-shaped images may be closed, and the separated images may be connected.

On the other hand, the process defined by the Minkowski difference is erosion, which corresponds to “scraping (or shrinking)”. When this process is performed, the narrow isthmus-like image is separated, and the thin peninsula-like image is shaved and lost.
Then, the hole shape located inside the image is enlarged.

Next, the structuring element is a binary image representing a circular area of a designated size, and the image extension is to widen the width and height necessary for performing a set operation on this structuring element. After the image processing as described above is performed, the image is extracted and output so as to cut out the image other than the image expanded part.

Next, each subroutine will be described below. (a) The minimum value filter (for example, step 3150 in FIG. 4) replaces the pixel value of the target pixel with the minimum value in the filter, and the filter size is, for example, 5 × 5. The minimum value filter has a function of lowering the luminance of the target pixel so that the luminance around the target pixel is low (if the density is high) and dragged by the low luminance.

(B) Maximum value filter (for example, step 31 in FIG. 4)
(00) replaces the pixel value of the pixel of interest with the maximum value in the filter. The filter size uses, for example, 5 × 5 only for the RGB maximum value image, and 3 × 3 for the others. The maximum value filter is used when the brightness around the pixel of interest is high.
It has the function of increasing the luminance of the pixel of interest so as to be dragged by it (if the density is low).

(C) In candidate registration, a pupil image candidate is additionally registered by a bit OR operation (OR of a logical operation). Then, conversion is performed on all pixels of the pupil image candidate.

(D) The histogram is created by a well-known calculation algorithm for the coordinate values of all pixels.

(E) In the labeling, labeling is performed with “8 adjacent” pixels. 11 and 12 show flowcharts of an algorithm for labeling with “8 adjacent” pixels.
Here, all the pixel values to be labeled have a predetermined upper limit value SUP
Is set, and the maximum value of the label can take up to SUP. In step 8859, if the call is a recursive call, the process returns to the calling source stage.

(F) In the label circumscribed rectangle calculation, the circumscribed rectangle of the designated label is calculated. The minimum value xmin, ymin and the maximum value xmax, ym of x and y of the pixel position having the designated label
ax is obtained and the upper left position x ₀ , y ₀ of the circumscribed rectangle and the width w,
Calculate the height h. Where x ₀ = xmin, y ₀ = ymin, w = xma
x−xmin + 1, h = ymax−ymin + 1.

(G) In calculating the label radius, the minimum value rmin, the maximum value rmax, and the average value rmean of the designated label from the image center are obtained. That is, first, a distance image from the center of the image is created, and the minimum value, the maximum value, and the average value are calculated from the distance image corresponding to the pixel having the designated label. This average value is calculated by dividing the sum of the distances by the number of pixels of the designated label.

(H) Filling is performed on a binary image using labeling. Fill-in-the-blank is performed by the algorithm shown in FIG. The inverted image of step 9010 is 2
This is an image obtained by inverting high and low of the value image. By setting the edge of the inverted image to high, images that are always in contact with the edge during labeling will be connected with the same label.
In step 9050, pixels having the same label as the edge are removed from the inverted image. Then, in step 9060, the value of the original pixel at the same position as the pixel position having the label in the inverted image is hi
Set to gh. In addition, w and h are each image size (width, height)
In this case, a pixel satisfying [0, y] [w-1, y] [x, 0] [x, h-1] is referred to as “touching the edge of the image”.

The thresholds and calculation formulas applied in each stage, each processing method and subroutine in each stage described above can be changed in order to appropriately reflect the color state of the pupil.

As described above, according to the automatic pupil color correction system according to the present invention, regardless of the shape of the pupil, even if the designated area includes the eyelids and white eyes, or the influence of the background, eyelashes, etc. Pupil portion can be detected, and it is possible to determine whether or not the pupil is really a pupil by detecting only the pupil portion with an extremely high probability without involving any work requiring advanced craftsmanship. Therefore, there is an advantage that the candidate specifying step in the correction step of the photographic processing can be made considerably rational.

The pupil image candidates are extracted as described above.
A pupil image is specified from the displayed pupil image candidates. Thereafter, color correction and the like are performed on the pupil image candidate. Hereinafter, referring to the flowcharts shown in FIGS. 14 to 19, in the correction process by the automatic pupil color correction system performed by the image processing unit 50 according to the present embodiment, the boundary region for performing the boundary region correction on the pupil image candidate The correction step will be described. The procedure in the boundary area correction stage is mainly processed by the boundary area correction calculation means 56.

In the present embodiment, it is assumed that one of a plurality of preset correction modes can be selected corresponding to the extracted pupil candidates before the actual correction is started. At this time, the correction mode not selected may be separately selected and processed after the processing of the selected correction mode is completed, and may be determined to be more preferable by comparing with the processing result of the correction mode previously selected. .

First, of a plurality of correction modes, a processing mode combining the boundary area processing and the density / saturation adjustment processing will be described. The first step in such processing is the production of a contour image, but prior to this, the minimum luminance value is calculated. The luminance d is obtained by d = (r + g + b) / 3.

Here, the boundary means a boundary between an area where the extracted pupil candidate image is subjected to the correction processing and an area where the correction processing is not performed, and r, g, and b are each the R of the pixel of interest.
The GB value is used.

Next, a contour image is prepared. In the present embodiment, the contour image assigns an altitude of 1 to pixels in contact with the boundary, and assigns an altitude of 2 to pixels which have not been assigned an altitude yet, which are in contact with the altitude. Hereinafter, a contour image is created with so-called eight neighbors, in which altitudes are assigned to all pixels and the calculation is repeated until contour lines are assigned to all pixels in the extraction area. FIG.
Indicates the algorithm for obtaining this contour image,
A contour image is calculated and produced according to this algorithm.

In the flowchart of FIG. 14, x, y
Represents coordinates, cnt represents a contour image, cnd represents a candidate image, bit represents a candidate bit, and & represents a bit operation. In step 2,
When w and h are the image size (width, height) respectively, [0, y]
A pixel that satisfies [w-1, y] [x, 0] [x, h-1] is referred to as “touching the edge of the image”.

Then, the color and the like are corrected for the contour image. When the altitude is equal to or higher than a predetermined constant value, the correction is unconditionally performed. When the altitude is lower than the predetermined value, the lower the altitude, the closer to the original RGB value. The values are mixed and corrected by changing the degree of mixing as described above. For example, in step 8040, when cnt [x, y] is at the edge of the image, cnt [x, y] = 1, and in step 8070, when cnt [x, y] is at the edge of the image, cnt [x, y] ] = Level and repeat steps 8020 to 8080. In this manner, in accordance with the contour line height, correction is performed so that pixels closer to the boundary are closer to the original color and luminance, that is, the color and density are continuously changing at the boundary of the region. At this time, the correspondence between the height of the contour line and the change in color and luminance may be linear in proportion, and may be a curve such as an exponential function or a higher-order curve so as to be a continuous change. Is also good.

Thereafter, a conversion L for performing the contour conversion is performed.
Create a UT. In the present embodiment, first, the maximum value of the contour line (return value of the contour image creation function) is PECSlope
The quotient level is calculated by dividing by (the ratio of mixing in the boundary processing). However, the level should be within the range of [PEMLevel, PECMax]. Therefore, when PEMLevel <level, level =
PEMLevel, and when level <PECMax, level = PECMax
Otherwise, the quotient level is set to level. However, the maximum value of the contour line is represented as “PECMax”, PEMLevel represents the minimum contour line mixed in the boundary processing, and PECMax represents the width of division by the threshold value.

Next, the conversion LUT is created by mixing the original pixel value with the correction value at a ratio of level: i below the level and at a ratio of 0: 1 when the level is larger than the level. LUT [i]
Is a conversion table, and A is a standardization constant. That is, i ≦ le
When vel, LUT [i] = A × i / level, and when A <i, LUT [i] = A.

As described above, since the contour line image is created and the color and the like are corrected by using the contour line, a correction effect that the color correction and the like can be performed while the correction area is smoothly continuous with the original image as the image is obtained. . Accordingly, there is an advantage that a natural-looking pupil image can be uniquely corrected as digital data processing.

Next, as a second step, correction of color and the like is performed in one of a plurality of modes described below according to the color state of the original pupil image. As the first form,
A mode for adjusting the density and the saturation will be described. FIG.
9 is a flowchart illustrating adjustment of density and saturation.

In the following description of the flowchart and the algorithm at each stage, cnd is a candidate image based on a bit image, bit is a bit value, rgb is an RGB color image, cn
t is a contour image, lut is a contour conversion LUT, w and h are the width and height of the image, r, g, and b are input RGB values of the i-th pixel.
r'g'b 'is the output RGB value of the ith pixel, cd is the cnd value of the ith pixel, ct is the cnt value of the ith pixel, Dlt is the saturation threshold, and Intensity is the density correction amount .

Next, the density adjustment in step 8150 will be described. Regarding the density, it is known that the pupil generally looks fine when the density is high, except for the blue eyes, so the density is converted according to the following equation [Equation 2].

## EQU2 ## d ′ = Intensity × (d−min) / A + min

Here, d = (r + g + b) / 3 is the luminance of the pixel, min is the minimum value of the area extracted by pupil extraction, Intensity
Is a parameter for determining a scale as a conversion width, and is set, for example, to Intensity = A / 2.

That is, the minimum value of the luminance of the extracted area is obtained, and the luminance is scaled based on the minimum value. Therefore, the luminance difference from the minimum value is set to half the original value.

Therefore, for example, each concentration dr, dg, db is
d (x, r, g, b, dmin) = x + (dv), where v = Inten
By expressing by sity × (d−din) / A + dmin,
dr = Fd (r, r, g, b, dmin), dg = Fd (g, r, g, b, dmi)
n), db = Fd (b, r, g, b, dmin).

Further, regarding the saturation, it has been found that when the saturation is weakened, the red eye can be alleviated, and the color of the eye approaches the natural eye color. However, the actual red eye color balance is various, and for example, it seems that a natural finish is obtained by subtracting the saturation of red. However, this is not always the case, and particularly, the balance between G and B is not preferable. Sometimes, the result is an unnatural finish with a purple or yellow tinge, which is not uniform.

Therefore, in the actual correction, the saturation is converted according to the following equation so as not to exceed the threshold value regardless of the hue. At this time, the saturation S is S (C) = 100 × (C−min) / (c + 1), where “c” is one of RGB values and “min” is the minimum value of RGB. As the threshold value, for example, 10 is used, but can be changed as appropriate.

As an example of actual correction, cmin represents the minimum value of r, g, and b, and Fs (x, cmin) is Dlt <100 × (x−cmin) / (x +
1), Fs (x, cmin) = (100 × cmin) / (100−Dl
t) and at other times, when Fs (x, cmin) = x,
sr = Fs (r, cmin), sg = Fs (g, cmin), and sb = Fs (b, cmin).

Then, when threshold value <S (C), each value of RGB is corrected so that S (C) = threshold value. Of the above, only the saturation may be corrected according to the finished state after the correction. In the case of correcting only the saturation, the processing is performed excluding steps 8110 and 8150 shown in FIG. Thus the first
Among the modes, the reason for performing the correction according to the finished situation after the correction is that in the case of an Oriental person, even if the red eyes are corrected, they may still feel reddish.

Finally, in step 8170, conversion for boundary area correction is performed. That is, rate1 = A-rate2, rate2 = lut [c
t], Fb (r, rate1, y, rate2) = (x × rate1 + y × rate
2) / A, r '= Fb (r, rate1, sr, rate2),
g ′ = Fb (g, rate1, sg, rate2), b ′ = Fb (b, rate1,
sb, rate2).

Next, as a second embodiment, a description will be given of processing in which the boundary area processing and the gray conversion are combined. FIG.
Is a flowchart showing such processing. FIG.
In the gray correction, the RGB value is converted to gray by the following equation. c = (r + g + b) / 3

At this time, the correction result changes from gray to a black system color, and in the case of strong red eyes, the finish is like a Westerner. For this process, the density adjustment performed in the first embodiment may not be performed depending on the finished state after the correction. In this case, processing is performed except for steps 8210 and 8260.

Next, as a third mode, a process combining the boundary area processing and the conversion to blue (or the conversion to green / blue, in the description of this paragraph, the reading in parentheses is simultaneously replaced)
(Or) will be described. This process (or)
It is a correction that finishes the eye itself like a Westerner. FIG. 17 shows an example of correction of conversion to blue, FIG.
8 is a flowchart showing an example of conversion to green-blue, respectively. The correction performs conversion to blue or green-blue when the original image has bright eyes.

That is, the conversion to blue is smaller than the RG value by B
In order to increase the value, for example, the following equation: r = min (g, b),
g = min (g, b) and b = max (g, b).
The conversion to green-blue is performed, for example, by the following equation:
b = min (g, b), g = max (g, b), b = max (g, b). The outline of the boundary area processing is the same as the processing or.

In this way, the minimum value and the maximum value are represented by G, B
When R is used, the R value is too high when strong red-eye is corrected, so that the human eyes become unnaturally vivid blue or blue-green eyes. Such conversion to green or green-blue is performed such that no correction is made if the original value is low, that is, the hue is changed so that the saturation approaches the blue side.

Next, as a fourth embodiment, a description will be given of processing in which the boundary area processing and the correction of the gold are combined. Since the gold eyes have an open pupil and the entire pupil shines, a simple process does not provide a natural finish.

Therefore, in the gold-eye correction, the luminance is reduced in a V-shape from the center to the end of the pupil (when viewed from the end to the end through the center, the brightness is reduced in a W-shape (see FIG. 20)). Degree adjustment and boundary area correction are performed. That is, after setting a circular region having a V-shaped luminance distribution in the specified rectangular region, the luminance is determined such that the center of the circle is max, the boundary is 0, and the center between the center and the boundary is min. In this way, the luminance distribution was adjusted to the luminance distribution seen by natural eyes. FIG. 19 is a flowchart showing the algorithm of such processing.

However, in FIG. 19, cth is the golden-eye corrected image, ch is the cth value of the pixel, dif is dif = dmin−dmax,
An example in which gold correction is performed by cmin = dif and cmax = dif / 4 is shown. Saturation adjustment and boundary area correction are processing ~
Is the same as In steps 8590 to 8595 of this processing,
Four-neighbor smoothing is performed using a 3 × 3 four-neighbor mask.

The pupil image with the most natural feeling is selected from the pupil images corrected as described above. As described above, by including the present boundary region correction step in the correction process,
The correction area can be corrected so as to be smoothly continuous with the original image as an image, and a natural-looking pupil image can be corrected in various correction candidate images regardless of the color state of the pupil or the influence of the surrounding area such as the background or eyelashes. There is an advantage that correction can be easily performed simply by making a comparison among them.

[0118]

According to the photographic processing method according to the present invention, when a region including the pupil is designated, each pupil image candidate is generated and displayed by sequentially performing each extraction process corresponding to the type of the pupil color state. The correction step includes a candidate specifying step of specifying a pupil image from the candidates.

According to this method, the pupil portion can be detected regardless of the shape of the pupil, even if the designated area includes eyelids and white eyes, or has an influence of the skin, eyelashes, and the like. It is possible to detect only the pupil portion with extremely high probability and determine whether or not the pupil is really a pupil, without involving any work requiring advanced craftsmanship.

Further, a threshold value is determined by a discriminant analysis method for each of the extraction processes, and a pupil image candidate is obtained from the image data according to the threshold value. Therefore, the optimum threshold value can be uniquely and dynamically determined. Therefore, the threshold value can be determined without any intervening subjective parameters.

In the photographic processing apparatus according to the present invention, when a region including the pupil is specified, each extraction process for extracting a pupil image candidate corresponding to the color state of the pupil is sequentially performed to create a pupil image candidate. And a candidate creating unit for displaying each pupil image candidate.

According to this apparatus, when performing pupil color correction, a plurality of pupil image candidates can be easily and objectively obtained, and only the optimum pupil portion is selected from the optimum pupil image candidates. Can be adjusted, the optimal pupil image candidate can be suitably specified, corrected, and the like can be performed to produce a photograph.

The storage medium according to the present invention can convert, extract, create candidate images,
A storage medium storing a control program for performing color correction of a pupil image by performing color correction and the like. When a region including the pupil is designated, each extraction process corresponding to the type of the pupil color state is sequentially performed. A pupil image candidate is created and displayed, and a pupil image is identified from each pupil image candidate displayed.

According to the storage medium, the specific step can be executed by using a storage medium suitable for the model of the photographic processing apparatus. Therefore, the storage medium storing the specific step can be widely applied to other models and the like. Becomes possible.

[Brief description of the drawings]

FIG. 1 is a block diagram schematically showing a configuration of a photo processing apparatus according to an embodiment of the present invention.

FIG. 2 is a block diagram schematically showing a configuration of a photo processing apparatus according to the embodiment of the present invention.

FIG. 3 is a flowchart showing a pupil candidate identification step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 4 is a flowchart showing a pupil candidate identification step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 5 is a flowchart showing a pupil candidate identification step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 6 is a flowchart showing a pupil candidate identification step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 7 is a flowchart showing a pupil candidate identification step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 8 is a flowchart showing a pupil candidate identification step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 9 is a flowchart showing a pupil candidate identification step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 10 is a flowchart showing a pupil candidate identification step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 11 is a flowchart showing a pupil candidate identification step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 12 is a flowchart showing a pupil candidate identification step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 13 is a flowchart showing a pupil candidate identification step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 14 is a flowchart showing a boundary region correction step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 15 is a flowchart showing a boundary region correction step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 16 is a flowchart showing the boundary region correction step of the correction step according to the embodiment of the present invention in the order of steps.

FIG. 17 is a flowchart showing a boundary region correction step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 18 is a flowchart showing a boundary region correction step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 19 is a flowchart showing a boundary region correction step of a correction step according to the embodiment of the present invention in the order of steps.

FIG. 20 is a graph showing a luminance distribution of gold correction according to the embodiment of the present invention.

[Explanation of symbols]

50 image processing section, 51 image display section, 53 arithmetic processing section, 54
... storage unit, 55 ... candidate creation means, 56 ... boundary area correction calculation means

 ────────────────────────────────────────────────── ─── Continued on the front page (72) Inventor Minoru Uchida 11-30 Enomachi, Suita-shi, Osaka F-term in Toyo Information System Co., Ltd. (reference) 5C076 AA21 AA22 AA26 BA03 BA07 BB42 CA02 CA10 5C079 HB01 HB06 JA12 LA02 LA10 LA12 LA37 LB12 MA01 MA04 MA11 NA05 NA27 PA08

Claims (4)

[Claims] 1. A photographic process including a correction step of converting an image including a pupil into image data of a color-separated digital image, specifying a pupil image from the image data, and correcting a color or the like of the pupil image. In the method, when a region including the pupil is specified, a candidate specifying step of sequentially performing each extraction process corresponding to the type of the pupil color state to generate a pupil image candidate and specifying a pupil image from each displayed pupil image candidate is performed. A photographic processing method, wherein the correction step includes: 2. The photographic processing method according to claim 1, wherein a threshold value is determined by a discriminant analysis method for each of the extraction processes, and a pupil image candidate is obtained from image data according to the threshold value. 3. Display means (51) for displaying a digital image.
Image data of a digital image, storage means (54) for storing the image data, designating means (52) for designating an arbitrary range in the image, and conversion for performing a conversion process for converting the image data into an image. Means (53), wherein a pupil image candidate corresponding to the color state of the pupil is extracted when a region including the pupil is specified in the photo processing apparatus for preparing a photograph by correcting the color and the like with respect to the pupil image. The photographic processing apparatus further comprises a candidate creating means (55) for sequentially performing each extraction process to create a pupil image candidate and displaying each pupil image candidate. 4. A storage medium storing a control program for performing color correction of a pupil image by performing conversion, extraction, candidate image creation, color correction, and the like of a digital image by a computer, and designates an area including a pupil. Then, each extraction process corresponding to the type of the pupil color state is sequentially performed to generate a pupil image candidate, and a pupil image is identified from each displayed pupil image candidate. Storage medium.