WO2013089051A1 - Image processing device, image processing method, image processing program, and recording medium storing image processing program - Google Patents

Abstract

Provided is an image processing device capable of extracting an area of minimal pixel change from an image, at low cost. The image processing device is provided with a seed region extraction section (10), a region expansion section (20), and a correction section (30). The seed region extraction section (10) extracts, as a seed region, a region that meets range-specifying conditions which are predetermined conditions relating to input image data (ID) luminance (Y), and saturation (r) and hue (θ) obtained from first color difference data (C_b) and second color difference data (C_r). Selecting a pixel of the seed region as the origin, the region expansion section (20) assesses whether the surrounding pixels meet gradient-specifying conditions which are conditions relating to luminance gradient (Y'), saturation gradient (r'), and hue gradient (θ'), and if the conditions are met, the pixel so assessed is included in a pre-correction target region. The correction section (30) performs a hole filling process on the pre-correction target region to acquire a post-correction target region.

Description

Image processing apparatus, image processing method, image processing program, and recording medium storing image processing program

The present invention relates to an image processing device that extracts a region of a specific part of an object represented by an input image from an input image, and more specifically, an image processing device suitable for extracting a region with a small change between pixels, The present invention relates to an image processing method, an image processing program, and a recording medium storing the image processing program.

When performing image processing using a specific part of an object represented by an image, it is necessary to extract the region of the specific part from the image. Therefore, conventionally, a technique for extracting a specific region in an image has been used. For example, extraction of an area where changes between pixels such as skin (face, etc.), blue sky, or sea are small (hereinafter referred to as “area with small pixel changes”) is related to brightness, saturation, and hue of each pixel in the image. This is performed by determining whether or not a predetermined threshold condition is satisfied. Hereinafter, region extraction using such a threshold condition is referred to as “threshold region extraction”.

In connection with the present invention, Patent Document 1 discloses a method for determining a region of interest in a skin pattern image. Here, the region of interest refers to a region including, for example, a fingerprint. In this method, values in the first value range (dark range) in the image are assigned to ridges, values in the second value range (light range) are assigned to valleys, and then the entire image Is shifted in the direction of the first value range. Then, after the entire image is divided into tiles, the average value of the shift values for the individual tiles is compared with the reference value. A tile whose average value is shifted in the first direction with respect to the reference value is regarded as a region of interest.

Further, Patent Document 2 discloses a system that adjusts the white balance of an image using the face color as a reference signal. That is, this system detects a face with a face detector and extracts skin color from the face. Then, the skin color is used for white balance adjustment.

Japanese Special Table 2006-507582 Japanese Special Table 2005-531189

However, in the method described in Patent Document 1, it is difficult to extract an area where there is no pattern such as a fingerprint, that is, an area where the pixel change is small. In the system described in Patent Document 2, a luminance-based face detection algorithm is used. For example, if a recognition technique such as SVM (Support Vector Vector Machine) is used, the entire face can be extracted. However, the cost is high, and in this case, an area with a small pixel change other than the face such as a blue sky or the sea is extracted. Not eligible. On the other hand, when a relatively inexpensive cost recognition technique is used, a part of the target area to be extracted such as a face may be missing and extracted. Such omission can be a cause of malfunction when performing predetermined image processing (for example, face recognition or face color correction) using the extracted target region.

Therefore, an object of the present invention is to provide an image processing apparatus, an image processing method, an image processing program, and a recording medium storing the image processing program, which can extract an area with a small pixel change from an image at low cost.

A first aspect of the present invention is an image processing apparatus that extracts a target region from an input image,
A seed region extraction unit that extracts, as a seed region, a region that satisfies at least conditions relating to a predetermined range of luminance, saturation, and hue from the input image;
It is determined whether or not the gradients of luminance, saturation, and hue in the peripheral region of the seed region are less than or equal to a first threshold for luminance, saturation, and hue, respectively, and the gradient is less than or equal to the first threshold. The region determined to be included in the target region and whether or not the gradient in the region around the region newly included in the target region is equal to or less than the first threshold value is determined. Including an area determined to be equal to or less than a first threshold, and an area expansion unit that acquires the area where the gradient is equal to or less than the first threshold and the seed area as the target area. Features.

According to a second aspect of the present invention, in the first aspect of the present invention,
The region expanding unit may determine whether or not the gradient is equal to or less than a first threshold, using each pixel in the seed region as a starting point.

According to a third aspect of the present invention, in the second aspect of the present invention,
The area extension is
Using each pixel in the seed region as a pixel of interest, it is determined whether or not the gradient in the surrounding pixels of the pixel is equal to or less than the first threshold value. If the determination result is equal to or less than the first threshold value, the pixel to be determined A first expansion process for including
A pixel newly included in the target region is set as a pixel of interest, and it is determined whether or not the gradient of pixels around the pixel is equal to or less than the first threshold. If the determination result is equal to or less than the first threshold, the determination is made. And a second extension process for repeatedly including the target pixel in the target area.

According to a fourth aspect of the present invention, in any one of the first to third aspects of the present invention,
The seed region extraction unit extracts a region that further satisfies a condition that the gradient is equal to or less than a predetermined value as the seed region.

According to a fifth aspect of the present invention, in the fourth aspect of the present invention,
The predetermined value is a second threshold value smaller than the first threshold value.

According to a sixth aspect of the present invention, in any one of the first to third aspects of the present invention,
When there are a plurality of regions that satisfy the conditions regarding the luminance, saturation, and hue ranges, the seed region extraction unit selects a region that satisfies a first predetermined condition regarding the area from the plurality of regions. It is characterized by extracting as a region.

A seventh aspect of the present invention is the sixth aspect of the present invention,
The first predetermined condition relating to the area is a condition that the area is maximum among a plurality of regions that satisfy the conditions relating to the luminance, saturation, and hue ranges.

According to an eighth aspect of the present invention, in the sixth aspect of the present invention,
The first predetermined condition regarding the area is a condition that the area is a predetermined value or more.

A ninth aspect of the present invention is the eighth aspect of the present invention,
When the region expansion unit obtains a plurality of regions each composed of the region determined to have the gradient equal to or less than the first threshold and the seed region, the region expansion unit A region satisfying the predetermined condition of 2 is set as a target region to be extracted.

According to a tenth aspect of the present invention, in a ninth aspect of the present invention,
The second predetermined condition regarding the area is a condition that the area is the maximum among the plurality of regions.

In an eleventh aspect of the present invention, in any one of the first to third aspects of the present invention,
The image processing apparatus further includes a correction unit that performs a predetermined correction process on the target area acquired by the area expansion unit.

A twelfth aspect of the present invention is the eleventh aspect of the present invention,
The predetermined correction process is a process of including, in the target area, an area whose area is equal to or less than a predetermined value among the areas that are not the target area in contact with the inner periphery of the target area acquired by the area expanding unit. Features.

According to a thirteenth aspect of the present invention, in any one of the first to third aspects of the present invention,
The input image is indicated by a luminance signal and a color difference signal.

A fourteenth aspect of the present invention is the thirteenth aspect of the present invention,
The image processing apparatus further includes a signal format conversion unit that converts an input image indicated by a primary color signal into the input image indicated by a luminance signal and a color difference signal.

A fifteenth aspect of the present invention is an image processing method for extracting a target region from an input image,
A seed region extraction step for extracting, from the input image, a region that satisfies at least the conditions relating to the predetermined luminance, saturation, and hue range as a seed region;
It is determined whether or not the gradients of luminance, saturation, and hue in the peripheral region of the seed region are less than or equal to a first threshold for luminance, saturation, and hue, respectively, and the gradient is less than or equal to the first threshold. The region determined to be included in the target region and whether or not the gradient in the region around the region newly included in the target region is equal to or less than the first threshold value is determined. Including an area determined to be equal to or lower than a first threshold, and an area expansion step for acquiring the area determined to be equal to or lower than the first threshold and the seed area as the target area. Features.

A sixteenth aspect of the present invention is the fifteenth aspect of the present invention,
In the region expansion step, the determination as to whether or not the gradient is equal to or less than a first threshold value is performed starting from each pixel in the seed region.

A seventeenth aspect of the present invention is the sixteenth aspect of the present invention,
The region expansion step includes
Using each pixel in the seed region as a pixel of interest, it is determined whether or not the gradient in the surrounding pixels of the pixel is equal to or less than the first threshold value. If the determination result is equal to or less than the first threshold value, the pixel to be determined A first expansion step of including in the region of interest;
A pixel newly included in the target region is set as a pixel of interest, and it is determined whether or not the gradient of pixels around the pixel is equal to or less than the first threshold. If the determination result is equal to or less than the first threshold, the determination is made. And a second extension step of repeatedly including the target pixel in the target region.

An eighteenth aspect of the present invention is an image processing program,
A computer is caused to execute each step in the image processing method according to any one of the fifteenth to seventeenth aspects of the present invention.

A nineteenth aspect of the present invention is a computer-readable recording medium,
An image processing program according to an eighteenth aspect of the present invention is recorded.

According to the first aspect or the fifteenth aspect of the present invention, an expansion process (with a gradient equal to or less than the first threshold value) is performed based on a seed region that satisfies at least conditions relating to predetermined luminance, saturation, and hue ranges. A process of determining whether or not there is and adding a region satisfying this to the target region) is performed. In the conventional threshold area extraction, since this kind of area is set as the target area to be extracted, the target area to be extracted originally lacks. However, in the first aspect or the fifteenth aspect of the present invention, the target area includes an area that should have been extracted by the expansion process but was not extracted as a seed area. For this reason, it is possible to extract almost the entire target region without missing the target region to be extracted. Accordingly, various image processing such as face recognition or face color correction can be appropriately performed using the extracted target region. Further, since high-cost recognition technology such as SVM (Support Vector Vector Machine) is not used, cost reduction can be achieved.

According to the second aspect or the sixteenth aspect of the present invention, the conditions relating to the gradient are determined using each pixel in the seed region as a starting point, and thus the same as the first aspect or the fifteenth aspect of the present invention. An effect can be obtained.

According to the third aspect or the seventeenth aspect of the present invention, by performing the first expansion process (first expansion step) and the second expansion process (second expansion step), Effects similar to those of the second aspect or the sixteenth aspect can be obtained.

According to the fourth aspect of the present invention, in the seed region extraction process, a condition is added in which the brightness, saturation, and hue gradient are equal to or less than predetermined values. For this reason, a region having a large gradient, that is, a region other than a region having a small pixel change such as a face can be excluded from the seed region. For this reason, the extraction accuracy of the target region can be increased.

According to the fifth aspect of the present invention, the conditions of the brightness, saturation, and hue gradient in the seed region extraction process are set more strictly than the conditions of the brightness, saturation, and hue gradient in the expansion process. Is done. For this reason, regions other than regions with small pixel changes, such as faces, can be more reliably excluded from the seed region. As a result, a more accurate seed region (that is, with less noise) can be used in the expansion process, and thus the extraction accuracy of the target region can be further increased.

According to the sixth aspect of the present invention, the same effect as any of the first to third aspects of the present invention is obtained by extracting a region that satisfies the first predetermined condition regarding the area as a seed region. Can be obtained.

According to the seventh aspect of the present invention, the condition that the area is the maximum among the plurality of regions that satisfy the conditions of the luminance, saturation, and hue ranges is set as the first predetermined condition relating to the area. The same effect as that of the sixth aspect of the present invention can be obtained.

According to the eighth aspect of the present invention, the same effect as that of the sixth aspect of the present invention can be obtained by setting the condition that the area is equal to or greater than a predetermined value as the first predetermined condition relating to the area. . In addition, a plurality of seed regions can be extracted. For this reason, for example, a plurality of faces can be extracted from an image in which a plurality of faces are depicted.

According to the ninth aspect of the present invention, an unnecessary target region can be excluded by setting a target region to be extracted as a target region that satisfies the second predetermined condition regarding the area.

According to the tenth aspect of the present invention, the same effect as that of the ninth aspect of the present invention is obtained by setting the condition that the area is the largest among the plurality of target regions as the second predetermined condition regarding the area. Can be obtained.

According to the eleventh aspect of the present invention, for example, the omission of the target region to be extracted can be further suppressed by the correction process.

According to the twelfth aspect of the present invention, the process of including, in the target region, a region whose area is equal to or less than a predetermined value among the regions that are not the target region that are in contact with the inner periphery of the target region acquired by the region expanding unit. Is called. For this reason, the omission of the target region to be extracted can be more sufficiently suppressed.

According to the thirteenth aspect of the present invention, in the case of extracting a target area from an image indicated by image data obtained by receiving a television broadcast or obtained from a video system, for example, The same effect as the third aspect can be obtained.

According to the fourteenth aspect of the present invention, for example, when the target area is extracted from the image indicated by the image data received from the multimedia system, the same effect as that of the first aspect or the third aspect of the present invention is obtained. be able to.

According to the eighteenth aspect of the present invention, in the image processing program, the same effect as any of the fifteenth aspect to the seventeenth aspect of the present invention can be obtained.

According to the nineteenth aspect of the present invention, in a computer-readable recording medium, the same effect as any of the fifteenth to seventeenth aspects of the present invention can be obtained.

It is a block diagram which shows the hardware constitutions of the image processing apparatus which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the functional structure of the principal part of the image processing apparatus which concerns on the said 1st Embodiment. It is a diagram schematically showing the positional relationship of the respective colors in C _r C _b coordinates. It is a flowchart of the image processing program in the said 1st Embodiment. It is a figure which shows the example which applied the image processing program in the said 1st Embodiment with respect to the sample image. (A) is a figure which shows a sample image. (B) is a diagram showing a processing result of extracting a designated color region. (C) is a figure which shows the processing result which extracted the seed area | region. (D) is a figure which shows the result of having performed the expansion process. (E) is a figure which shows the result of having performed the hole-filling process. In C _r C _b coordinate a diagram schematically showing a range of saturation and hue related skin tone. It is a flowchart which shows the detail of step S3 in FIG. It is a flowchart which shows the detail of step S33 in FIG. It is a figure for demonstrating the example of the expansion process in the said 1st Embodiment. It is a figure for demonstrating the example of the expansion process in the said 1st Embodiment. It is a figure for demonstrating the example of the expansion process in the said 1st Embodiment. It is a schematic diagram for demonstrating the expansion process object in the said 1st Embodiment. (A) is a schematic diagram when four adjacent pixels are subject to expansion processing. (B) is a schematic diagram in the case where 8 neighboring pixels are subject to expansion processing. It is a figure which shows the filter used for calculation of the gradient in the said 1st Embodiment. (A) is a figure which shows the Roberts filter of a 45 degree direction. (B) is a figure which shows the Roberts filter of a 135 degree direction. (C) is a figure which shows the pre-wit filter of a horizontal direction. (D) is a diagram showing a pre-wit filter in the vertical direction. (E) is a figure which shows the Sobel filter of a horizontal direction. (F) is a figure which shows the Sobel filter of a perpendicular direction. It is a flowchart of the image processing program in the 2nd Embodiment of this invention. It is a figure which shows the example which applied the image processing program in the said 2nd Embodiment with respect to the sample image. (A) is a figure which shows a sample image. (B) is a diagram showing a processing result of extracting a designated color region. (C) is a figure which shows the processing result which extracted the seed area | region. (D) is a figure which shows the result of having performed the expansion process. (E) is a figure which shows the result of having performed the hole-filling process. It is a block diagram which shows the functional structure of the principal part of the image processing apparatus which concerns on the 3rd Embodiment of this invention.

Hereinafter, first to third embodiments of the present invention will be described with reference to the accompanying drawings.

<1. First Embodiment>
<1.1 Configuration and processing overview>
FIG. 1 is a block diagram illustrating a hardware configuration of the image processing apparatus according to the present embodiment. As shown in FIG. 1, the image processing apparatus includes a CPU 1, an input unit 2, a memory 3, a hard disk device 4, and an output unit 5. The hard disk device 4 stores an image processing program 6 for causing the CPU 1 to execute the edge enhancement processing according to the present embodiment. For example, the image processing program 6 may be read and installed by a DVD drive or the like (not shown) from a recording medium such as a DVD-ROM storing the image processing program 6, or supplied via a network (not shown) It can be installed.

The input image data ID included in the input image signal is input to the image processing apparatus from the outside via the input unit 2. The input image data ID indicates an input image. The memory 3 temporarily stores the input image data ID and the image processing program 6 read from the hard disk device 4. The CPU 1 stores the input image data ID and the image processing program 6 based on the input image data ID and the image processing program 6. Extracted data ED indicating a target region that is a region of a specific portion (for example, a face in the present embodiment) of the target object expressed by the input image is generated. The generated extracted data ED is output to the outside via the output unit 5.

FIG. 2 is a block diagram illustrating a functional configuration of a main part of the image processing apparatus according to the present embodiment. As shown in FIG. 2, the image processing apparatus includes a seed region extraction unit 10, a region expansion unit 20, and a correction unit 30. This image processing apparatus extracts a target region from an image indicated by image data obtained by receiving a television broadcast or obtained from a video system. Therefore, the input image data ID in the present embodiment is luminance data (hereinafter also simply referred to as “luminance”) Y as a luminance signal, first color difference data C _b as a first color difference signal, and second as a second color difference signal. It consists of color difference data _Cr . Here, the first color difference data _Cb is color difference data representing a difference between the blue component in the input image and the luminance data Y. The second color difference data C _r is the color difference data representing the difference between the red component and the luminance data Y in the input image. The input image data ID is given to the seed region extraction unit 10 and the region expansion unit 20.

Seed region extraction unit 10, from the first color difference data C _b and the second color difference data C _r of the input image data ID received, obtains the saturation r and hue θ by the following equation (1).

FIG. 3 is a diagram schematically showing the positional relationship of each color in the C _{r Cb} coordinates. The positional relationship between the colors shown here is ITU-R BT. It is calculated based on the 709 standard. As shown in FIG. 3, for example, θ = 50 ° corresponds to magenta (M), θ = 103 ° corresponds to red (R), θ = 175 ° corresponds to yellow (Ye), and θ = 230 °. Corresponds to green (G), θ = 283 ° corresponds to cyan (C), and θ = 355 ° corresponds to blue (B).

The seed region extraction unit 10 extracts a region satisfying predetermined ranges of luminance Y, saturation r, and hue θ as a seed region, and stores seed region data SD indicating the seed region in the region expansion unit 20. give.

Similar to the seed region extraction unit 10, the region expansion unit 20 uses the first color difference data C _b and the second color difference data C _r in the received input image data ID to calculate the saturation r and the hue θ using Equation (1). Ask for. Note that the calculation of the saturation r and the hue θ may be performed by the seed region extraction unit 10 and the region expansion unit 20 in common. The area expansion unit 20 acquires, as target areas, a seed area indicated by the seed area data SD and an area obtained by an expansion process described later based on the seed area. It should be noted that it is actually desirable to perform a predetermined correction process by the correction unit 30 on the target region acquired by the region expansion unit 20. In order to distinguish the target areas before and after such correction processing, the target area acquired by the area expansion unit 20 is hereinafter referred to as “target area before correction”, and after correction processing acquired by the correction unit 30. This target area is referred to as a “target area after correction”. The area expanding unit 20 provides the correction unit 30 with the pre-correction target area data EDc indicating the pre-correction target area.

The correction unit 30 performs a predetermined correction process on the pre-correction target area indicated by the pre-correction target area data EDc, thereby obtaining the post-correction target area, and outputs extracted data ED indicating the acquired target area to the outside.

<1.2 Extraction process>
FIG. 4 is a flowchart of the image processing program in the present embodiment. Steps S1 and S2 correspond to the seed region extraction unit 10, step S3 corresponds to the region expansion unit 20, and step S4 corresponds to the correction unit 30. The seed region extraction unit 10, the region expansion unit 20, and the correction unit 30 are realized in software by the CPU 1 executing each step of the image processing program shown in FIG.

FIG. 5 is a diagram showing an example in which the image processing program in the present embodiment is applied to a sample image. More specifically, FIG. 5A is a diagram showing a sample image. FIG. 5B is a diagram illustrating a processing result obtained by extracting a designated color region. Here, the “designated color” refers to the color of the corrected target area to be extracted. FIG. 5C is a diagram illustrating a processing result of extracting the seed region. FIG. 5D is a diagram illustrating a result of performing the extension process. FIG. 5E is a diagram illustrating a result of performing a filling process described later. In the sample image of FIG. 5A, a woman wearing a hat is depicted on the right side of the image, and a flower is depicted on the left side of the image. It should be noted that the sample image in FIG. 5A is actually a color image. 5B to 5E are, for example, binary images on display, but are handled as color images, for example, in various calculations related to these images. The following description will be made assuming that the face is extracted as the corrected target area from the sample image shown in FIG. 5A by the image processing program, but the present invention is not limited to this. This image processing program is applicable not only to the extraction of a region with a small pixel change (for example, skin other than the face, blue sky, or sea). This also applies to second and third embodiments described later. In addition, the “face” in the description relating to the sample image refers to a portion made up of a female face and neck in the sample image. In the following, the area acquired (extracted) in each step (the area shown in white in the images of FIGS. 5B to 5E) may be referred to as a “white area”. An area that has not been extracted (area shown in black in the images of FIGS. 5B to 5E) may be referred to as a “black area”.

First, in step S1, a specified color, for example, a skin color region is extracted from the sample image. In the present embodiment, as a condition for specifying a specified color (skin color) (hereinafter referred to as “specified color specifying condition”), a condition relating to a range of luminance Y, saturation r, and hue θ (hereinafter referred to as “range specifying condition”). Is used). Hereinafter, the ranges of luminance Y, saturation r, and hue θ are referred to as “luminance range”, “saturation range”, and “hue range”, respectively. The range specifying condition is given by the following equation (2).

Here, Y ₁ and Y ₂ are a lower limit threshold and an upper limit threshold of the luminance range for specifying the skin color, respectively. r ₁ and r ₂ are a lower limit threshold and an upper limit threshold of the saturation range for specifying the skin color, respectively. θ ₁ and θ ₂ are a lower limit threshold and an upper limit threshold of the hue range for specifying the skin color, respectively. FIG. 6 is a diagram schematically showing a range satisfying the saturation range and hue range shown in Expression (2) in the C _{r Cb} coordinates. In FIG. 6, a portion surrounded by a thick line is a range that satisfies the saturation range and hue range shown in Expression (2). The seed region extraction unit 10 determines that the range specifying condition is satisfied when all of the luminance range, saturation range, and hue range shown in Expression (2) are satisfied, and should extract a region that satisfies the range specifying condition Suppose that it is a skin color area. The extraction result is shown in FIG. FIG. 5B shows that in step S1, a part of the face that should be extracted is not extracted. This is because there is a region that does not satisfy the range specifying condition even in the face. It can also be seen that a part of the hat and a part of the flower that should not be extracted are extracted. This is because not only the face but also a hat and a flower have regions that satisfy the range specifying condition. This step S1 corresponds to, for example, the same processing as the conventional threshold area extraction.

Next, in step S2, a seed region is extracted from the white region (including not only the face but also part of the hat and part of the flower) extracted in step S1. More specifically, a region that satisfies the first predetermined condition regarding the area is extracted as a seed region from the white region extracted in step S1. The “first predetermined condition relating to the area” in the present embodiment is a condition in which the area is the maximum among the white regions extracted in step S1. For this reason, a region having the maximum area is extracted as a seed region from the white regions extracted in step S1. The extraction result is shown in FIG. Among the white areas extracted in step S1, the face area that is the area corresponding to the face (however, part of it is missing) has the largest area, so that the face area is extracted as the seed area. The Here, the region having the maximum area is extracted as the seed region, but the present invention is not limited to this. For example, the seed region can be extracted based on the area and / or shape of the region. As a method for determining the region shape, for example, a method for determining a region having a circularity of a predetermined value or more as a desired shape is known. The circularity is given by the following equation (3).
C = 4πS / L ² (3)
Here, C is the circularity, S is the area of the determination target region, and L is the perimeter of the region. The extraction of the seed region in consideration of the shape in this way is particularly suitable for extracting a region having a specific shape such as a face. In the process so far, the entire face area has not been completely extracted. For this reason, if a process such as face recognition is performed using an extraction result having such a lack, a malfunction may occur. However, the white balance can be adjusted as in the system according to Patent Document 2, using the color of the seed region obtained in step S2 as the face color.

Next, in step S3, it is determined whether or not a predetermined condition is satisfied in an area around the seed area obtained in step S2, and an area determined to satisfy the condition is included in the pre-correction target area, and Determining whether or not the condition is satisfied in an area around the area newly included in the pre-correction target area and including the area determined to satisfy the condition in the pre-correction target area. An area determined to satisfy and a seed area are acquired as a pre-correction target area. Thus, in step S3, not only the seed region but also the region determined to satisfy the predetermined condition is included in the target region. That is, processing as if the seed region is expanded is performed. For this reason, in this specification, the process of step S3 for including the area determined to satisfy the predetermined condition in the target area is referred to as an “expansion process”. The result of this expansion process is shown in FIG. FIG. 5D shows that the face area can be extracted more accurately than the seed area obtained in step S2. However, in the outer periphery of the pre-correction target area obtained in step S3, black areas corresponding to the inner periphery of the pre-correction target area and corresponding to the eyebrows, the face, the nose, the mouth, and the like are located. Details of step S3 will be described later.

Finally, in step S4, the above-described corrected target area is extracted by performing a so-called hole filling process in which the black area in contact with the inner periphery of the white area (target area before correction) is replaced with the white area. The hole filling process is a process of replacing, for example, a black area whose area is equal to or smaller than a predetermined value among black areas in contact with the inner periphery of the white area. The result after the hole filling process is shown in FIG. FIG. 5E shows that the entire face area has been extracted.

<1.3 Extended processing>
The expansion process is performed by determining whether or not a gradient specifying condition described later is satisfied, starting from each pixel in the seed region. More specifically, in the extension process, each pixel in the seed region is set as a target pixel (referred to as a central pixel in the determination of the gradient specifying condition performed in the extension process), and pixels around the target pixel are specified in the gradient. It is determined whether or not the condition is satisfied, and if the condition is satisfied, a process of including the determination target pixel in the pre-correction target area (hereinafter referred to as “first extension process”) and a first extension process are newly performed. Using a pixel included in the target region as a target pixel, determine whether or not pixels around the target pixel satisfy the gradient specifying condition, and if the condition is satisfied, include the target pixel in the pre-correction target region (Hereinafter referred to as “second extension process”).

FIG. 7 is a flowchart showing details of step S3 (extension processing) in FIG. As shown in FIG. 7, step S3 includes steps S31 to S34, S34a, S35, and S35a. Hereinafter, the horizontal direction of the image is referred to as “X direction”, and the vertical direction is referred to as “Y direction”. Also, variables representing the positions of the coordinates in the X direction and the Y direction (X coordinate and Y coordinate) are x and y, respectively. Note that x = 1 to m and y = 1 to n (m and n are integers of 1 or more). A flag is set for each pixel. The flag is 1 if the pixel belongs to the white area, and the flag is 0 if the pixel belongs to the black area. In the following, the flag value of each pixel is represented by F, and in particular, the flag value of a pixel whose X coordinate is x and Y coordinate is y (hereinafter referred to as a pixel of coordinates (x, y)) is F ( x, y).

After step S2 in FIG. 4, first, y is set to 1 in step S31. Next, x is set to 1 in step S32. In step S33, an expansion process is performed with the pixel at the coordinates (x, y) as the starting point. Details of step S33 will be described later. Next, in step S34, it is determined whether x = m. If x = m, the process proceeds to step S34a, and if x = m, the process proceeds to step S35. In step S34a, x is incremented, and then the process proceeds to step S33. In step S35, it is determined whether y = n. If y = n, the process proceeds to step S35a, and if y = n, the process proceeds to step S4 in FIG. In step S35a, y is incremented, and then the process proceeds to step S32.

Next, details of step S33 will be described. FIG. 8 is a flowchart showing details of step S33 in FIG. As shown in FIG. 8, step S33 includes steps S331 to S335. Of steps S331 to S335, steps S332 to 334 correspond to the first extension process, and steps S333 to S335 (more specifically, repetition of steps S333 to S335) correspond to the second extension process. 9 to 11 are diagrams for explaining an example of the extension process. 9 to 11, the rightward arrow indicates the X direction, and the downward arrow indicates the Y direction. Each block in FIGS. 9 to 11 shows one pixel. Here, for convenience of explanation, it is assumed that m = n = 20. In FIG. 9, dark blocks correspond to pixels with F = 1 (hereinafter referred to as “seed region pixels”), hatched blocks, dark blocks, and hatched blocks. Blocks other than (white blocks) correspond to pixels with F = 0 (hereinafter referred to as “black area pixels”). The hatched blocks also correspond to pixels that should be extracted as face regions but are not extracted as seed regions but are black region pixels (hereinafter referred to as “probable seed region pixels”). In FIG. 10, 1 is attached to the seed region pixel in FIG. In addition, although 0 should be attached to the black region pixel, it is omitted here for convenience. Further, the outer periphery of the expected seed region pixel in FIG. 9 is indicated by a broken line, and a range to be expanded (described later) is indicated by a thick line. Note that the description regarding FIG. 10 is the same with respect to FIG. 11 except for the pixel at the coordinates (12, 7) and (11, 8).

First, in step S331, it is determined whether F (x, y) = 1. If F (x, y) = 1, the process proceeds to step S34 in FIG. 7, and if F (x, y) = 1, the process proceeds to step S332. In the example shown in FIG. 10, while step S33 is repeated for each pixel, the process proceeds to step S332 for the first time at the pixel of coordinates (12, 8).

In step S332, whether or not there is a pixel with F = 0 in the vicinity of the pixel with the coordinate (x, y), that is, F = 0 during the extension processing target with the pixel with the coordinate (x, y) as the target pixel. It is determined whether or not there are pixels. If there is no pixel with F = 0, the process proceeds to step S34 in FIG. 7, and if there is a pixel with F = 0, the process proceeds to step S333. Here, for example, as shown in FIG. 12A, the extension processing target can be four neighboring neighboring pixels of the pixel of interest, or eight neighboring neighboring pixels of the pixel of interest as shown in FIG. can do. Note that in FIG. 12A and FIG. 12B, a pixel with F = (i, j) surrounded by a thick line is a target pixel (i = 1 to m, j = 1 to n). In the following description, it is assumed that the extension processing target is four neighboring adjacent pixels of the target pixel. However, the same description holds when the extension processing target is eight neighboring neighboring pixels of the target pixel. In the example shown in FIG. 10, the pixel at the coordinate (12, 8) is the target pixel, and the coordinates (12, 7), (11, 8), (13, 8), and (12) that are the four neighboring pixels. , 9) is an extension processing target.

Before describing step S333, the gradient of luminance Y, saturation r, and hue θ between adjacent pixels will be described. In the following, the gradients of luminance Y, saturation r, and hue θ are referred to as “luminance gradient”, “saturation gradient”, and “hue gradient”, respectively, and are denoted by Y ′, r ′, θ ′, respectively. Represented by For calculation of these gradients, for example, a Roberts filter, a pre-witt filter, or a Sobel filter can be used. FIG. 13A is a diagram illustrating a 45-degree Roberts filter. FIG. 13B is a diagram illustrating a 135 degree direction Roberts filter. FIG. 13C is a diagram showing a pre-wit filter in the horizontal direction. FIG. 13D is a diagram illustrating a pre-wit filter in the vertical direction. FIG. 13E shows a horizontal Sobel filter. FIG. 13F is a diagram illustrating a vertical Sobel filter. Although the filter size is 3 × 3 here, the present invention is not limited to this.

The luminance gradient Y ′, the saturation gradient r ′, and the hue gradient θ ′ are calculated by the following equation (4).

Here, Y _h ′ is a luminance gradient in the first direction (45 ° direction or horizontal direction), and is calculated by the filter shown in FIG. 13A when the Roberts filter is used, and when the pre-wit filter is used. Is calculated by the filter shown in FIG. 13C, and when using a Sobel filter, it is calculated by the filter shown in FIG. Y _v ′ is a luminance gradient in the second direction (135 ° direction or vertical direction), and is calculated by the filter shown in FIG. 13B when the Roberts filter is used, and when the pre-wit filter is used. It is calculated by the filter shown in FIG. 13D, and when using a Sobel filter, it is calculated by the filter shown in FIG.

In step S333, the pixel with F = 0 as the extension process target is a condition relating to the luminance gradient Y ′, the saturation gradient r ′, and the hue gradient θ ′ given by the following equation (5) (hereinafter referred to as “gradient specifying condition”). .) Is satisfied.

Here, Y _a ′, r _a ′, and θ _a ′ are respectively the threshold value of the luminance gradient Y ′, the threshold value of the saturation gradient r ′, and the hue gradient θ ′ for specifying pixels having relatively gentle gradients. It is a threshold value. These threshold values correspond to the first threshold value. As shown in Equation (5), the luminance gradient Y 'is the threshold Y _a' smaller than the saturation gradient r 'is the threshold r _a' smaller than that when the color gradient theta 'threshold theta _a' smaller than the extended It is determined that the pixel of processing target F = 0 satisfies the gradient specifying condition. Satisfying this gradient specifying condition means that the gradient of the pixel of F = 0 that is the target of the extension processing is relatively gentle. For example, an area with small pixel change such as a face has a relatively gentle gradient. Therefore, a pixel of F = 0 that satisfies the gradient specifying condition can be regarded as a potential seed area pixel. For example, in the example shown in FIG. 10, the pixels of coordinates (12, 7) and (11, 8) satisfy the gradient specifying condition shown in equation (5). In addition, about the pixel of coordinates (13,8) and (12,9), since F = 1, it is not included in the determination object whether the gradient specific condition shown in Formula (5) is satisfied. If the pixel of F = 0 that is subject to the expansion process does not satisfy Expression (5), the process proceeds to Step S34 in FIG. 7, and if the expression (5) is satisfied, the process proceeds to Step S334. When there are a plurality of pixels with F = 0 in the expansion processing target, for example, steps S333 to S335 are performed for each pixel or performed in parallel for the plurality of pixels. In the following, for the sake of convenience of explanation, steps S333 to S335 are described in parallel for a plurality of pixels with F = 0.

In step S334, the flag is converted to F = 1 for the pixel of F = 0 that satisfies Expression (5). For this reason, for the pixels at coordinates (12, 7) and (11, 8) satisfying the expression (5), the flag is converted to F = 1 as shown in FIG. That is, the pixels at coordinates (12, 7) and (11, 8) that are regarded as the expected seed region pixels are included in the pre-correction target region. This is the first extension process.

In step S335, whether or not there is a pixel with F = 0 in the vicinity of the pixel whose flag is converted to F = 1, that is, there is a pixel with F = 0 in the extension processing target with the pixel as the target pixel. It is determined whether or not. If there is no pixel with F = 0, the process proceeds to step S34 in FIG. 7, and if there is a pixel with F = 0, the process proceeds to step S333. In this way, Steps S333 to S335 are repeatedly performed using a pixel whose flag is newly converted to F = 1 (an expected seed region pixel included in the pre-correction target region) as a target pixel. Then, when there is no pixel with F = 0 that satisfies Expression (5) in the extension process target with the pixel whose flag is newly converted to F = 1 as the target pixel, the second extension process is stopped. (Proceed from step S333 to step S34). Then, such processing in step S33 is performed over the entire image (step S3). As a result, the flag is converted to F = 1 for almost all the expected seed area pixels (that is, the expected seed area pixels excluding the expected seed area pixels located in the closed space of the pre-correction target area). Processing as if expanded. In this way, the pre-correction target area is obtained. In addition, since the flag is not converted to F = 1 for the pixel of F = 0 that does not satisfy Expression (5), the flag of the black region pixel that is not the expected seed region pixel is not converted to F = 1. That is, the expansion process is stopped at a location where the gradient changes suddenly (a boundary between regions where the pixel change is small).

<1.4 Effect>
According to the present embodiment, the expansion process is performed based on a seed region that is a region that satisfies a predetermined specified color specifying condition. In the conventional threshold area extraction, since this kind of area is set as the target area to be extracted, the target area to be extracted originally lacks. However, in the present embodiment, the expected seed region pixel is included in the target region by the expansion process. For this reason, it is possible to extract almost the entire target region without missing the target region to be extracted. Accordingly, various image processing such as face recognition or face color correction can be appropriately performed using the extracted target region. In addition, since high-cost recognition technology such as SVM (Support Vector Machine) is not used, the cost can be reduced.

Further, according to the present embodiment, the omission of the target region to be extracted can be more sufficiently suppressed by performing the hole filling process.

<1.5 Modification>
In the first embodiment, only the range specifying condition is used as the specified color specifying condition in step S1, but a gradient specifying condition may be further used. That is, the specified color specifying condition may be satisfied when both the range specifying condition and the gradient specifying condition are satisfied. Hereinafter, in order to distinguish the range specifying condition used as the specified color specifying condition from the gradient specifying condition used in the expansion process in step S333, the former is referred to as “extraction range specifying condition” and the latter is referred to as “extended use specifying condition”. This is called “range identification condition”. The expansion range specifying condition is given by the equation (5) as described above. On the other hand, the extraction range specifying condition is given by the following equation (6).

Here, Y _b ′, r _b ′, and θ _b ′ are respectively a luminance gradient Y ′ threshold, a saturation gradient r ′ threshold, and a hue gradient θ ′ for specifying a relatively gentle pixel. It is a threshold value. These threshold values correspond to the second threshold value. The relationship between the threshold value related to the extraction range specifying condition and the threshold value related to the expansion range specifying condition is given by the following equation (7).

As can be seen from Expression (7), the extraction range specifying condition is set more strictly than the extended range specifying condition.

As described above, by adding the extraction range specifying condition to the specified color specifying condition, an area having a large gradient, that is, an area other than an area having a small pixel change such as a face can be excluded from the seed area. In addition, by setting the extraction range specifying condition more strictly than the extended range specifying condition, it is possible to more reliably exclude regions other than the region where the pixel change is small, such as a face, from the seed region. For this reason, a more accurate (that is, less noise) seed region can be used in the expansion process. Thereby, the extraction accuracy of the target region can be increased. Note that, not limited to this modification, the extraction range specifying condition may be the same as the expansion range specifying condition. That is, Y _a ′ = Y _b ′, r _a ′ = r _b ′, and θ _a ′ = θ _b ′ may be set.

<2. Second Embodiment>
<2.1 Extraction process>
FIG. 14 is a flowchart of an image processing program according to the second embodiment of the present invention. As shown in FIG. 14, step S2 in the first embodiment is replaced with step S2a, and step S5 is further added. Since the configuration of the image processing apparatus is the same as that of the first embodiment, description thereof is omitted. Also, with regard to the extraction process, description of parts common to the first embodiment will be omitted as appropriate. Step S2a corresponds to the seed region extraction unit 10 as in step S2 in the first embodiment. Step S5 corresponds to the area expansion unit 20, for example. FIG. 15 is a diagram illustrating an example in which the image processing program in the present embodiment is applied to a sample image. More specifically, FIG. 15A shows a sample image. FIG. 15B is a diagram illustrating a processing result of extracting a specified color area. FIG. 15C is a diagram illustrating a processing result of extracting the seed region. FIG. 15D is a diagram illustrating a result of performing the extension process. FIG. 15E is a diagram illustrating a result of the hole filling process. The sample image is the same as that in the first embodiment, that is, FIG. 15A is the same as FIG. 5A. 15B and 15E are the same as FIGS. 5B and 5E, respectively.

In step S2a, a region that satisfies the first predetermined condition regarding the area is extracted as a seed region from the white region extracted in step S1. However, unlike step S2 in the first embodiment, the “first predetermined condition regarding the area” in the present embodiment is a condition in which the area is equal to or larger than a predetermined value in the white region extracted in step S1. It is. For this reason, a region having an area equal to or larger than a predetermined value is extracted as a seed region from the white regions extracted in step S1. The extraction result is shown in FIG. Here, for example, a region having an area equal to or larger than a predetermined value is a face region (however, part thereof is missing) and a hat region (however, part thereof is missing) corresponding to the hat. ). Thus, in the present embodiment, a plurality of seed regions can be extracted according to the setting of the area value of the region to be extracted.

In step S3, processing similar to that in the first embodiment is performed based on the various areas obtained in step S2a. The processing result is shown in FIG. From FIG. 15D, the hat region can be extracted more accurately than the seed region obtained in step S2 together with the face region similar to that in the first embodiment.

Next, in step S5, a region to be selected as a pre-correction target region is selected from the white regions extracted in step S3. The “second predetermined condition relating to the area” in the present embodiment is a condition in which the area is the maximum among the white regions extracted in step S3. For this reason, the face area is a target area before correction, and the hat area, which is an unnecessary area, is excluded from the target area before correction. In step S5 as well, the pre-correction target region may be selected based on the area and / or shape of the region, and the circularity may be used as described above. In particular, the selection of the pre-correction target area using the circularity is suitable for extracting a plurality of faces and the like.

And in step S4, the same as the first embodiment, the filling process is performed on the pre-correction target area obtained in step S5, and the post-correction target area is obtained. This result is shown in FIG. FIG. 15E shows that the same result as in the first embodiment can be obtained.

<2.2 Effect>
According to the present embodiment, in the case where a region having an area of a predetermined value or more is extracted as a seed region and a region having an area of a predetermined value or more and / or a circularity of a predetermined value or more is set as the pre-correction target region, The same effect as that of the first embodiment can be obtained. Here, it is considered that the image processing program according to the present embodiment is applied to an image in which a plurality of faces are depicted instead of the sample image shown in FIG. In this case, according to the image processing program in the present embodiment, a plurality of seed regions can be extracted, and a plurality of pre-correction target regions can be extracted. For this reason, for example, a plurality of faces can be extracted from an image in which a plurality of faces are depicted. That is, this embodiment is suitable for an aspect in which there are a plurality of target regions to be extracted.

<3. Third Embodiment>
<3.1 Configuration>
FIG. 16 is a block diagram illustrating a functional configuration of a main part of an image processing apparatus according to the third embodiment of the present invention. As shown in FIG. 16, this image processing apparatus is obtained by adding an RGB-YC _{b Cr} conversion unit 40 to the image processing apparatus according to the first embodiment. The RGB-YC _{b Cr} conversion unit 40 is realized by the CPU 1 as software. Since other configurations and operations are the same as those in the first embodiment, description thereof is omitted. The image processing apparatus according to the present embodiment extracts a target area from an image indicated by image data received from a multimedia system. Therefore, unlike the first embodiment, in this embodiment, the input image data ID is composed of R data, B data, and G data as primary color signals.

R data input to the image processing apparatus, B data, and G data converted luminance data Y by RGB-YC _b C _r conversion unit 40, first color difference data C _b, and the second color difference data C _r Thereafter, it is given to the seed region extraction unit 10 and the region expansion unit 20. Here, the signal conversion in the RGB-YC _{b Cr} converter 40 is performed based on the following equation (8).

The operations of the seed region extraction unit 10, the region expansion unit 20, and the correction unit 30 are the same as those in the first embodiment.

<3.2 Effects>
According to the present embodiment, when the target area is extracted from the image indicated by the image data received from the multimedia system, the same effect as in the first embodiment can be obtained.

<4. Other>
The modification of the first embodiment may be combined with the second embodiment or the third embodiment. The third embodiment may be combined with the second embodiment. In the above description, the seed region extraction unit 10, the region expansion unit 20, the correction unit 30, and the RGB-YC _{b Cr} conversion unit 40 are realized by software. May be realized by hardware. Further, in the third embodiment, R data, B data, and G data luminance data Y, first color difference data C _b, and after converting the second color difference data C _r, the saturation r and hue θ Although it has been determined, the present invention is not limited to this. For example, luminance Y, saturation r, and hue θ may be obtained directly from R data, B data, and G data. In addition, the above-described embodiments can be variously modified and implemented without departing from the spirit of the present invention.

As described above, according to the present invention, it is possible to provide an image processing apparatus, an image processing method, an image processing program, and a recording medium storing the image processing program that can extract an area with small pixel change from an image at low cost. .

The present invention can be applied to an image processing apparatus, an image processing method, an image processing program, and a recording medium storing an image processing program that extract a region of a specific part of an object represented by the input image from the input image.

1 ... CPU
2 ... input unit 3 ... memory 4 ... hard disk 5 ... output section 6 ... image processing program 10 ... seed region extraction section 20 ... area expansion unit 30 ... correcting section 40 ... RGB-YC _b C _r conversion unit (signal format conversion portion)
ID: input image data SD: seed region data EDc: target region data ED before correction: extraction data

Claims (19)

1. An image processing apparatus that extracts a target area from an input image,
   A seed region extraction unit that extracts, as a seed region, a region that satisfies at least conditions relating to a predetermined range of luminance, saturation, and hue from the input image;
   It is determined whether or not the gradients of luminance, saturation, and hue in the peripheral region of the seed region are less than or equal to a first threshold for luminance, saturation, and hue, respectively, and the gradient is less than or equal to the first threshold. The region determined to be included in the target region and whether or not the gradient in the region around the region newly included in the target region is equal to or less than the first threshold value is determined. Including an area determined to be equal to or less than a first threshold, and an area expansion unit that acquires the area where the gradient is equal to or less than the first threshold and the seed area as the target area. An image processing apparatus as a feature.
2. 2. The image processing apparatus according to claim 1, wherein the area expanding unit determines whether or not the gradient is equal to or less than a first threshold from each pixel in the seed area.
3. The area extension is
   Using each pixel in the seed region as a pixel of interest, it is determined whether or not the gradient in the surrounding pixels of the pixel is equal to or less than the first threshold value. If the determination result is equal to or less than the first threshold value, the pixel to be determined A first expansion process for including
   A pixel newly included in the target region is set as a pixel of interest, and it is determined whether or not the gradient of pixels around the pixel is equal to or less than the first threshold. If the determination result is equal to or less than the first threshold, the determination is made. The image processing apparatus according to claim 2, wherein a second extension process that repeats including a target pixel in the target area is performed.
4. The image processing according to any one of claims 1 to 3, wherein the seed region extraction unit extracts, as the seed region, a region that further satisfies a condition that the gradient is equal to or less than a predetermined value. apparatus.
5. The image processing apparatus according to claim 4, wherein the predetermined value is a second threshold value that is smaller than the first threshold value.
6. When there are a plurality of regions that satisfy the conditions regarding the luminance, saturation, and hue ranges, the seed region extraction unit selects a region that satisfies a first predetermined condition regarding the area from the plurality of regions. The image processing apparatus according to claim 1, wherein the image processing apparatus is extracted as a region.
7. The first predetermined condition relating to the area is a condition that the area is a maximum among a plurality of regions that satisfy the conditions relating to the luminance, saturation, and hue ranges. The image processing apparatus described.
8. The image processing apparatus according to claim 6, wherein the first predetermined condition relating to the area is a condition that the area is equal to or greater than a predetermined value.
9. When the region expansion unit obtains a plurality of regions each composed of the region determined to have the gradient equal to or less than the first threshold and the seed region, the region expansion unit The image processing apparatus according to claim 8, wherein an area that satisfies the predetermined condition of 2 is a target area to be extracted.
10. 10. The image processing apparatus according to claim 9, wherein the second predetermined condition relating to the area is a condition that the area is the maximum among the plurality of regions.
11. 4. The image processing apparatus according to claim 1, further comprising a correction unit that performs a predetermined correction process on the target area acquired by the area expansion unit.
12. The predetermined correction process is a process of including, in the target area, an area whose area is equal to or less than a predetermined value among the areas that are not the target area in contact with the inner periphery of the target area acquired by the area expanding unit. The image processing apparatus according to claim 11, wherein the image processing apparatus is characterized.
13. The image processing apparatus according to any one of claims 1 to 3, wherein the input image is represented by a luminance signal and a color difference signal.
14. The image processing apparatus according to claim 13, further comprising a signal format conversion unit that converts an input image indicated by a primary color signal into the input image indicated by a luminance signal and a color difference signal.
15. An image processing method for extracting a target area from an input image,
    A seed region extraction step for extracting, from the input image, a region that satisfies at least the conditions relating to the predetermined luminance, saturation, and hue range as a seed region;
    It is determined whether or not the gradients of luminance, saturation, and hue in the peripheral region of the seed region are less than or equal to a first threshold for luminance, saturation, and hue, respectively, and the gradient is less than or equal to the first threshold. The region determined to be included in the target region and whether or not the gradient in the region around the region newly included in the target region is equal to or less than the first threshold value is determined. Including an area determined to be equal to or lower than a first threshold, and an area expansion step for acquiring the area determined to be equal to or lower than the first threshold and the seed area as the target area. A characteristic image processing method.
16. 16. The image processing method according to claim 15, wherein, in the region expansion step, the determination as to whether or not the gradient is equal to or less than a first threshold is performed starting from each pixel in the seed region.
17. The region expansion step includes
    Using each pixel in the seed region as a pixel of interest, it is determined whether or not the gradient in the surrounding pixels of the pixel is equal to or less than the first threshold value. If the determination result is equal to or less than the first threshold value, the pixel to be determined A first expansion step of including in the region of interest;
    A pixel newly included in the target region is set as a pixel of interest, and it is determined whether or not the gradient of pixels around the pixel is equal to or less than the first threshold. If the determination result is equal to or less than the first threshold, the determination is made. The image processing method according to claim 16, further comprising a second extension step of repeatedly including a target pixel in the target region.
18. An image processing program causing a computer to execute each step in the image processing method according to any one of claims 15 to 17.
19. A computer-readable recording medium on which the image processing program according to claim 18 is recorded.

Images