JP2003196649A - Image processing device and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing device capable of realizing highly- accurate heightening of image quality to a multi spectral image easily by hardware, and an image processing program for realizing the highly-accurate heightening of the image quality by a computer. <P>SOLUTION: A first direction coefficient pattern, a second direction coefficient pattern, a third direction coefficient pattern and a fourth direction coefficient pattern are recorded. A first coefficient pattern group comprising three or more kinds of coefficient patterns is generated by using the first direction coefficient pattern and the second direction coefficient pattern, and a second coefficient pattern group comprising three or more kinds of coefficient patterns is generated by using the third direction coefficient pattern and the fourth direction coefficient pattern. A third coefficient pattern group comprising nine or more kinds of coefficient patterns is generated by composing the first and the second coefficient pattern groups, and weighted addition of color information of the multi spectral image is performed by using one coefficient pattern in the third coefficient pattern group, to thereby generate a brightness component. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multispectral image represented by a plurality of colors and having one color information for each pixel arranged in a predetermined state. The present invention relates to an image processing device that performs image processing that generates high-quality color components, and an image processing program that causes the computer to implement the image processing.

[0002]

2. Description of the Related Art Conventionally, an image pickup device in which color filters of a plurality of colors are arranged at predetermined positions has been used to generate color image data. Image data generated by such an image pickup device is used. Has only one color component of color information for each pixel. Therefore, such image data is generally subjected to image processing for assuring high image quality by associating each pixel with color information of at least one common color component.

For example, in US Pat. No. 5,901,242, image pickup in which three color filters of R, G, B (red, green, blue) are arranged at a ratio of 3: 4: 1. For the image data generated by the element, R, G,
A technique of image processing (so-called interpolation processing) that associates the color information of the three color components B is disclosed. In the technology disclosed in US Pat. No. 5,901,242,
Although the color information of the color component missing in each pixel is calculated, such color information is calculated by performing weighted addition of color information arranged within the range shown below by a predetermined coefficient. It is realized by

When calculating the color information of the red component in a pixel having the color information of the green component: a range of 9 pixels × 9 pixels centered on the pixel. When calculating the color information of the blue component in the pixel having the color information of the green component: 9 pixels centered on the pixel x
A range of 7 pixels. When calculating the color information of the green component in the pixel having the color information of the red component: a range of 11 pixels × 11 pixels to 9 pixels × 9 pixels with the pixel as the center.

When calculating the color information of the blue component in a pixel having the color information of the red component: a range of 7 pixels × 7 pixels to 5 pixels × 5 pixels with the pixel as the center. When calculating the color information of the green component in a pixel having the color information of the blue component: 9 pixels centered on that pixel x
A range of 9 pixels.

When calculating the color information of the red component in a pixel having the color information of the blue component: a range of 9 pixels × 9 pixels centered on the pixel. The weighted addition of color information arranged in the range as described above is first performed using a predetermined coefficient pattern. For example, when calculating the color information of the green component in a pixel having the color information of the red component, a coefficient pattern as shown in FIG. 26 is used over the entire image. Then, if the result of the color information generated in this way is poor, the coefficient pattern is adjusted, and the adjusted coefficient pattern is applied to the entire image again to generate color information again,
After that, the adjustment of the coefficient pattern and the recalculation of the color information by the weighted addition are repeatedly performed until the appropriate color information is generated.

That is, in the technique disclosed in US Pat. No. 5,901,242, the weighted addition coefficient to be used when generating the color information missing in each pixel is
It will be decided by the iterative process by trial and error. Further, Japanese Patent Laid-Open No. 2000-78597 (US Pat. No. 6,075,889) describes that image data generated by an image pickup device in which color filters of three colors R, G, B are arranged in a Bayer array. On the other hand, an image processing technique for generating a luminance value and a chrominance value for each pixel is disclosed.

In the technique disclosed in US Pat. No. 5,901,242, the luminance value is generated by using the following processing 1 to processing 6, and the chrominance value refers to the luminance value. Generated. << Process 1 >>: Calculation of "blurred luminance value". The high-frequency component is removed from the image data generated by the image sensor, and the "blurred luminance value" is calculated.

<< Process 2 >>: Calculation of "classifier" (similarity). Based on the “blurred brightness value”, the degrees of similarity in the horizontal direction, the vertical direction, the diagonal direction 1, and the diagonal direction 2 are calculated.

<< Process 3 >>: Judgment of similarity. The strength of similarity is determined based on the degree of similarity. According to the similarity determination result, each pixel has five similar directions (horizontal direction, vertical direction, flat, diagonal direction 1,
It is classified into any one of diagonal directions 2). << Process 4 >>: Generation of luminance value.

Of the coefficient patterns shown in FIGS. 27 (a) to 27 (e), the coefficient pattern corresponding to the similar direction is subjected to weighted addition to obtain a brightness value (corresponding to the "predicted brightness value") for each pixel. ) Is generated. << Process 5 >>: Calculation of correction value. A “correction value” for correcting the brightness value is calculated for the horizontal direction, the vertical direction, the oblique direction 1, and the oblique direction 2 using the “blurred brightness value”.

<< Process 6 >>: Correction of luminance value. The brightness value is selectively corrected using the correction value in accordance with the similar direction classification result of each pixel.

[0013]

As described above,
In the case of implementing the image processing for associating the color information of the three color components of R, G, B with each pixel by the technique disclosed in US Pat. No. 5,901,242, as described above, It is necessary to prepare in advance a large number of coefficient values for weighted addition of a wide range of color information. Therefore,
When implemented by hardware, it is necessary to previously provide a mechanism for implementing weighted addition over a wide range for each of a plurality of coefficient patterns, which causes a problem that the circuit scale becomes large.

In the technique disclosed in US Pat. No. 5,901,242, when adjusting a coefficient pattern prepared in advance by iterative processing by trial and error, it cannot be realized by hardware. near.

The coefficient pattern used in the technique disclosed in US Pat. No. 5,901,242 is
Since it is fixed over the entire image and the directional similarity depending on the location is not reflected, it is highly possible that high image quality cannot be realized with high accuracy. On the other hand, in the technique disclosed in Japanese Unexamined Patent Publication No. 2000-78597 (US Pat. No. 6,075,889), in the process 4, the weighting by the coefficient pattern of any one of FIGS. Although the luminance value is generated by the addition, it is clear that the luminance value is generated by using the two-component color information, as is clear from the observation of the pixel having the red component color information and the blue component color information. Basically, especially in the coloring part, the RGB ratio of the generated luminance values fluctuates drastically every time the pixel for which the luminance value is generated changes, and the luminance generated even between adjacent pixels. It is highly likely that the values will not connect smoothly.

Further, the calculation of the correction value for the brightness value is performed by switching for each similar direction in which each pixel is classified. Therefore, when such processing is realized by hardware, it is necessary to calculate correction values for all directions in advance, which causes a problem that the circuit scale of hardware becomes large. Further, in the process 2, the similarity is calculated based on the “blurred luminance value”, and thus the resolution performance in a fine portion is deteriorated and is inaccurate.

Based on the above, the inventions described in claims 1 to 36 provide an image processing apparatus capable of easily realizing high-quality high-quality image processing for a multispectral image by hardware. For the purpose of
It is an object of the present invention to provide an image processing program for causing a computer to implement image processing for highly accurately improving the image quality of a multispectral image.

[0018]

An image processing apparatus according to claim 1 is a multispectral image represented by a plurality of colors, in which one color information is present in each pixel arranged in a two-dimensional square lattice. On the other hand, in the image processing device that performs the image processing for generating the luminance component of the color system different from the color system of the multi-spectral image, the first coefficient indicating the weighted addition for the color information arranged in the first direction A unidirectional coefficient pattern, and a second directional coefficient pattern indicating a weighted addition coefficient for color information arranged in a second direction orthogonal to the first direction,
A third direction coefficient pattern indicating a weighted addition coefficient for color information arranged in a third direction different from the first direction and the second direction, and a fourth direction orthogonal to the third direction. Recording means for recording a fourth directional coefficient pattern indicating a coefficient for weighted addition for the color information arranged in the above, and at least three types of coefficient patterns using the first directional coefficient pattern and the second directional coefficient pattern. A second coefficient pattern composed of at least three types of coefficient patterns by using a first pattern generating means for generating a first coefficient pattern group consisting of, and the third direction coefficient pattern and the fourth direction coefficient pattern. A second pattern generating means for generating a group, the first coefficient pattern group and the second coefficient pattern group are combined to form a third pattern including at least nine types of coefficient patterns. Luminance that generates a luminance component by weighted addition of color information of the multispectral image using a third pattern generation unit that generates a coefficient pattern group and any one coefficient pattern of the third coefficient pattern group. And a component generating means.

An image processing apparatus according to a second aspect is the image processing apparatus according to the first aspect, wherein the first pattern generating means has two of the first direction coefficient pattern and the second direction coefficient pattern. It is characterized in that a first coefficient pattern group consisting of a coefficient pattern of two kinds and a coefficient pattern obtained by averaging the two kinds of coefficient patterns is generated.
The image processing apparatus according to claim 3 is the image processing apparatus according to claim 1, wherein the second pattern generation means is
Generating a second coefficient pattern group consisting of two kinds of coefficient patterns, the third direction coefficient pattern and the fourth direction coefficient pattern, and a coefficient pattern obtained by averaging the two kinds of coefficient patterns. Characterize.

An image processing apparatus according to a fourth aspect is the image processing apparatus according to the first aspect, wherein the third pattern generating means comprises at least three types of coefficient patterns forming the first coefficient pattern group. And at least three types of coefficient patterns forming the second coefficient pattern group,
By averaging in combination, said at least 9
It is characterized in that it generates different types of coefficient patterns.

According to a fifth aspect of the present invention, in the image processing apparatus according to the first aspect, the degree of similarity regarding the first direction and the degree of similarity regarding the second direction are calculated, and the degree of similarity is calculated. A first similarity determination means for determining the similarity between the first direction and the second direction in at least three stages based on
The first direction coefficient pattern and the second direction coefficient pattern are selected according to the difference in similarity that can be determined by the first similarity determination means.
It is characterized in that the first coefficient pattern group is generated by synthesizing with the direction coefficient pattern.

According to a sixth aspect of the present invention, in the image processing apparatus according to the first aspect, the similarity regarding the third direction and the similarity regarding the fourth direction are calculated, and the similarity is calculated. A second similarity determining means for determining the similarity between the third direction and the fourth direction in at least three stages, wherein the second pattern generating means comprises:
The third direction coefficient pattern and the fourth direction coefficient pattern are generated according to the difference in similarity that can be determined by the second similarity determination means.
It is characterized in that the second coefficient pattern group is generated by synthesizing with the direction coefficient pattern.

An image processing apparatus according to a seventh aspect is the image processing apparatus according to the first aspect, wherein the recording means displays the multi-spectral image in a color system including first to third color components. And the spatial density of the first color component is higher than the spatial densities of the second color component and the third color component, the first direction coefficient pattern is arranged in the first direction. A coefficient pattern for averaging color information of one color component is recorded, and as the second direction coefficient pattern, a coefficient pattern for averaging color information of the first color component arranged in the second direction. Is recorded, and second patterns arranged in the third direction are used as the third direction coefficient pattern.
And a coefficient for weighted averaging the color information of the third color component is recorded, and as the fourth direction coefficient pattern, the fourth direction coefficient pattern is recorded.
The coefficient for weighted averaging the color information of the second and third color components arranged in the direction is recorded.

An image processing apparatus according to an eighth aspect is the image processing apparatus according to the first aspect, wherein the recording means records a coefficient pattern containing no negative value. According to a ninth aspect of the present invention, in the image processing apparatus according to the first aspect, the luminance component generating means generates a luminance component at the same pixel position as the multispectral image.

An image processing apparatus according to a tenth aspect is the image processing apparatus according to the first aspect, wherein the recording means is
As a coefficient pattern forming at least one of the first coefficient pattern group and the second coefficient pattern group, the coefficient of the weighted addition corresponding to the color information of the luminance component generation target pixel position is a coefficient showing a positive value. It is characterized by recording a pattern.

An image processing apparatus according to an eleventh aspect is the image processing apparatus according to the first aspect, wherein the recording means is
It is characterized by recording a coefficient pattern for weighted addition of color information in the narrowest range including color information of all color components constituting the color system of the multi-spectral image centered on a luminance component generation target pixel And The image processing apparatus according to claim 12 is the image processing apparatus according to claim 1, further comprising a correction unit that performs edge enhancement processing by a fixed filter on the luminance component generated by the luminance component generation unit. Is characterized by.

An image processing apparatus according to a thirteenth aspect is the image processing apparatus according to the first aspect, further comprising color difference component generating means for generating a color difference component using the color information of the multispectral image. Characterize. The image processing apparatus according to claim 14 is the image processing apparatus according to claim 13, wherein the color difference component generation means calculates similarity in at least two directions to determine similarity, and determines the similarity. The color difference component is calculated based on the result.

An image processing apparatus according to a fifteenth aspect is the image processing apparatus according to the fourteenth aspect, wherein the similarity is
It is characterized in that it is a similarity between different colors composed of color information between different color components of the multispectral image. The image processing apparatus according to claim 16, wherein a multispectral image represented by a plurality of colors and having one color information in each of the pixels arranged in a two-dimensional grid is used to represent the color of the multispectral image. In an image processing device that performs image processing for generating a luminance component of a color system different from the system, the luminance component generation target pixel is connected to the closest pixel arranged at a position closest to the luminance component generation target pixel. n indicating a coefficient of weighted addition for color information arranged in each of n ₁ (n ₁ ≧ 2) directions
And _one coefficient pattern, the luminance component subject pixels and luminance components subject pixels connecting the second closest pixels arranged in a position closer to the next of the closest pixels for n ₂ (n ₂ ≧
2) recording means for recording the number of direction _two n indicating the coefficients of the weighted addition for the color information arranged in each of the coefficient pattern, using the n ₁ coefficients patterns, m ₁ (m ₁
M ₂ (m ₁ > n ₂ ) coefficients using the first pattern generating means for generating a first coefficient pattern group composed of> n ₁ ) coefficient patterns and the n ₂ coefficient patterns. A second pattern generating means for generating a second coefficient pattern group including a pattern, the first coefficient pattern group and the second coefficient pattern group are combined to obtain L (L> m ₁ , and , L> m ₂ ) and a third pattern generating means for generating a third coefficient pattern group consisting of the coefficient patterns,
Luminance coefficient generating means for generating the luminance component by performing weighted addition of the color information of the multi-spectral image using any one of the coefficient pattern groups.

An image processing apparatus according to a seventeenth aspect is the image processing apparatus according to the sixteenth aspect, wherein the first pattern generating means includes the n ₁ coefficient patterns and the n ₁ coefficient patterns. It is characterized in that a first coefficient pattern group consisting of at least one coefficient pattern obtained by averaging between is generated.

The image processing apparatus according to claim 18 is the image processing apparatus according to claim 16, wherein the second pattern generating means includes the n ₂ coefficient patterns and the n ₂ coefficient patterns. It is characterized in that a second coefficient pattern group consisting of at least one coefficient pattern obtained by averaging between is generated. The image processing device according to claim 19 is the image processing device according to claim 16, wherein the third pattern generation unit includes m ₁ coefficient patterns that form the first coefficient pattern group, and It is characterized in that m ₁ × m ₂ coefficient patterns are generated as the L coefficient patterns by combining and averaging m ₂ coefficient patterns forming the second coefficient pattern group. .

An image processing apparatus according to a twentieth aspect is the image processing apparatus according to the sixteenth aspect, wherein the first pattern generation means connects the luminance component generation target pixel and the closest pixel to n ₁ The first coefficient pattern group is generated by synthesizing the n ₁ coefficient patterns according to the difference in similarity between the respective directions. Claim 2
The image processing apparatus according to item 1 is the image processing apparatus according to claim 16, wherein the second pattern generation unit connects the luminance component generation target pixel and the second adjacent pixel to n ₂
The second coefficient pattern group is generated by combining the n ₂ coefficient patterns according to the difference in similarity between the two directions.

An image processing apparatus according to a twenty-second aspect is the image processing apparatus according to the sixteenth aspect, wherein the recording means records a coefficient pattern containing no negative value. The image processing apparatus according to a twenty-third aspect is the image processing apparatus according to the sixteenth aspect, characterized in that the luminance component generating means generates a luminance component at the same pixel position as the multispectral image.

An image processing apparatus according to a twenty-fourth aspect is the image processing apparatus according to the sixteenth aspect, wherein the recording means includes at least one of the first coefficient pattern group and the second coefficient pattern group. As a coefficient pattern to be configured, a coefficient pattern in which the weighted addition coefficient corresponding to the color information of the luminance component generation target pixel position has a positive value is recorded.

The image processing apparatus according to claim 25
Item 16. The image processing apparatus according to Item 16, wherein the recording unit
Is the first to Nth (N ≧)
When represented by a color system consisting of the color components of 3), n₁
At least the color of the first color component as the coefficient pattern
Record a coefficient pattern for weighted addition of information,
n ₂The second to Nth color components as the coefficient patterns
Coefficient for weighted addition of color information of at least one component of
It is characterized by recording a pattern.

An image processing apparatus according to a twenty-sixth aspect is the image processing apparatus according to the sixteenth aspect, further comprising correction means for performing edge enhancement processing by a fixed filter on the luminance component generated by the luminance component generation means. It is characterized by having. An image processing apparatus according to a twenty-seventh aspect is the image processing apparatus according to the sixteenth aspect, further comprising color difference component generating means for generating a color difference component using color information of the multispectral image. .

An image processing apparatus according to a twenty-eighth aspect is the image processing apparatus according to the twenty-seventh aspect, wherein the color difference component generating means calculates the degree of similarity in at least two directions to determine the similarity, and the similarity is calculated. It is characterized in that the color difference component is calculated based on the sex determination result. The image processing device according to claim 29 is the image processing device according to claim 28, wherein the similarity is a similarity between different colors configured by color information between different color components of the multispectral image. Is characterized by.

An image processing apparatus according to claim 30 is the first
A multispectral image represented by an nth (n ≧ 2) color component and having one color information in each pixel arranged in a two-dimensional square lattice, In an image processing device that performs image processing for generating luminance components of different color systems, the color information of the multispectral image is directly referred to, and the degree of similarity in the first direction is orthogonal to the first direction. A similarity regarding a second direction, a similarity regarding a third direction different from the first direction and the second direction, and a similarity regarding a fourth direction orthogonal to the third direction are calculated. Similarity calculating means, a first direction coefficient pattern indicating a weighted addition coefficient for color information of at least a first color component arranged in the first direction, and at least a first direction coefficient pattern arranged in the second direction. For color information of color components A second direction coefficient pattern indicating the coefficients of the heavy addition, at least a first is arranged in the third direction 1
Third showing the coefficient of weighted addition for color information other than color components
Recording means for recording the directional coefficient pattern and a fourth directional coefficient pattern indicating a coefficient of weighted addition for color information other than at least the first color component arranged in the fourth direction;
It is characterized by further comprising: a luminance component generating means for generating a luminance component using the similarity calculated by the similarity calculating means and the coefficient pattern recorded in the recording means.

An image processing apparatus according to a thirty-first aspect is the image processing apparatus according to the thirtieth aspect, wherein at least one coefficient pattern of the first directional coefficient pattern and the second directional coefficient pattern, and the third directional coefficient pattern. It is characterized by further comprising synthesizing means for synthesizing the coefficient pattern for generating the luminance component by using the directional coefficient pattern and at least one coefficient pattern of the fourth directional coefficient pattern.

An image processing apparatus according to a thirty-second aspect is the image processing apparatus according to the thirtieth aspect, wherein the similarity calculation means calculates the similarity regarding the first direction and the similarity regarding the second direction. Using the predetermined color information, and independently of the two similarities, the similarity regarding the third direction and the similarity regarding the fourth direction are calculated using the predetermined color information. Is characterized by.

An image processing apparatus according to a thirty-third aspect is the image processing apparatus according to the thirty-second aspect, wherein the similarity calculation means calculates the similarity regarding the third direction and the similarity regarding the fourth direction. Is calculated, color information of at least one color component forming the third directional coefficient pattern or the fourth directional coefficient pattern is used.

An image processing apparatus according to a thirty-fourth aspect is the image processing apparatus according to the thirtieth aspect, further comprising color difference component generating means for generating a color difference component using the color information of the multispectral image. Characterize. The image processing device according to claim 35 is the image processing device according to claim 34, in which the color difference component generation means calculates a degree of similarity in at least two directions to determine similarity, and determines the similarity. The color difference component is calculated based on the result.

An image processing apparatus according to a thirty-sixth aspect is the image processing apparatus according to the thirty-fifth aspect, wherein the similarity is
It is characterized in that it is a similarity between different colors composed of color information between different color components of the multispectral image. The image processing program according to claim 37, wherein the multispectral image is represented by a plurality of colors, and for each multispectral image in which one color information is present in each pixel arranged in a two-dimensional square lattice, the multispectral image table is displayed. In an image processing program for causing a computer to realize image processing for generating a luminance component of a color system different from the color system, a first direction coefficient pattern indicating a weighted addition coefficient for color information arranged in the first direction. And a second directional coefficient pattern indicating a weighted addition coefficient for color information arranged in a second direction orthogonal to the first direction, and a third directional coefficient pattern different from the first and second directions. Shows a third direction coefficient pattern indicating the weighted addition coefficient for the color information arranged in the direction and the weighted addition coefficient for the color information arranged in the fourth direction orthogonal to the third direction. A first procedure for generating a first coefficient pattern group including at least three types of coefficient patterns by using a recording procedure for recording a 4-direction coefficient pattern and the first direction coefficient pattern and the second direction coefficient pattern. A pattern generation procedure, a second pattern generation procedure for generating a second coefficient pattern group consisting of at least three types of coefficient patterns using the third direction coefficient pattern and the fourth direction coefficient pattern, A third pattern generation procedure for combining the first coefficient pattern group and the second coefficient pattern group to generate a third coefficient pattern group consisting of at least nine kinds of coefficient patterns; and the third coefficient pattern. A luminance component generation procedure for generating a luminance component by weighted addition of color information of the multispectral image using any one of the coefficient patterns of the group. And it features.

According to a thirty-eighth aspect of the present invention, there is provided an image processing program, wherein a multispectral image represented by a plurality of colors and one color information is present in each pixel arranged in a two-dimensional lattice form is used. In an image processing program for causing a computer to realize image processing for generating a brightness component of a color system different from that of the color system, the brightness component generation target pixel and the array at the position closest to the brightness component generation target pixel n ₁ (n ₁ ≧ 2) pieces of direction and n ₁ coefficients pattern indicating the coefficients of the weighted addition for the color information arranged in each said luminance component subject pixels and the line connecting the closest pixels which are N ₂ connecting a second adjacent pixel arranged at a position next to the closest pixel to the luminance component generation target pixel
A recording procedure for recording n ₂ coefficient patterns indicating weighted addition coefficients for color information arranged in each of (n ₂ ≧ 2) directions, and using the n ₁ coefficient patterns,
Using the first pattern generation procedure for generating a first coefficient pattern group consisting of m ₁ (m ₁ > n ₁ ) coefficient patterns and the n ₂ coefficient patterns, m ₂ (m ₁ > n ₂ ) A second pattern generation procedure for generating a second coefficient pattern group consisting of a number of coefficient patterns and the first coefficient pattern group and the second coefficient pattern group are combined to obtain L (L > M
₁ and L> m ₂ ) A third pattern generation procedure for generating a third coefficient pattern group consisting of coefficient patterns, and using any one coefficient pattern of the third coefficient pattern group, Weighted addition of color information of multispectral image,
And a luminance component generation procedure for generating the luminance component.

The image processing program according to a thirty-ninth aspect is represented by the first to nth (n ≧ 2) color components, and one color information exists in each of the pixels arranged in a two-dimensional square lattice. In the image processing program for causing the computer to realize the image processing for generating the luminance component of the color system different from the color system of the multi-spectral image, the color information of the multi-spectral image is directly For reference, a similarity regarding a first direction, a similarity regarding a second direction orthogonal to the first direction, and a similarity regarding a third direction different from the first direction and the second direction Degree and a similarity degree calculation procedure for calculating a similarity degree in a fourth direction orthogonal to the third direction, and a weighted addition coefficient for color information of at least the first color component arranged in the first direction. Shows A first direction coefficient pattern, the second direction coefficient pattern indicating the coefficients of the weighted addition for the at least color information of the first color component second are arranged in a direction,
A third direction coefficient pattern indicating a weighted addition coefficient for color information other than at least the first color component arranged in the third direction, and at least a first direction coefficient pattern arranged in the fourth direction;
Fourth showing a coefficient of weighted addition for color information other than color components
A recording procedure for recording the direction coefficient pattern and a luminance component generating procedure for generating a luminance component using the similarity calculated by the similarity calculating procedure and the coefficient pattern recorded in the recording procedure are provided. It is characterized by

[0045]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings. The first embodiment to the sixth embodiment are embodiments corresponding to the image processing apparatus of the present invention. In the first embodiment to the sixth embodiment, the image processing performed by the image processing apparatus of the present invention is performed. The description will be made using an image input device (for example, an electronic camera, a scanner, etc.) having the above function. The seventh embodiment is an embodiment in which the image processing program of the present invention is executed by a PC (personal computer) to realize image processing.

In particular, in the first to fourth embodiments, image quality improvement by image restoration processing (corresponding to processing for generating a luminance component) for restoring a monochrome image in the image input apparatus will be described. In the fifth embodiment and the sixth embodiment, image quality improvement by an image restoration process (corresponding to a process of generating a luminance component and a color difference component) for restoring a color image in an image input device will be described.

FIG. 1 is a functional block diagram of the image input device. In FIG. 1, the image input device 1 includes an image sensor 1
1, A / D conversion unit 12, image processing unit 13, control unit 14,
The memory card (card-shaped removable memory) 15 is provided with a memory card interface section 16 for realizing an interface and an external interface section 18 for realizing an interface with an external device such as a PC 17 via a predetermined cable or a wireless transmission path. ing.

The output of the image pickup device 11 is connected to the A / D converter 12, and the output of the A / D converter 12 is connected to the bus. The image processing unit 13, the control unit 14, the memory card interface unit 16 and the external interface unit 18 are
Connected to each other via a bus. A display 19, a printer 20, etc. are connected to the PC 17, and a CD-
It is assumed that the application program recorded in the ROM 21 is installed in advance. Also, PC1
Reference numeral 7 denotes a CPU, a memory, a hard disk (not shown), a memory card interface unit (not shown) that realizes an interface with the memory card 15, a predetermined cable or a wireless transmission path, and the like of the image input device 1. An external interface unit (not shown) that realizes an interface with an external device is provided.

In the image input device 1 having the configuration as shown in FIG. 1, the image processing unit 13 performs the image processing described later in ASIC.
It is realized by hardware such as (application specific IC). Further, in the image input device 1, an optical image is formed on the image sensor 11 via an optical system (not shown). The image sensor 11 generates an image signal corresponding to the optical image. The image signal is
The data is digitized by the A / D converter 12 and supplied as image data to the image processor 13 via the bus.

In the image input device 1 of the first to fifth embodiments, the image pickup device 11 is provided with a plurality of pixels arranged in a square lattice, and each pixel is
The R, G, and B color filters arranged in a Bayer pattern are associated with each other. Therefore, the image data supplied to the image processing unit 13 via such an image pickup device 11 is represented by the RGB color system, and each pixel has one of R, G, and B color components. Color information will exist. Hereinafter, such image data will be referred to as “Bayer array image data”.

FIG. 2 is a diagram showing an array of color components of the Bayer array type image data. In FIG. 2, the types of color components are shown using R, G, and B, and the position of the pixel corresponding to each color component is shown using the value of coordinates [X, Y]. Hereinafter, the pixel position with the R color filter in the Bayer array is referred to as the R position, the pixel position with the G color filter as the G position, and the pixel position with the B color filter as the B position.

In particular, FIG. 2A shows the case where the R position corresponds to the pixel to be processed, FIG. 2B shows the case where the B position corresponds to the pixel to be processed, and FIG. Shows the case where the G position corresponds to the pixel to be processed. In the arithmetic expression described later, when the color information in each pixel is distinguished between the green color component and the other color components, R and B in FIG. 2 are replaced with Z, and the color information at the R position or the B position is changed to Z. It is expressed as [i, j], and the color information at the G position is expressed as G [i, j].

In the image input device 1 of the sixth embodiment, the image pickup device 11 is provided with a plurality of pixels arranged in a honeycomb shape, and each pixel is in a state in which the Bayer array is rotated by 45 degrees. The color filters of R, G, and B arranged at are associated with each other. Image data supplied to the image processing unit 13 via the image sensor 11 (hereinafter,
“Honeycomb array type image data” is represented by the RGB color system like the Bayer array type image data, and each pixel has color information of any one of R, G, and B color components. Will exist.

The image processing unit 13 performs an image restoration process, which will be described later, on the Bayer array type image data (or the honeycomb array type image data). The image data that has been subjected to the image restoration processing by the image processing unit 13 is subjected to predetermined compression processing by a compression unit (not shown) as necessary, and is recorded in the memory card 15 via the memory card interface unit 16. .

The image data for which the image restoration processing has been completed is recorded in the memory card 15 without being compressed, or converted into the color system adopted in the display 19 or printer 20 on the PC 17 side, It may be supplied to the PC 17 via the external interface unit 18. FIG. 3 shows the first
Image processing unit 1 in any one of the first to fourth embodiments
3 is a diagram showing a configuration related to implementation of image restoration processing in FIG.

In FIG. 3, the image processing unit 13 includes a luminance component generation processing unit 100, and the luminance component generation processing unit 1
00 includes an R / B position brightness component generation unit 110, a G position brightness component generation unit 120, and a brightness plane correction unit 130.
The R / B position luminance component generation unit 110 includes a vertical / horizontal direction similarity determination unit 111, a diagonal direction similarity determination unit 112, a G component generation unit 113, a Mg (magenta) component generation unit 114, a vertical / horizontal luminance component generation unit 115, and an oblique direction. The brightness component generation unit 116 and the brightness component synthesis unit 117 are provided.

The outputs of the vertical / horizontal direction similarity determining section 111 and the G component generating section 113 are connected to the vertical / horizontal luminance component generating section 115, and the outputs of the diagonal direction similarity determining section 112 and the Mg component generating section 114 are the diagonal luminance component generating section. The output of the vertical / horizontal luminance component generating unit 115 and the diagonal luminance component generating unit 116 is connected to the luminance component synthesizing unit 117.
Luminance component synthesis unit 117 and G position luminance component generation unit 12
The output of 0 is connected to the luminance plane correction unit 130.

<< Explanation of First Embodiment >> FIG. 4 is a diagram for explaining the operation in the luminance component generation processing unit 100 in the first embodiment. Hereinafter, the first embodiment will be described. Hereinafter, referring to FIGS. 3 and 4, a monochrome image restoration process for the Bayer array image data realized by each unit in the luminance component generation processing unit 100 will be described. However, the description of other processes will be omitted.

First, the operation of each section in the R / B position luminance component generating section 110 will be described. << Longitudinal direction similarity determination unit 11
Description of 1 >> The vertical / horizontal direction similarity determination unit 111 calculates the vertical direction similarity Cv and the horizontal direction similarity Ch for the pixel at the R position or the B position by referring to the Bayer array image data, The calculated vertical similarity Cv is compared with the horizontal similarity Ch. Then, the vertical / horizontal direction similarity determination unit 111 determines the similarity in the vertical direction and the horizontal direction (hereinafter, referred to as “vertical / horizontal similarity”) in three stages according to the comparison result, and shows the determination result. Set the direction index HVs.

In the vertical / horizontal direction similarity determining section 111, the degrees of similarity Cv and Ch may be arbitrarily defined. For example, using Bayer array image data or data obtained by processing the image data, the vertical similarity Cv is calculated by the absolute value of the difference in color information between pixels arranged in the vertical direction, and the horizontal similarity Cv is calculated. The similarity Ch in the horizontal direction is calculated by the absolute value of the difference in color information between the pixels arranged in.

When the coordinates of the pixel to be processed are [i, j], the smaller the value of the similarity Cv [i, j] in the vertical direction and the similarity Ch [i, j] in the horizontal direction, the smaller each direction. When the definition is made so that the similarity with respect to becomes strong, the direction index HVs [i, j] can be set as follows. That is, the vertical / horizontal similarity determination unit 111 determines that for the threshold value Th1, if | Cv [i, j] -Ch [i, j] | ≦ Th1 ... It is determined that the similarities are similar, and "0" is set to the direction index HVs [i, j].

If the condition 1 is not satisfied and the condition Cv [i, j] <Ch [i, j] ... Condition 2 is satisfied, the vertical / horizontal similarity determining unit 111 determines that the pixel to be processed is in the vertical direction. It is determined that the similarity is strong, and the direction index HVs [i, j] is set to "1".

Further, the vertical / horizontal direction similarity determining section 111 is
If both Condition 1 and Condition 2 are not satisfied, it is determined that the processing target pixel has a strong similarity in the horizontal direction, and the direction index
Set "-1" to HVs [i, j]. Here, the condition 1 is a condition for determining whether or not the similarity Cv [i, j] in the vertical direction and the similarity Ch [i, j] in the horizontal direction are approximately the same, and the threshold Th1
Plays a role of avoiding erroneous determination that one of the two similarities is strong due to the influence of noise when the difference between these two similarities is small. Therefore, when there is a lot of noise in the Bayer array type image data, it is possible to improve the accuracy of the similarity determination by setting the value of the threshold Th1 high.

<< Explanation of Oblique Direction Similarity Determining Section 112 >> The oblique direction similarity determining section 112 refers to the Bayer array image data for the pixel at the R position or the B position, and calculates the diagonal 4
5 degree similarity C45 and diagonal 135 degree similarity C135
Is calculated, and the similarity C45 in the diagonal direction of 45 degrees and the diagonal degree of 135 are calculated.
The degree of similarity C135 is compared. Then, the diagonal direction similarity determination unit 112 determines whether the diagonal direction is 45 degrees depending on the comparison result.
Similarity to the degree direction and the diagonal 135 degree direction (hereinafter,
This is called "oblique similarity". ) Is determined in three steps, and the direction index DN indicating the determination result is set.

The degrees of similarity C45 and C135 may be arbitrarily defined in the diagonal direction similarity determining section 112. For example, using the Bayer array type image data or data obtained by processing the image data, the similarity C45 in the diagonal 45 degree direction is calculated by the absolute value of the difference in color information between pixels arranged in the diagonal 45 degree direction. However, the similarity in the diagonal 135 degree direction is determined by the absolute value of the difference in color information between pixels arranged in the diagonal 135 degree direction.
Calculate C135.

The coordinates of the pixel to be processed are [i, j], and the diagonal angle is 4
If the values of the similarity C45 [i, j] in the 5 degree direction and the similarity C135 [i, j] in the diagonal 135 degree direction are smaller, the similarity in each direction becomes stronger. Indicator DN
[i, j] can be set as follows.

That is, the diagonal direction similarity determination section 112.
For threshold Th2, | C45 [i, j] -C135 [i, j] | ≦ Th2 ... If Condition 3 is satisfied, the pixels to be processed have similar degrees of similarity in two diagonal directions. Therefore, the direction index DN [i, j] is set to "0". Also, the diagonal direction similarity determination unit 112
If condition 3 is not satisfied and C45 [i, j] <C135 [i, j] ... Condition 4 is satisfied, it is determined that the processing target pixel has a strong similarity in the diagonal 45 degree direction, and Set “1” to the indicator DN [i, j].

Further, when both the condition 3 and the condition 4 are not satisfied, it is judged that the pixel to be processed has a strong similarity in the oblique 135 degree direction, and "-1" is set to the direction index DN [i, j]. To do. Here, the threshold value Th2 in the condition 3 is similar to the threshold value Th1 in the above-described condition 1 and has two similarities (similarity C45 [i, j] in the oblique 45-degree direction and similarity C135 [i in the oblique 135-degree direction). , j]) is small, it plays a role of avoiding erroneous determination that one similarity is strong due to the influence of noise. Therefore, it is desirable that the threshold value Th2 be set to a value similar to the threshold value Th1.

<< Explanation of Coefficient Pattern >> Here, in order to simplify the description of the operations of the G component generating unit 113 and the Mg component generating unit 114 performed on the pixel at the R position or the B position, the G component generating unit 113 and A coefficient pattern (a coefficient pattern for weighted addition of the color information of the Bayer array image data) pre-recorded in the Mg component generation unit 114 will be described.

FIG. 5 is a diagram showing such a coefficient pattern. FIG. 5A shows a coefficient pattern in the vertical direction showing weighted addition coefficients for the color information arranged in the vertical direction of the processing target pixel, and FIG. 5B shows that the processing target pixel is arranged in the horizontal direction. 5C shows a horizontal coefficient pattern showing the weighted addition coefficient for the color information, and FIG. 5C shows a diagonal 45 degree direction showing the weighted addition coefficient for the color information arranged in the diagonal 45 degree direction of the processing target pixel. 5D shows the coefficient pattern of the diagonal 135 degree direction showing the coefficient of the weighted addition for the color information arranged in the diagonal 135 degree direction of the pixel to be processed.

In FIG. 5A, u ₁ and u ₂ are “u ₁ + u ₂
= 1 ”,“ u ₁ ≧ 0 ”,“ u ₂ ≧ 0 ”,
Normally, the value is set to the extent that “u ₁ ≈u ₂ ” is satisfied. Figure 5
In (b), v ₁ and v ₂ are “v ₁ + v ₂ = 1” and “v ₁ ≧
It is a constant that satisfies “0” and “v ₂ ≧ 0”, and usually “v ₁ ≈
v ₂ "is set to a value that satisfies the above. In FIG. 5C, t ₁ and t ₂ are “t ₁ + t ₂ = 1”, “t ₁ ≧ 0”, and “t ₂ ≧”.
It is a constant that satisfies "0", and is normally set to a value that satisfies "t ₁ ≈t ₂ ". In FIG. 5D, s ₁ and s ₂ are
It is a constant that satisfies “s ₁ + s ₂ = 1”, “s ₁ ≧ 0”, and “s ₂ ≧ 0”, and usually a value that satisfies “s ₁ ≈s ₂ ” is set.

The coefficient pattern in the vertical direction is shown in FIG.
It plays a role of averaging color information of green components arranged in the vertical direction as shown in (a) to generate a G component. The coefficient pattern in the horizontal direction plays a role of averaging the color information of the green components arranged in the horizontal direction as shown in FIG. 6B to generate the G component. Further, the coefficient pattern in the oblique 45-degree direction is weighted with the color information of the red component and the blue component arranged in the oblique 45-degree direction as shown in FIG. 6C at the R position and FIG. 6D at the B position. It plays a role of generating Mg (magenta) component on average. Further, the coefficient pattern in the direction of oblique 135 degrees is as shown in FIG. 6E at the R position and as shown in FIG.
As shown in (f), it plays a role of generating a Mg component by performing weighted averaging on the color information of the red component and the blue component arranged in the oblique 135 degree direction.

<< Explanation of G Component Generating Unit 113 >> The G component generating unit 113 refers to the Bayer array image data for the pixel at the R position or the B position, and the coefficient in the vertical direction as shown in FIG. The vertical G component Gv is generated using the pattern, and the horizontal G component Gh is generated using the horizontal coefficient pattern as shown in FIG.

That is, when the coordinates of the pixel to be processed are [i, j], the vertical G component Gv [i, j] and the horizontal G component Gh
[i, j] means u ₁ , u ₂ , v ₁ and v shown in FIGS.
It will be calculated by the following equations 1 and ₂ using ₂ . Gv [i, j] = u ₁ · G [i, j-1] + u ₂ · G [i, j + 1] ・ ・ ・ Equation 1 Gh [i, j] = v ₁ · G [i-1 , j] + v ₂ · G [i + 1, j] ... Equation 2 << Explanation of Mg component generation unit 114 >> Mg component generation unit 114
Refers to the Bayer array type image data for the pixel at the R position or the B position, and uses the coefficient pattern in the diagonal 45 degree direction as shown in FIG.
Generate Mg45 and use the coefficient pattern in the oblique 135 degree direction as shown in FIG.
Generates Mg135.

That is, when the coordinates of the pixel to be processed are [i, j], the Mg component Mg45 [i, j] in the 45 ° diagonal direction and the Mg component Mg135 [i, j] in the 135 ° diagonal direction are FIG. 5 (c),
Using t ₁ , t ₂ , s ₁ , and s ₂ shown in (d), they are calculated by the following equations 3 and 4. Mg45 [i, j] = (t ₁ · Z [i-1, j + 1] + Z [i, j] + t ₂ · Z [i + 1, j-1]) / 2 ... Equation 3 Mg135 [i, j] = (s ₁ · Z [i-1, j-1] + Z [i, j] + s ₂ · Z [i + 1, j + 1]) / 2 ... Equation 4 << Description of Vertical and Horizontal Luminance Component Generation Unit 115 >> The vertical and horizontal luminance component generation unit 115 uses the vertical direction G component Gv and the horizontal direction G component Gh generated by the G component generation unit 113, and the above-described vertical and horizontal direction similarity. The vertical / horizontal luminance component Y _HV is generated according to the value of the direction index HVs set by the determination unit 111.

When the coordinates of the pixel to be processed are [i, j],
The vertical and horizontal luminance components Y _HV [i, j] are calculated by any one of the following equations 5 to 7. When HVs [i, j] = 1: Y _HV [i, j] = Gv [i, j] ... Equation 5 When HVs [i, j] = 0: Y _HV [i, j] = ( Gv [i, j] + Gh [i, j]) / 2 ・ ・ ・ When Formula 6 HVs [i, j] =-1: Y _HV [i, j] = Gh [i, j] ・ ・ ・Expression 7 Here, when Expressions 5 to 7 are expanded by Expressions 1 and 2, the vertical and horizontal luminance components Y _HV are generated using any of the three types of coefficient patterns as shown in FIG. 7A. Equivalent to that.

It should be noted that such three types of coefficient patterns are different from the coefficient pattern in the vertical direction shown in FIG.
It corresponds to the coefficient pattern in the horizontal direction shown in (b) and the coefficient pattern obtained by averaging these coefficient patterns. The coefficient pattern in the vertical direction and the coefficient pattern in the horizontal direction are classified according to the difference in vertical and horizontal similarity. Represents a coefficient pattern generated by combining.

<< Explanation of Oblique Luminance Component Generating Unit 116 >> The oblique luminance component generating unit 116 includes the Mg component Mg45 in the direction of 45 degrees obliquely generated by the Mg component generating unit 114 and the oblique component 13
The diagonal luminance component Y _DN is generated according to the value of the direction index DN set by the diagonal direction similarity determination unit 112 described above using the Mg component Mg135 in the 5 ° direction.

When the coordinates of the pixel to be processed are [i, j],
The diagonal luminance component Y _DN [i, j] is calculated by any of the following formulas 8 to 10. When DN [i, j] = 1: Y _DN [i, j] = Mg45 [i, j] Equation 8 When DN [i, j] = 0: Y _DN [i, j] = ( Mg45 [i, j] + Mg135 [i, j]) / 2 ・ ・ ・ Equation 9 When DN [i, j] =-1: Y _DN [i, j] = Mg135 [i, j] ・ ・ ・Expression 10 Here, when Expression 8 to Expression 10 are expanded by Expression 3 and Expression 4, the diagonal luminance component Y _DN is generated using any of the three types of coefficient patterns as shown in FIG. 7B. Equivalent to that.

Note that such three types of coefficient patterns include the coefficient pattern in the oblique 45-degree direction shown in FIG. 5C and the coefficient pattern in the oblique 135-degree direction shown in FIG. 5D. Corresponding to the coefficient pattern obtained by averaging, the coefficient pattern generated by synthesizing the coefficient pattern in the diagonal 45 degree direction and the coefficient pattern in the diagonal 135 degree direction according to the difference in diagonal similarity. Represent

<< Description of Luminance Component Synthesizing Unit 117 >> The luminance component synthesizing unit 117 synthesizes the vertical and horizontal luminance components Y _HV and the diagonal luminance components Y _DN to generate the luminance component Y at the R / B position. When the coordinates of the pixel to be processed are [i, j], the luminance component Y [i, j] at the R / B position is calculated by the following Expression 11.

Y [i, j] = (α · Y _HV [i, j] + β · Y _DN [i, j]) / 2 Equation 11 However, in Equation 11, α and β are α + β = 1 ”,
These are constants that satisfy “α ≧ 0” and “β ≧ 0”, and examples of the combination of the value of α and the value of β include the following combinations. (α, β) = (1 / 3,2 / 3), (4 / 9,5 / 9), (5 / 11,6 / 11), (1 / 2,1 / 2),
(5 / 9,4 / 9), (3 / 5,2 / 5), (2 / 3,1 / 3) where Equation 11 is replaced by Equation 5 to Equation 7 and Equation 8 to Equation 10
When the combination is developed, the luminance component Y at the R / B position is generated by performing weighted addition of the color information of the Bayer array type image data using any of nine types of coefficient patterns as shown in FIG. It is equivalent to that.

In FIG. 8, the value of (HVs, DN) indicates a combination of the vertical and horizontal similarity determination result and the diagonal similarity determination result, and the strength of the similarity in the pixel to be processed is shown below. It is meant to show such characteristics. (HVs, DN) = (1,1): Strong similarity in the vertical and diagonal 45 degree directions. (HVs, DN) = (1,0): Strong vertical similarity.

(HVs, DN) = (1, -1): The similarity in the vertical and diagonal 135 degree directions is strong. (HVs, DN) = (0,1): Similarity is strong in the 45 ° diagonal direction. (HVs, DN) = (0,0): Strong similarity in all directions, or
Weak similarity in all directions (direction with strong similarity is unknown).

(HVs, DN) = (0, -1): The similarity in the oblique 135 degree direction is strong. (HVs, DN) = (-1,1): Similar in the horizontal and diagonal 45 degree directions. (HVs, DN) = (-1,0): Strong horizontal similarity. (HVs, DN) = (-1, -1): Strong similarity in lateral and diagonal 135 degree directions. That is, the nine types of coefficient patterns as shown in FIG.
It is associated with the strength of similarity in the processing target pixel.

FIG. 9 is a diagram showing the direction of strong similarity corresponding to the value of (HVs, DN). As shown in FIG. 9, in the first embodiment, the similarity in the processing target pixel is determined in nine ways, and each pixel is determined in five similar directions (horizontal direction, vertical direction, flat direction, diagonal direction). 1, the diagonal direction 2), the more detailed and accurate determination of the similarity can be made as compared with the technique disclosed in Japanese Patent Laid-Open No. 2000-78597 (US Pat. No. 6,075,889). Since it can be performed, it is possible to generate a luminance surface having high continuity with surrounding pixels.

The nine types of coefficient patterns shown in FIG. 8 are the three types of coefficient patterns (vertical coefficient pattern and horizontal coefficient pattern) related to the vertical and horizontal luminance components Y _HV shown in FIG. 7A. (The coefficient pattern generated using and)
And three types of coefficient patterns relating to the diagonal luminance component Y _DN shown in FIG. 7B (coefficient patterns generated using a coefficient pattern in the diagonal 45 degree direction and a coefficient pattern in the diagonal 135 degree direction). , And a coefficient pattern obtained by averaging in combination.

That is, the R / B position luminance component generator 11
0 G component generator 113, Mg component generator 114, vertical and horizontal luminance component generator 115, diagonal luminance component generator 116,
The luminance component synthesizing unit 117 causes the luminance component Y at the R / B position.
In the process of generating, the 9 coefficient patterns are automatically generated from the 4 previously recorded coefficient patterns.

Next, the operation of the G position luminance component generating section 120 will be described. << Description of G-Position Luminance Component Generation Unit 120 >> The G-position luminance component generation unit 120 refers to the Bayer array image data
A luminance component Y is generated for the pixel at the position. When the coordinates of the pixel to be processed are [i, j], the same α and β as in Expression 11 are used, the luminance component Y [i, j] at the G position is given by the following Expression 1
Calculated according to 2.

Y [i, j] = α · G [i, j] + (β / 4) · (Z [i-1, j] + Z [i + 1, j] + Z [i, j- 1] + Z [i, j + 1]) ... Expression 12 The calculation by Expression 12 uses the color information of the pixel to be processed and its surrounding pixels by using one type of coefficient pattern as shown in FIG. Can be regarded as combining the G component of the processing target pixel and the Mg component of only the peripheral pixels.

As described above, the luminance component Y of the R / B position generated by each unit in the R / B position luminance component generation unit 110 and the G position generated by the G position luminance component generation unit 120 are calculated. When combined with the luminance component Y, it means that a luminance plane in which the luminance component is associated with each pixel position of the Bayer array image data is generated. Since such a luminance surface considers the strength of similarity in many directions in each pixel of the Bayer array type image data,
Generates very smoothly. Therefore, such correction of the luminance surface can be realized by filtering with a fixed filter as in the case of normal edge enhancement.

<< Explanation of Luminance Surface Correction Unit 130 >> The luminance surface correction unit 130 adds the result obtained by applying a fixed bandpass filter as shown in FIG. 11 to the luminance surface to the original luminance surface, for example. , Realizes edge enhancement processing. That is, the correction of the luminance plane is realized by the calculation of the following Expressions 13 and 14.

YH [i, j] = (8 · Y [i, j]-(Y [i-1, j] + Y [i + 1, j] + Y [i, j-1] + Y [ i, j + 1] + Y [i-1, j-1] + Y [i + 1, j-1] + Y [i-1, j + 1] + Y [i + 1, j + 1] )) / 16 ・ ・ ・ Equation 13 Y [i, j] = Y [i, j] + K · YH [i, j] ・ ・ ・ Equation 14 However, in Equation 14, K is the edge emphasis strength. Is a coefficient for changing the value, and since the original luminance surface is constituted by only positive values, it is preferable to take a value of "1" or more.

As described above, in the first embodiment, nine coefficient patterns are generated from four coefficient patterns in the process of generating the luminance component Y at the R / B position, so that nine coefficient patterns are generated in advance. It is not necessary to record the coefficient pattern, and the circuit configuration of the luminance component generation processing unit 100 can be simplified. Further, in the first embodiment, the luminance component at the R / B position
When generating Y, refer to the Bayer array image data 4
If the weighted addition is performed using the different coefficient patterns, nine kinds of coefficient patterns can be generated by a simple mechanism such as averaging the results obtained by the weighted addition. Therefore, it is possible to reduce the scale of the circuit configuration of the luminance component generation processing unit 100 as compared with the case where a mechanism for realizing weighted addition by nine types of coefficient patterns is provided in advance.

Further, in the first embodiment, the coefficient patterns recorded in advance are combined to generate a new coefficient pattern in accordance with the difference in similarity between the pixel positions. Therefore, the coefficient pattern used when generating the luminance component is automatically switched according to the difference in local similarity, and the luminance component can be accurately generated. Also, the optimum coefficient pattern to be used when generating the luminance component can be immediately determined according to the difference in similarity, and the technique disclosed in US Pat. No. 5,901,242 is disclosed. It is not necessary to carry out repeated processing by trial and error as performed in.

In addition to the above-described coefficient pattern automatic composition, the following several effects can be obtained together.
For example, in the first embodiment, as shown in FIGS. 6A and 6B, a coefficient pattern for averaging color information of green components arranged in the vertical direction is recorded as the coefficient pattern in the vertical direction. As a horizontal coefficient pattern, a coefficient pattern for averaging color information of green components arranged in the horizontal direction is recorded. Further, as shown in FIGS. 6C to 6F, the coefficient pattern in the diagonal 45 degree direction is diagonal 45 degrees.
A coefficient pattern for weighted averaging the color information of the red and blue components arranged in the degree direction is recorded.
A coefficient pattern for weighted averaging the color information of the red component and the blue component arranged in the direction of 135 degrees diagonally is recorded as the coefficient pattern of the 5 degree direction. Therefore, the nine coefficient patterns generated from the four coefficient patterns recorded in this way are weighted average of the color information of the three colors. In the coefficient pattern shown in FIG. 10, the color information of the three colors is weighted and averaged at the same ratio.

Therefore, according to the first embodiment, R
The luminance component Y at the / B position and the luminance component Y at the G position are
It can be generated by using color information of three colors of G and B at a constant ratio. Therefore, it is possible to suppress the variation in the RGB ratio of the luminance value in the colored portion, which is likely to occur due to the technique disclosed in Japanese Patent Laid-Open No. 2000-78597 (US Pat. No. 6,075,889). Therefore, it is possible to generate a luminance component that is smoothly connected between adjacent pixels.

Further, in the first embodiment, the four types of coefficient patterns are the most arranged color information of the color components of R, G, and B colors as shown in FIGS. 5A to 5D. It is a coefficient pattern that does not include a negative value and is set so as to perform weighted addition of color information included in a narrow range (3 pixels × 3 pixels), and uses color information of R, G, and B colors as described above. Not only is it possible to generate nine types of coefficient patterns for realizing the generation of the luminance component, but it also contributes to the reduction in the scale of the circuit configuration of the luminance component generation processing unit 100.

Further, in the first embodiment, as shown in FIG.
As shown in (d), the coefficient of weighted addition corresponding to the color information of the pixel to be processed indicates "1" in the coefficient pattern in the diagonal 45-degree direction and the coefficient pattern in the diagonal 135-degree direction. Therefore, according to nine kinds of coefficient patterns generated from four kinds of coefficient patterns including such a coefficient pattern, it is possible to generate a jaggies-free luminance component in which the color information of the pixel to be processed is reflected.

Further, in the first embodiment, the correction of the luminance plane can be realized by the edge emphasis processing by the fixed band pass filter regardless of the strength of the similarity in each pixel. Therefore, it is not necessary to switch the correction method at each pixel position, and the luminance component generation processing unit 1
The circuit configuration of 00 does not become complicated. Further, since the correction value for the luminance plane can be extracted from the luminance plane itself, the method disclosed in Japanese Patent Laid-Open No.
Unlike the technique disclosed in 0-78597 (U.S. Pat. No. 6,075,889), it is not necessary to pre-calculate a plurality of terms necessary for correction and keep the data.

In the first embodiment, the Bayer array image data is referred to in the vertical / horizontal direction similarity determination section 11.
1, diagonal direction similarity determination unit 112, G component generation unit 11
3, Mg component generation unit 114, G position luminance component generation unit 12
It is performed only by 0, and is not necessary in the vertical and horizontal luminance component generation unit 115, the diagonal luminance component generation unit 116, the luminance component synthesis unit 117, and the luminance plane correction unit 130. Therefore, in the first embodiment, the vertical / horizontal direction similarity determination unit 111, the diagonal direction similarity determination unit 112, the G component generation unit 113, and the Mg component generation unit 11 are described.
4. The Bayer array image data can be released when the operation by the G position luminance component generation unit 120 is completed. In particular, in the first embodiment, since these units are realized by hardware, they can be operated independently of each other and in parallel, so that the time for holding the Bayer array type image data can be further shortened. It is possible, and the Bayer array type image data can be released quickly.

In the first embodiment, in the R / B position luminance component generating section 110, as shown in FIGS. 6 (c), 6 (d),
(E) and (f) are arranged by averaging the color information of the red component and the blue component, and the 45-degree diagonal Mg component Mg45 and the 135-degree diagonal Mg component Mg13 are generated.
5 is used to generate the diagonal luminance component Y _DN . However, in the present invention, when the pixel at the R position is the pixel to be processed, the luminance generated by averaging the color information of the green component and the red component arranged as shown in FIGS. The vertical and horizontal luminance components Y _HV are generated using the components, and FIG.
The diagonal luminance component Y _DN can be generated using the luminance component generated by averaging the color information of the blue components arranged as shown in (c) and (d). Similarly, in the case where the pixel at the B position is the pixel to be processed, FIG.
Using the luminance component generated by averaging the color information arranged as shown in (f), (g), and (h), the vertical and horizontal luminance components
Y _HV and diagonal luminance component Y _DN can be generated.

That is, the color information of the pixel to be processed is the first
Although it may be used when the oblique luminance component Y _DN is generated as in the above embodiment, it can also be used when the vertical and horizontal luminance components Y _HV are generated. << Description of Second Embodiment >> Hereinafter, as a description of the second embodiment, a monochrome image restoration process for the Bayer array image data realized by each unit in the luminance component generation processing unit 100 will be described.

FIG. 13 is a diagram for explaining the operation in the luminance component generation processing section 100 in the second embodiment. Second
The operation performed by the vertical and horizontal direction similarity determination unit 111, the diagonal direction similarity determination unit 112, the vertical and horizontal luminance component generation unit 115, and the diagonal luminance component generation unit 116 is different between the first exemplary embodiment and the second exemplary embodiment. The operation is the same.

Therefore, hereinafter, only the operations of the vertical / horizontal direction similarity determining section 111, the diagonal direction similarity determining section 112, the vertical / horizontal luminance component generating section 115, and the diagonal luminance component generating section 116 will be described, and the other operations will not be described. Omit it. << Explanation of Vertical / Horizontal Direction Similarity Determining Unit 111 >> The vertical / horizontal direction similarity determining unit 111 refers to the Bayer array type image data, as in the case of the first embodiment, with respect to the pixel at the R position or the B position. The directional similarity Cv and the lateral similarity Ch are calculated. Then, the vertical / horizontal direction similarity determination unit 111 sets the direction index HVs [i, j] (direction index indicating the determination result of vertical / horizontal similarity) in the processing target pixel [i, j] by the following Expression 15.

HVs [i, j] = (Ch [i, j] -Cv [i, j]) / (Cv [i, j] + Ch [i, j]) Equation 15 Such setting The direction index HVs is continuously represented by the vertical similarity Cv and the horizontal similarity Ch.
The closer the similarity in the vertical direction is, the closer to "1", and the stronger the similarity in the horizontal direction is, the closer to "-1". That is, the value of the vertical similarity Cv [i, j] and the horizontal similarity Ch [i, j]
The smaller the value of, the stronger the similarity to each direction, so if the following conditions hold, the direction index HVs [i, j] is
The setting is made in the same manner as in the first embodiment.

When Ch [i, j] >> Cv [i, j]: HVs [i, j] = 1, which means that the pixel to be processed has a strong similarity in the vertical direction. When Ch [i, j] ≈Cv [i, j]: HVs [i, j] = 0, which means that the processing target pixel has similar similarity in both vertical and horizontal directions. When Ch [i, j] << Cv [i, j]: HVs [i, j] =-1, which indicates that the pixel to be processed has a strong similarity in the horizontal direction.

<< Explanation of Oblique Direction Similarity Determining Section 112 >> As in the first embodiment, the oblique direction similarity determining section 112 refers to the Bayer array type image data and selects the pixel at the R position or the B position. On the other hand, the similarity C45 in the diagonal 45 degree direction and the similarity C135 in the diagonal 135 degree direction are calculated. And
The diagonal direction similarity determination unit 112 sets the direction index DN [i, j] (direction index indicating the determination result of diagonal similarity) in the processing target pixel [i, j] by the following Expression 16.

DN [i, j] = (C135 [i, j] -C45 [i, j]) / (C45 [i, j] + C135 [i, j]) Equation 16 Such setting Thus, the direction index DN is continuously represented by the similarity C45 in the diagonal 45-degree direction and the similarity C135 in the diagonal 135-degree direction. The stronger the similarity in the diagonal 45-degree direction is, the closer it is to "1", and the diagonal 135. The stronger the degree of similarity is, the closer it is to "-1".

That is, the degree of similarity C45 [i,
The smaller the value of j] and the value of the similarity C135 [i, j] in the diagonal direction of 135 degrees, the stronger the similarity of the pixel to be processed in each direction. Therefore, if the following conditions are satisfied, the direction index DN
[i, j] will be set as in the first embodiment. When C135 [i, j] >> C45 [i, j]: DN [i, j] = 1, indicating that the pixels to be processed have similar degrees of similarity in the diagonal 45 ° direction. Become.

When C135 [i, j] ≈C45 [i, j]: DN [i, j] = 0, and the processing target pixel has a strong similarity (or weakness) in two diagonal directions. Will be represented. When C135 [i, j] << C45 [i, j]: DN [i, j] = − 1, which means that the pixel to be processed has a strong similarity in the oblique 135 ° direction. << Description of Vertical / Horizontal Brightness Component Generation Unit 115 >> The vertical / horizontal brightness component generation unit 115 generates vertical G components in the same manner as in the first embodiment by the G component generation unit 113 for pixels at the R position or the B position. By using Gv and the horizontal G component Gh and the direction index HVs set by the vertical / horizontal direction similarity determination unit 111 described above, the vertical / horizontal luminance component Y _HV [i, j] of the processing target pixel [i, j] is obtained. Is calculated by the following equation 17.

Y _HV [i, j] = Kv [i, j] · Gv [i, j] + Kh [i, j] · Gh [i, j] Equation 17 However, in Equation 17, Kv [i, j] = (1 + HVs [i, j]) / 2, Kh
[i, j] = (1-HVs [i, j]) / 2. Here, Equation 17 is changed to the first
When expanded by Expression 1 and Expression 2 described in the embodiment, the vertical / horizontal luminance component Y _HV corresponds to being generated using a coefficient pattern as shown in FIG.

However, in FIG. 14A, HVs are
The coefficient pattern shown in FIG. 14A represents various coefficient patterns associated with the value of HVs, because the coefficient pattern continuously changes within a range satisfying “−1 ≦ HVs ≦ 1”. Also,
The coefficient pattern shown in FIG. 14A is, for example, “HVs =
1 ”corresponds to the coefficient patterns shown in FIGS. 7 (a)-(1), and when“ HVs = −1 ”, FIG. 7 (a)-(3)
It matches the coefficient pattern shown in, and when "HVs = 0",
This coincides with the coefficient pattern shown in FIGS.

That is, the coefficient pattern shown in FIG. 14A is obtained by weighted averaging two coefficient patterns (a vertical coefficient pattern and a horizontal coefficient pattern) recorded in advance in the G component generator 113. Corresponding to the generated coefficient pattern, and represents a coefficient pattern generated by combining according to the difference in vertical and horizontal similarities. << Explanation of Oblique Luminance Component Generation Unit 116 >>
The Mg component Mg45 in the oblique 45 degree direction and the oblique 135 generated by the component generation unit 114 in the same manner as in the first embodiment.
Using the Mg component Mg135 in the degree direction and the direction index DN set by the diagonal direction similarity determination unit 112 described above,
The diagonal luminance component Y _DN [i, j] in the processing target pixel [i, j] is
It is calculated by the following equation 18.

Y _DN [i, j] = K45 [i, j] .Mg45 [i, j] + K135 [i, j] .Mg135 [i, j] Equation 18 However, in Equation 18, K45 [i, j] = (1 + DN [i, j]) / 2, K1
35 [i, j] = (1-DN [i, j]) / 2. Here, when Expression 18 is expanded by Expression 3 and Expression 4 described in the first embodiment, the diagonal luminance component Y _DN is generated using the coefficient pattern as shown in FIG. 14B. Equivalent to.

However, in FIG. 14B, the DN is
The coefficient pattern shown in FIG. 14B represents various coefficient patterns associated with the value of DN, because the coefficient pattern changes continuously within the range of “−1 ≦ DN ≦ 1”.

Further, the coefficient pattern shown in FIG. 14B coincides with the coefficient patterns shown in FIGS. 7B to 7C when "DN = 1", and "DN = -1". In the case of, it matches the coefficient pattern shown in FIGS.
= 0 ”, the coefficient pattern matches the coefficient pattern shown in FIGS. That is, the coefficient pattern shown in FIG. 14B is a weighted average of two coefficient patterns (a coefficient pattern in the oblique 45-degree direction and a coefficient pattern in the oblique 135-degree direction) recorded in advance in the Mg component generation unit 114. It corresponds to the obtained coefficient pattern, and represents the coefficient pattern generated by combining according to the difference in diagonal similarity.

As described above, vertical and horizontal luminance component generation
Vertical and horizontal luminance components Y generated by the composition unit 115_HVAnd the diagonal
Diagonal luminance generated by the luminance component generating unit 116
Ingredient Y _DNIs the luminance component synthesis as in the first embodiment.
It is synthesized by the section 117. That is, the first embodiment
The luminance component at the R / B position in the same manner as the equation 11 described in
Y is generated.

Here, when Expression 11 is expanded by Expression 17 and Expression 18, the luminance component Y at the R / B position is calculated as shown in FIG.
This is equivalent to generation of weighted addition of color information of the Bayer array type image data using a coefficient pattern as shown in FIG. However, in FIG. 15, HVs is “−1 ≦ HVs ≦
1 continuously changes within the range satisfying “1”, and DN is “−1 ≦ DN
It continuously changes within a range satisfying ≦ 1. In addition, FIG.
The coefficient pattern as shown in FIG. 14 is a coefficient pattern related to the vertical and horizontal luminance components Y _HV shown in FIG. 14A (a coefficient pattern generated using the vertical coefficient pattern and the horizontal coefficient pattern), Diagonal luminance component Y shown in 14 (b)
It is configured by a coefficient pattern obtained by combining and averaging coefficient patterns related to _DN (a coefficient pattern generated by using a coefficient pattern in the diagonal direction of 45 degrees and a coefficient pattern in the diagonal direction of 135 degrees).

Therefore, in the second embodiment, R / B
The G component generation unit 113, M in the position / luminance component generation unit 110
In the process of the luminance component Y at the R / B position being generated by the g component generation unit 114, the vertical and horizontal luminance component generation unit 115, the diagonal luminance component generation unit 116, and the luminance component combination unit 117, four pre-recorded coefficients An infinite number of coefficient patterns will be generated from the pattern.

That is, in the second embodiment, the optimum coefficient pattern according to the vertical / horizontal similarity or the diagonal similarity of the pixel position to be processed can be generated from the four kinds of previously recorded coefficient patterns. Further, in the second embodiment, if the weighted addition using four types of coefficient patterns is performed with reference to the Bayer array type image data, the results obtained by the weighted addition are converted into vertical and horizontal similarity and diagonal similarity. Accordingly, the optimum coefficient pattern at the pixel position to be processed can be generated by a simple mechanism such as weighted averaging.

Therefore, according to the second embodiment, it is possible to perform the monochrome image restoration process with high accuracy while minimizing the scale of the circuit configuration of the luminance component generation processing section 100. << Description of Third Embodiment >> Hereinafter, as a description of the third embodiment, a monochrome image restoration process for Bayer array image data realized by each unit in the luminance component generation processing unit 100 will be described. The third embodiment is the same as the first or second embodiment except for the operations of the G component generating unit 113 and the Mg component generating unit 114 among the operations of the respective units in the luminance component generating processing unit 100.

Therefore, only the operations of the G component generating unit 113 and the Mg component generating unit 114 will be described below, and the description of the other operations will be omitted. FIG. 16 is a diagram illustrating operations of the G component generation unit 113 and the Mg component generation unit 114 according to the third embodiment. FIG. 17 shows the G component generation unit 113 and the Mg component generation unit 114 in the third embodiment.
It is a figure which shows the coefficient pattern previously recorded on.

FIG. 17A shows an upward coefficient pattern showing the weighted addition coefficient for the color information arranged in the upward direction of the pixel to be processed, and FIG. 17B shows the downward pattern of the pixel to be processed. 7 shows a downward coefficient pattern indicating weighted addition coefficients for the color information arranged in FIG. FIG. 17 (c)
Shows a coefficient pattern in the left direction showing the weighted addition coefficient for the color information arranged in the left direction of the processing target pixel, and FIG. 17D shows weighting for the color information arranged in the right direction of the processing target pixel. The coefficient pattern of the right direction which shows the coefficient of addition is shown.

FIG. 17E shows a coefficient pattern in the upper right direction showing the weighted addition coefficient for the color information arranged in the upper right direction of the processing target pixel, and FIG. 17F shows the lower left direction of the processing target pixel. 7 shows a coefficient pattern in the lower left direction showing the weighted addition coefficient for the color information arranged in FIG. Figure 1
7 (g) shows a coefficient pattern in the upper left direction showing the weighted addition coefficient for the color information arranged in the upper left direction of the processing target pixel, and FIG. 17 (h) is arranged in the lower right direction of the processing target pixel. 9 shows a coefficient pattern in the lower right direction, which indicates the coefficient of weighted addition for the color information.

<< Explanation of G Component Generating Unit 113 >> The G component generating unit 113 refers to the Bayer array image data for the pixel at the R position or the B position, and moves in the upward direction as shown in FIG. A coefficient pattern in the vertical direction as shown in FIG. 5A is generated using the coefficient pattern and the coefficient pattern in the downward direction as shown in FIG. 17B.

That is, when the coordinates of the pixel to be processed are [i, j], u ₁ and u ₂ which constitute the coefficient pattern in the vertical direction are defined as the vertical weighting coefficients, and the similarity in the upward direction Cup [i, j]
And the similarity Cdn [i, j] in the downward direction are used to calculate as follows. u ₁ = Cdn [i, j] / (Cup [i, j] + Cdn [i, j]) Equation 19 u ₂ = Cup [i, j] / (Cup [i, j] + Cdn [ i, j]) ・ ・ ・ Equation 20 However, Cup [i, j] = | Z [i, j-2] -Z [i, j] | ・ ・ ・ Equation 21 Cdn [i, j] = | Z [i, j + 2] -Z [i, j] | Equation 22

Then, the G component generator 113 uses the direction weighting coefficient calculated in this way, and the upward G component Gup generated from the upward coefficient pattern and the downward direction generated from the downward coefficient pattern. The G component Gv of the vertical direction is used to generate the G component Gv in the vertical direction as follows. Gv [i, j] = u ₁ · Gup [i, j] + u ₂ · Gdn [i, j] ・ ・ ・ Equation 23 where Gup [i, j] = G [i, j-1] ・ ・Formula 24 Gdn [i, j] = G [i, j + 1] ... Formula 25.

Further, the G component generating unit 113 refers to the Bayer array image data for the pixel at the R position or the B position, and the coefficient pattern in the left direction as shown in FIG. 5) is used to generate a horizontal coefficient pattern as shown in FIG. 5B. That is, when the coordinates of the pixel to be processed are [i, j], v ₁ and v ₂ which form the coefficient pattern in the horizontal direction are defined as the direction weighting coefficients between the left and right, and the similarity Clt [i, j]
And the similarity Crt [i, j] in the right direction are used to calculate as follows.

V ₁ = Clt [i, j] / (Clt [i, j] + Crt [i, j]) Equation 26 v ₂ = Crt [i, j] / (Clt [i, j] + Crt [i, j]) ・ ・ ・ Equation 27 where Clt [i, j] = | Z [i-2, j] -Z [i, j] | ・ ・ ・ Equation 28 Crt [i, j] = | Z [i + 2, j] -Z [i, j] | ... Equation 29.

Then, the G component generator 113 uses the direction weighting coefficient calculated in this way to generate a leftward G component Glt generated from the leftward coefficient pattern and a rightward direction generated from the rightward coefficient pattern. And the G component Grt in the horizontal direction, the horizontal G component Gh is generated as follows. Gh [i, j] = v ₁ · Glt [i, j] + v ₂ · Grt [i, j] ・ ・ ・ Equation 30 where Glt [i, j] = G [i-1, j] ・ ・Formula 31 Grt [i, j] = G [i + 1, j] ... Formula 32.

Next, the vertical G component Gv and the horizontal G component Gh described above are the vertical and horizontal luminance components Y _HV by the vertical and horizontal luminance component generating section 115, as in the first or second embodiment.
Used to generate << Description of Mg Component Generation Unit 114 >> Mg Component Generation Unit 114
Refers to the Bayer array image data for the pixel at the R position or the B position, and the coefficient pattern in the upper right direction as shown in FIG. 17E and the coefficient pattern in the lower left direction as shown in FIG. 17F. And are used to generate a coefficient pattern in the direction of oblique 45 degrees as shown in FIG.

That is, when the coordinates of the pixel to be processed are [i, j], a coefficient pattern in the direction of an oblique 45 degree is formed.
We define t ₁ and t ₂ as direction weighting coefficients between upper right and lower left, and use the similarity Cur [i, j] in the upper right direction and the similarity Cdl [i, j] in the lower left direction as follows. Calculate to. t ₁ = Cdl [i, j] / (Cur [i, j] + Cdl [i, j]) Equation 33 t ₂ = Cur [i, j] / (Cur [i, j] + Cdl [ i, j]) ・ ・ ・ Equation 34 However, Cur [i, j] = | Z [i + 2, j-2] -Z [i, j] | ・ ・ ・ Equation 35 Cdl [i, j] = | Z [i-2, j + 2] -Z [i, j] | ... Expression 36.

Then, the G component generating unit 113 uses the direction weighting coefficient calculated in this way, and the Mg component Mg_ur in the upper right direction generated from the coefficient pattern in the upper right direction and the lower left direction generated from the coefficient pattern in the lower left direction. Mg component of Mg_d
With l, the Mg component Mg_45 in the oblique 45 ° direction is generated. Mg_45 [i, j] = t ₁ · Mg_ur [i, j] + t ₂ · Mg_dl [i, j] ・ ・ ・ Equation 37 However, Mg_ur [i, j] = (Z [i, j] + Z [ i + 1, j-1]) / 2 ... Equation 38 Mg_dl [i, j] = (Z [i, j] + Z [i-1, j + 1]) / 2 ... Equation 39 is there.

Further, the Mg component generator 114 refers to the Bayer array image data for the pixel at the R position or the B position, and the coefficient pattern in the upper left direction as shown in FIG. 5) is used to generate a coefficient pattern in the direction of an oblique 135 degree as shown in FIG. 5D. That is, when the coordinates of the pixel to be processed are [i, j], s ₁ and s ₂ that form the coefficient pattern in the diagonal 135 degree direction are defined as the direction weighting coefficients between the upper left and lower right, and Using the similarity Cul [i, j] and the lower right direction similarity Cdr [i, j], the calculation is performed as follows.

S ₁ = Cdr [i, j] / (Cul [i, j] + Cdr [i, j]) Equation 40 s ₂ = Cul [i, j] / (Cul [i, j] + Cdr [i, j]) ・ ・ ・ Equation 41 However, Cul [i, j] = | Z [i-2, j-2] -Z [i, j] | ・ ・ ・ Equation 42 Cdr [i, j] = | Z [i + 2, j + 2] -Z [i, j] | ... Equation 43.

Then, the G component generator 113 uses the direction weighting coefficients calculated in this way to generate the Mg component Mg_ul in the upper left direction and the coefficient pattern in the lower right direction, which are generated from the coefficient patterns in the upper left and right directions. Mg component Mg in the lower right direction
_dr, the Mg component Mg_135 in the oblique direction of 135 degrees is generated.

Mg_135 [i, j] = s ₁ · Mg_ul [i, j] + s ₂ · Mg_dr [i, j] ・ ・ ・ Equation 44 However, Mg_ul [i, j] = (Z [i, j] + Z [i-1, j-1]) / 2 ・ ・ ・ Equation 45 Mg_dr [i, j] = (Z [i, j] + Z [i + 1, j + 1]) / 2 ・ ・ ・Equation 46 is given.

Next, the above-mentioned Mg component in the oblique 45 ° direction
Mg45 and Mg component Mg135 in the oblique 135 degree direction are the first
Alternatively, similarly to the second embodiment, the diagonal luminance component generating unit 116 generates the diagonal luminance component Y _DN . Then, the above-described vertical and horizontal luminance components Y _HV and diagonal luminance components Y _DN
Is used for generation of the luminance component Y at the R / B position by the luminance component synthesizing unit 117, as in the first or second embodiment.

Such a vertical and horizontal luminance component Y _HV is, for example,
When applied to the case of the second embodiment, it corresponds to generation using a coefficient pattern as shown in FIG. The diagonal luminance component Y _DN is equivalent to being generated using a coefficient pattern as shown in FIG.
The brightness component Y at the R / B position corresponds to being generated using a coefficient pattern as shown in FIG.

Therefore, in the third embodiment, the G component generating unit 113, the Mg component generating unit 114, the vertical and horizontal luminance component generating unit 115, the diagonal luminance component generating unit 116, and the luminance component synthesizing unit 117 make the luminance at the R / B position. In the process of generating the component Y, an infinite number of coefficient patterns localized in detail from the eight kinds of coefficient patterns recorded in advance are generated.

Therefore, according to the third embodiment, the monochrome image restoration process can be performed with high accuracy.
It is effective for restoring fine structures such as characters. << Description of Fourth Embodiment >> Hereinafter, the fourth embodiment will be described. The feature of the fourth embodiment is that the similarity Cv in the vertical direction and the similarity in the horizontal direction by the vertical / horizontal similarity determination unit 111. The method of calculating the degree Ch and the method of calculating the degree of similarity C45 in the diagonal direction of 45 degrees and the degree of similarity C135 in the direction of diagonal 135 degrees by the diagonal direction similarity determination unit 112, and other operations are described above. This is the same as the first to third embodiments.

Therefore, in the following, as a description of the fourth embodiment, three representative examples (representative example 1, representative example 2, representative example 3) of the method for calculating the similarity will be described. However, the coordinates of the pixel to be processed are [i, j], and the similarity defined below has a higher similarity as the value of the similarity is smaller. << Description of Representative Example 1 >> First, the representative example 1 will be described.

The vertical / horizontal similarity determination unit 111 refers to the Bayer array image data, and with respect to the pixel at the R position or the B position, the vertical similarity Cv [i, j] and the horizontal similarity Ch. [i, j] is calculated by the following Expression 47 and Expression 48. Cv [i, j] = | G [i, j-1] -G [i, j + 1] | ... Equation 47 Ch [i, j] = | G [i-1, j] -G [ i + 1, j] | Equation 48 The diagonal similarity determination unit 112 refers to the Bayer array image data, and with respect to the pixel at the R position or the B position, the similarity C45 [in the diagonal direction of 45 degrees]. i, j] and the similarity C135 [i, j] in the diagonal direction of 135 degrees are calculated by the following Expression 49 and Expression 50.

C45 [i, j] = (| G [i, j-1] -G [i-1, j] | + | G [i + 1, j] -G [i, j + 1] | ) / 2 Equation 49 C135 [i, j] = (| G [i, j-1] -G [i + 1, j] | + | G [i-1, j] -G [i, j + 1] |) / 2 Equation 50 That is, in the representative example 1, the similarity Cv in the vertical direction, the similarity Ch in the horizontal direction, the similarity C45 in the diagonal 45 degree direction, and the similarity in the diagonal 135 degree direction. Both C135 is calculated using the color information of the green component, and is composed only of the similarity component (hereinafter, referred to as “the inter-GG similarity component”) that is composed of the absolute value of the difference in the color information of the green component. There is.

FIG. 18A is a diagram showing the positional relationship of the color information used when calculating the degree of similarity in the first representative example. << Description of Representative Example 2 >> Next, Representative Example 2 will be described. The vertical / horizontal direction similarity determination unit 111 refers to the Bayer array image data, and with respect to the pixel at the R position or the B position, the vertical direction similarity Cv [i, j] and the horizontal direction similarity Ch [i, j]. j] is calculated by the following Equation 51 and Equation 52.

Cv [i, j] = (| G [i, j-1] -G [i, j + 1] | + | G [i-1, j-2] -G [i-1, j ] | + | G [i-1, j + 2] -G [i-1, j] | + | G [i + 1, j-2] -G [i + 1, j] | + | G [ i + 1, j + 2] -G [i + 1, j] |) / 5 Equation 51 Ch [i, j] = (| G [i-1, j] -G [i + 1, j] | + | G [i-2, j-1] -G [i, j-1] | + | G [i + 2, j-1] -G [i, j-1] | + | G [i-2, j + 1] -G [i, j + 1] | + | G [i + 2, j + 1] -G [i, j + 1] |) / 5 Equation 52 In the representative example 2, the vertical direction similarity Cv and the horizontal direction similarity Ch are both calculated using the color information of the green component, and are composed of only the inter-GG similarity component.

The diagonal similarity determination unit 112 refers to the Bayer array type image data, and with respect to the pixel at the R position or the B position, the similarity C45 [i, j] in the diagonal 45 degree direction and the diagonal 135 degree direction. The similarity C135 [i, j] of is calculated by the following equations 53 and 54. C45 [i, j] = (| G [i, j-1] -G [i-1, j] | + | G [i + 1, j] -G [i, j + 1] | + | G [i-1, j-2] -G [i-2, j-1] | + | G [i + 2, j + 1] -G [i + 1, j + 2] | + | Z [i + 1, j-1] -Z [i-1, j + 1] |) / 5 ・ ・ ・ Equation 53 C135 [i, j] = (| G [i, j-1] -G [i + 1 , j] | + | G [i-1, j] -G [i, j + 1] | + | G [i-2, j + 1] -G [i-1, j + 2] | + | G [i + 1, j-2] -G [i + 2, j-1] | + | Z [i-1, j-1] -Z [i + 1, j + 1] |) / 5 Equation 54 Here, Z [i + 1, j-1], Z [i-1, j + 1] in Equation 53 and Z [i-1, j-1], Z [i + in Equation 54 1, j + 1] is the color information adjacent to the processing target pixel in the diagonal direction, and indicates the color information of the blue component when the pixel at the R position is the processing target pixel,
When the pixel at the B position is the pixel to be processed, it indicates the color information of the red component.

That is, in the representative example 2, the similarity C45 in the oblique 45-degree direction and the similarity C135 in the oblique 135-degree direction are both the color information of the green component and the color information of the blue component (in some cases, the red component Color information) and a similarity component not only for the GG similarity component but also for the absolute value of the difference in the color information of the blue component (hereinafter referred to as “BB similarity component”), or It is composed of a similarity component (hereinafter referred to as “RR similarity component”) that is composed of the absolute value of the difference in color information of the red component.

FIG. 18B is a diagram showing the positional relationship of color information used when calculating the degree of similarity according to the second representative example.
However, in FIG. 18B, the degree of similarity in the diagonal direction of 45 degrees
For C45 and the similarity C135 in the diagonal 135 degree direction,
An example is shown in which the pixel at the R position is the pixel to be processed. << Description of Representative Example 3 >> Next, Representative Example 3 will be described.

The vertical / horizontal similarity determination section 111 refers to the Bayer array type image data, and with respect to the pixel at the R position or the B position, the vertical similarity Cv [i, j] and the horizontal similarity Ch. [i, j] is calculated by the following equations 55 and 56. Cv [i, j] = (4 ・ | G [i, j-1] -G [i, j + 1] | +2 ・ (| G [i-1, j-2] -G [i-1 , j] | + | G [i-1, j + 2] -G [i-1, j] | + | G [i + 1, j-2] -G [i + 1, j] | + | G [i + 1, j + 2] -G [i + 1, j] |) + | G [i, j-3] -G [i, j-1] | + | G [i, j + 3 ] -G [i, j + 1] | + | G [i-2, j-1] -G [i-2, j + 1] | + | G [i + 2, j-1] -G [ i + 2, j + 1] |) / 16 Equation 55 Ch [i, j] = (4 ・ | G [i-1, j] -G [i + 1, j] | +2 ・ ( | G [i-2, j-1] -G [i, j-1] | + | G [i + 2, j-1] -G [i, j-1] | + | G [i-2 , j + 1] -G [i, j + 1] | + | G [i + 2, j + 1] -G [i, j + 1] |) + | G [i-3, j] -G [i-1, j] | + | G [i + 3, j] -G [i + 1, j] | + | G [i-1, j-2] -G [i + 1, j-2 ] | + | G [i-1, j + 2] -G [i + 1, j + 2] |) / 16 Equation 56 That is, in the representative example 3, the similarity Cv in the vertical direction and the similarity Cv in the horizontal direction. And the similarity Ch of both are calculated using the color information of the green component, and are composed of only the inter-GG similarity component.

The diagonal similarity determination unit 112 refers to the Bayer array type image data, and with respect to the pixel at the R position or B position, the similarity C45 [i, j] in the diagonal 45 degree direction and the diagonal 135 degree direction. The similarity C135 [i, j] of is calculated by the following Expressions 57 and 58. C45 [i, j] = (6 ・ (| G [i, j-1] -G [i-1, j] | + | G [i + 1, j] -G [i, j + 1] | ) +2 ・ (| G [i-1, j-2] -G [i-2, j-1] | + | G [i + 2, j + 1] -G [i + 1, j + 2 ] |) +3 ・ (| G [i + 1, j-2] -G [i, j-1] | + | G [i + 2, j-1] -G [i + 1, j] | ) +1 · (| G [i, j-3] -G [i-1, j-2] | + | G [i + 3, j] -G [i + 2, j + 1] |) + 3 ・ (| G [i-1, j] -G [i-2, j + 1] | + | G [i, j + 1] -G [i-1, j + 2] |) +1 ・(| G [i-2, j-1] -G [i-3, j] | + | G [i + 1, j + 2] -G [i, j + 3] |)) / 64 + ( 2 · | Z [i + 1, j-1] -Z [i-1, j + 1] | + | Z [i, j-2] -Z [i-2, j] | + | Z [i + 2, j] -Z [i, j + 2] |) / 8 Equation 57 C135 [i, j] = (6 · (| G [i, j-1] -G [i + 1, j] | + | G [i-1, j] -G [i, j + 1] |) +2 ・ (| G [i-2, j + 1] -G [i-1, j + 2] | + | G [i + 1, j-2] -G [i + 2, j-1] |) +3 ・ (| G [i-1, j-2] -G [i, j-1] | + | G [i-2, j-1] -G [i-1, j] |) +1 ・ (| G [i-3, j] -G [i-2, j + 1] | + | G [i, j-3] -G [i + 1, j-2] |) +3 ・ (| G [i + 1, j] -G [i + 2, j + 1] | + | G [i, j + 1] -G [i + 1, j + 2] |) +1 ・ (| G [i-1, j + 2] -G [i, j + 3] | + | G [i + 1, j-1] -G [i + 3, j] |)) / 64 +2 ・ (| Z [i-1, j-1] -Z [i + 1, j + 1] | + | Z [i, j-2] -Z [i + 2, j] | + | Z [i-2, j] -Z [i, j + 2] |) / 8 ... Equation 58 Here, the equation Z [i + 1, j-1] and Z [i-1, j + 1] in 57 and Z [i-1, j-1] and Z [i + 1, j + 1] in Expression 58 are Similar to the representative example 2, the diagonal direction of the pixel to be processed When the pixel at the R position is the processing target pixel, the color information about the blue component is shown, and when the pixel at the B position is the processing target pixel, the color information about the red component is shown. Further, Z [i, j-2], Z [i-2, j], Z [i + 2, j], Z [i, j + 2] and Formula 5 in Formula 57
Z [i, j-2], Z [i + 2, j], Z [i-2, j], Z [i, j + 2] in 8
Indicates color information of the same color as the pixel to be processed.

That is, in the representative example 3, the similarity C45 in the oblique 45 degree direction and the similarity C135 in the oblique 135 degree direction are both the color information of the green component, the color information of the blue component, and the color information of the red component. And is composed of not only the inter-GG similarity component but also the inter-BB similarity component and the inter-RR similarity component. FIG. 18C is a diagram showing a positional relationship of color information used when calculating the degree of similarity according to the representative example 3. However, in FIG. 18C, regarding the similarity C45 in the diagonal 45 degree direction and the similarity C135 in the diagonal 135 degree direction, an example in which the pixel at the R position is the processing target pixel is shown.

As described above, in the representative examples 1 to 3 of the fourth embodiment, the similarity Cv in the vertical direction and the similarity Ch in the horizontal direction are calculated, and the similarity C45 in the oblique 45 ° direction. And when calculating the similarity C135 in the oblique 135 degree direction, individually defined color information is used, and the similarity Cv in the vertical direction and the similarity Ch in the horizontal direction and the similarity C45 in the oblique 45 degree direction are used. And the similarity C135 in the diagonal direction of 135 degrees
And are calculated independently.

Therefore, according to the fourth embodiment, the similarity C45 in the oblique 45-degree direction and the similarity C135 in the oblique 135-degree direction can be calculated by directly referring to the Bayer array type image data. In U.S. Pat. No. 2000-78597 (U.S. Pat. No. 6,075,889), the filtering for generating the blurred Y plane, which is performed when calculating the similarity in each direction, is unnecessary.

Therefore, the accurate similarity can be calculated without losing the information on the fine structure. Also, the fourth
In the representative example 2 and the representative example 3 of the embodiment, the similarity C45 in the diagonal 45 degree direction and the similarity C135 in the diagonal 135 degree direction are described.
When calculating, the color information of at least one color component of the red component and the blue component is used in addition to the green component. M using the coefficient pattern in the 135 degree direction
It is included in the color component indicated by the color information referred to when the g component is generated.

That is, when calculating the similarity C45 in the oblique 45-degree direction and the similarity C135 in the oblique 135-degree direction by the representative examples 2 and 3, the coefficient pattern in the oblique 45-degree direction and the similarity pattern in the oblique 135-degree direction are calculated. M using coefficient pattern
Color information of at least one color component referred to when generating the g component is used. Therefore, the degree of similarity C45 in the oblique 45 degree direction calculated by the representative example 2 and the representative example 3
Further, according to the similarity C135 in the diagonal 135 degree direction, it is possible to accurately set the direction index DN indicating the determination result of the diagonal similarity. As a result, the diagonal luminance component Y _DN can be accurately generated, and the luminance plane can be restored with high precision.

Incidentally, the above-mentioned first to fourth embodiments
In the embodiment, the luminance component is generated at the same pixel position as the Bayer array image data. For example, the similarity Cv in the vertical direction, the similarity Ch in the horizontal direction, and the similarity C4 in the diagonal 45 ° direction are
5, color information to be referred to when calculating the similarity C135 in the oblique 135 degree direction, the G component generation unit 113, the Mg component generation unit 1
By changing the coefficient pattern recorded in advance in 14, it is possible to generate a luminance component at an intermediate position between adjacent pixels of the Bayer array image data.

<< Explanation of Fifth Embodiment >> As an explanation of the fifth embodiment, a color image restoration process for the Bayer array type image data realized by the respective units in the image processing unit 13 will be explained below. However, in the fifth embodiment, among the operations of the respective units in the image processing unit 13, the operation of the luminance component generation processing unit 100 is performed in the same manner as in the above-described first to fourth embodiments. Therefore, detailed description is omitted.

FIG. 19 is a diagram showing a configuration relating to the realization of the image restoration processing in the image processing unit 13 in the fifth embodiment. 19, the image processing unit 13 includes a luminance component generation processing unit 100, a color difference component generation processing unit 200, and a color system conversion processing unit 300, and the color difference component generation processing unit 2
00 is a vertical / horizontal direction similarity determination unit 210, a Crb component generation unit 220, a color difference component generation unit 230, and a color difference interpolation processing unit 24.
It has 0.

Vertical / horizontal direction similarity determination section 210 and Crb
The output of the component generation unit 220 is connected to the color difference component generation unit 230, and the output of the color difference component generation unit 230 is the color difference interpolation processing unit 2.
The output of the luminance component generation processing unit 100 and the color difference interpolation processing unit 240 (corresponding to the output of the color difference component generation processing unit 200) is connected to the color system conversion processing unit 300.

FIG. 20 is a diagram for explaining the operation in the color difference component generation processing section 200. Hereinafter, with reference to FIGS. 19 and 20, a color image restoration process for the Bayer array image data realized by each unit in the image processing unit 13 will be described. First, the operation of each unit in the color difference component generation processing unit 200 will be described. << Description of Vertical-Horizontal Direction Similarity Determining Unit 210 >> The vertical-horizontal direction similarity determining unit 210 refers to the Bayer array image data and
Similarity in the vertical direction (hereinafter referred to as “vertical dissimilarity CvN between different colors”) and horizontal similarity (hereinafter referred to as “horizontal direction”) with respect to the pixel at position B or position B. Of the different color similarity ChN ")).

Then, the vertical / horizontal similarity determination unit 210
The similarity CVN between different colors in the vertical direction is compared with the similarity ChN between different colors in the horizontal direction, and the vertical and horizontal similarities are determined in three stages according to the comparison result, and the direction index HVd indicating the determination result is set. . For example, when the coordinates of the pixel to be processed are [i, j],
The different color similarity CvN [i, j] in the vertical direction and the different color similarity ChN [i, j] in the horizontal direction of the pixel to be processed can be calculated by the following Expressions 59 and 60.

CvN [i, j] = (| G [i, j-1] -Z [i, j] | + | G [i-1, j] -Z [i-1, j + 1] | ) / 2 Equation 59 ChN [i, j] = (| G [i + 1, j] -Z [i, j] | + | G [i, j-1] -Z [i-1, j-1] |) / 2 ... Expression 60 FIG. 21A is a diagram showing the positional relationship of the color information used when calculating the similarity CvN between different colors in the vertical direction.
(B) is a diagram showing a positional relationship of color information used when calculating the similarity ChN between different colors in the horizontal direction. However, in FIG.
In (a) and (b), an example is shown in which the pixel at the R position is the pixel to be processed.

The different color similarity CvN [i, j] in the vertical direction and the different color similarity ChN [i, j] in the horizontal direction calculated as described above.
Indicates that the smaller the value, the stronger the similarity to each direction. Therefore, the direction index HVd [i, j] can be set as follows. That is, the vertical / horizontal direction similarity determination unit 210 determines that for the threshold value Th0, | CvN [i, j] -ChN [i, j] | ≦ Th0 ... It is determined that the similarities are similar, and "0" is set to the direction index HVd [i, j].

If the condition 5 is not satisfied and CvN [i, j] <ChN [i, j] ... Condition 6 is satisfied, the vertical / horizontal similarity determining unit 210 determines that the pixel to be processed is in the vertical direction. It is determined that the similarity is strong, and the direction index HVd [i, j] is set to "1". Further, the vertical / horizontal direction similarity determination unit 210 determines that the processing target pixel has a strong similarity in the horizontal direction when both Condition 5 and Condition 6 are not satisfied, and the direction index HVd [i, j].
Is set to "-1".

Here, the threshold value Th0 in condition 5 is the first
Similar to the threshold value Th1 in the condition 1 described in the above embodiment, the difference between the two similarities (vertical dissimilarity similarity CvN [i, j] and horizontal dissimilarity similarity ChN [i, j]). Is very small, it plays a role of avoiding erroneous determination that one of the similarities is strong due to the influence of noise. Therefore, the threshold Th0
Is preferably set to a value similar to the threshold Th1.

<< Explanation of Crb Component Generation Unit 220 >> Crb
Two types of coefficient patterns for generating color difference components (coefficient patterns for weighted addition of color information of Bayer array image data) are recorded in the component generation unit 220 in advance. Figure 2
2 is a diagram showing such a coefficient pattern.

FIG. 22A shows a coefficient pattern in the vertical direction showing weighted addition coefficients for the color information arranged in the vertical direction of the pixel to be processed, and FIG. 22B shows the coefficient pattern in the horizontal direction of the pixel to be processed. 7 shows a coefficient pattern in the horizontal direction showing the weighted addition coefficient for the color information arranged in FIG. The Crb component generation unit 220 refers to the Bayer array type image data, and for the pixel at the R position or the B position, uses the coefficient pattern in the vertical direction as shown in FIG.
b_v is generated, and the color difference component Crb_h in the horizontal direction is generated using the coefficient pattern in the horizontal direction as shown in FIG.

That is, when the coordinates of the pixel to be processed are [i, j], the color difference component Crb_v [i, j] in the vertical direction and the color difference component Crb_h [i, j] in the horizontal direction are expressed by the following equation 61. And calculated by the equation 62. Crb_v [i, j] = Z [i, j]-(G [i, j-1] + G [i, j + 1]) / 2 ・ ・ ・ Equation 61 Crb_h [i, j] = Z [i , j]-(G [i-1, j] + G [i + 1, j]) / 2 Equation 62 << Description of Color Difference Component Generation Unit 230 >> Color Difference Component Generation Unit 230
Is the color difference component Crb in the vertical direction according to the value of the direction index HVd set by the vertical / horizontal direction similarity determination unit 210 described above.
_v and the color difference component Crb_h in the horizontal direction are used, the color difference component Cr (the color difference component between the red component and the green component) is made to correspond to the pixel at the R position, and the color difference component Cb (the blue component is compared to the pixel at the B position). The color difference component with the green component) is made to correspond.

That is, when the coordinates of the pixel to be processed are [i, j], the color difference component Cr [i, j] at the pixel at the R position is
Calculated by one of the following equations 63 to 65,
The color difference component Cb [i, j] at the pixel at the B position is calculated by the following equation 6
It is calculated by any of 6 to 68. << Example of Calculation of Color Difference Component Cr at Pixel at R Position >> When HVd [i, j] = 1: Cr [i, j] = Crb_v [i, j] ... Equation 63 HVd [i, j] = 0 When: Cr [i, j] = (Crb_v [i, j] + Crb_h [i, j]) / 2 ・ ・ ・ Equation 64 HVd [i, j] =-1: Cr [i, j] = Crb_h [i, j] ・ ・ ・ Equation 65 << Calculation example of color difference component Cb in pixel at B position >> When HVd [i, j] = 1: Cb [i, j] = Crb_v [i, j] ..When Expression 66 HVd [i, j] = 0: Cb [i, j] = (Crb_v [i, j] + Crb_h [i, j]) / 2 ... Expression 67 HVd [i, j] = -1: Cb [i, j] = Crb_h [i, j] (Equation 68) Further, the color difference component generation unit 230 implements the interpolation processing by the color difference interpolation processing unit 240 described later by filtering processing. , "0" is set as the color difference component Cr in pixels other than the R position (pixels at the G position and B position), and "0" is set as the color difference component Cb in pixels other than the B position (pixels at the G position and R position). Set.

As described above, the surface in which the color difference component Cr is associated with the position of each pixel of the Bayer array type image data is called the Cr surface, and the surface in which the color difference component Cb is associated is called the Cb surface. << Explanation of Color Difference Interpolation Processing Unit 240 >> Color difference interpolation processing unit 240
Performs the filtering by the fixed filter as shown in FIG. 23 on the Cr plane and the Cb plane to interpolate the color difference component Cr and the color difference component Cb.

That is, the interpolation of the color difference component Cr can be described in the form of a separable filter by the following expressions 69 and 70.
It is realized by the calculation of. tmp_Cr [i, j] = (2 · Cr [i, j] + Cr [i-1, j] + Cr [i + 1, j]) / 2 ・ ・ ・ Equation 69 Cr [i, j] = ( 2 · tmp_Cr [i, j] + tmp_Cr [i, j-1] + tmp_Cr [i, j + 1]) / 2 Equation 70 Further, the interpolation of the color difference component Cb is also realized by the same calculation. It

Color difference component obtained as described above
The Cr and the color difference component Cb are supplied to the color system conversion processing unit 300 as two types of color difference surfaces. Note that the color difference surface may be supplied to the color system conversion processing unit 300 after being subjected to correction processing by a low-pass filter, a median filter, or the like. << Explanation of Color System Conversion Processing Unit 300 >> Color system conversion processing unit 3
00 is a luminance plane obtained via the luminance component generation processing unit 100 as in the first to fourth embodiments,
As described above, the color difference conversion processing is performed from the color difference surface supplied from the color difference interpolation processing unit 240 to generate image data in which color information of R, G, and B colors is associated with each pixel. The image data generated in this way is output as image data that has been subjected to color image restoration processing for the Bayer array image data.

For example, when the coordinates of the pixel to be processed are [i, j], R, which is output in association with the pixel to be processed,
Color information of G and B color components Rout [i, j], Gout [i, j], Bout [i, j]
Can be calculated by the following Expressions 71 to 73 using β defined in Expression 11. Rout [i, j] = Y [i, j] + (1-β / 2) ・ Cr [i, j]-(β / 2) ・ Cb [i, j] ・ ・ ・ Equation 71 Gout [i, j] = Y [i, j]-(β / 2) ・ Cr [i, j]-(β / 2) ・ Cb [i, j]) / 4 ・ ・ ・ Equation 72 Bout [i, j] = Y [i, j]-(β / 2) · Cr [i, j] + (1-β / 2) · Cb [i, j]) / 4 Equation 73 As described above, the fifth In the embodiment of the present invention, by adding the color difference surface generation processing of a simple mechanism to the high definition luminance surface generation processing of the first to fourth embodiments independently and newly referring to the Bayer array type image data, Fine color image restoration processing becomes possible.

In the fifth embodiment, the color difference component is calculated in accordance with the value of the direction index HVd indicating the result of the vertical / horizontal similarity determination section 210 indicating the vertical / horizontal similarity. The determination of the vertical-horizontal similarity by 210 is shown in FIG.
1 (a) and (b), the similarity CvN, Ch between different colors composed of color information of different color components arranged at one pixel intervals
Performed using N. Therefore, even if the Bayer array type image data has vertical stripes or horizontal stripes of Nyquist frequency, vertical-horizontal similarity can be accurately determined, and the peripheral portion of the vertical stripes or horizontal stripes of Nyquist frequency can be determined. It is possible to reduce the occurrence of false color in.

Therefore, according to the fifth embodiment, it is possible to perform high-definition color image restoration processing with a high degree of false color suppression on the Bayer array image data. In addition,
In the fifth embodiment, the color difference component Cr at the R position and the color difference component Cb at the B position are directional indexes set by comparing the similarity CvN between different colors in the vertical direction and the similarity ChN between different colors in the horizontal direction. It is calculated using HVd, but instead of the direction index HVd, the vertical similarity Cv and the horizontal similarity Ch calculated by the representative examples 1 to 3 in the fourth embodiment are compared. The direction index HVs set by the above may be used.

As described above, when the direction index HVs is used instead of the direction index HVd, the R / B position luminance component generating section 110
Since the color difference component generation unit 230 in the color difference component generation processing unit 200 may refer to the direction index HVs set by the vertical / horizontal direction similarity determination unit 111 in FIG. Since it is not necessary to provide the determination unit 210, the scale of the hard wafer can be reduced.

<< Explanation of Sixth Embodiment >> As an explanation of the sixth embodiment, a color image restoration process for the honeycomb array image data realized by the respective units in the image processing unit 13 will be explained below. FIG. 24 is a diagram showing a configuration relating to implementation of image restoration processing in the image processing unit 13 in the sixth embodiment.

However, in FIG. 24, the same reference numerals as those in FIG. 19 are given to the same components as those in the fifth embodiment described above. 24, the image processing unit 13 includes a 45-degree rotation processing unit 400, a luminance component generation processing unit 100, a color difference component generation processing unit 200, a color system conversion processing unit 300, and a 45-degree reverse rotation processing unit 500. .

The output of the 45 ° rotation processing unit 400 is connected to the luminance component generation processing unit 100 and the color difference component generation processing unit 200, and the output of the luminance component generation processing unit 100 and the output of the color difference component generation processing unit 200 are Color system conversion processing unit 30
0, and the output of the color system conversion processing unit 300 is 45
The reverse rotation processing unit 500 is connected. Hereinafter, with reference to FIG. 24, a color image restoration process for the honeycomb array image data realized by each unit in the image processing unit 13 will be described.

<< Description of 45-degree Rotation Processing Unit 400 >> The 45-degree rotation processing unit 400 performs image processing for rotating the honeycomb array image data by 45 degrees. As a result, FIG.
The honeycomb array image data as shown in FIG.
5 (b) is converted into the Bayer array type image data. Such Bayer array image data is supplied to the luminance component generation processing unit 100 and the color difference component generation processing unit 200.

<< Explanation of Luminance Component Generation Processing Unit 100 >> As in the fifth embodiment, the luminance component generation processing unit 100 refers to the Bayer array type image data and refers to the luminance components Y and G at the R / B positions. The position luminance component Y is generated, and the luminance surface obtained by combining these luminance components is corrected and output as necessary.

<< Description of Color Difference Component Generation Processing Unit 200 >> The color difference component generation processing unit 200 calculates the color difference component Cr and the color difference component Cb by referring to the Bayer array image data, as in the fifth embodiment. Interpolate to generate a color difference surface. << Explanation of Color System Conversion Processing Unit 300 >> Color system conversion processing unit 3
As in the fifth embodiment, 00 generates image data in which color information of R, G, and B colors is associated with each pixel by colorimetric conversion processing from the luminance surface and the color difference surface.

<< Description of 45-degree Reverse Rotation Processing Unit 500 >> 45
The degree reverse rotation processing unit 500 performs image processing of reverse rotation of 45 degrees on the image data generated by the color system conversion processing unit 300. As a result, image data that has been subjected to color image restoration processing for the honeycomb array type image data is obtained.

As described above, according to the sixth embodiment, a highly accurate color image restoration process can be performed on the honeycomb array image data as in the fifth embodiment. . Similarly, by using the first to fourth embodiments rotated by 45 degrees, highly accurate image restoration processing of a monochrome image can be performed.

In each of the above-described embodiments, the image processing unit 13 implements the image restoration process by hardware such as ASIC. However, the series of operations performed by each unit in the image processing unit 13 is It is also possible to realize it by software such as. << Seventh Embodiment >> The operation of the seventh embodiment will be described below.

The seventh embodiment is equivalent to executing the image processing program of the present invention by the PC 17 shown in FIG. 1 and performing image restoration processing on the image data input by the image input apparatus 1. However, in the seventh embodiment, unlike the above-described embodiments, the image processing unit 13 illustrated in FIG. 1 does not perform the image restoration processing, and performs image processing other than the image restoration processing (for example, pixel defect correction). Assumed to be performed.

Further, the PC 17 is instructed by the PC to perform the operations performed by the respective units in the image processing unit 13 of any of the above-described embodiments.
An image processing program to be sequentially realized by 17 is installed in advance via a recording medium such as a CD-ROM 21. In the image input device 1, the image data generated by the image sensor 11 and digitized by the A / D converter 12 is supplied to the image processor 13 via the bus. The image processing unit 13 performs image processing such as pixel defect correction on the image data thus supplied. The image data that has undergone such image processing is supplied to the PC 17 via the external interface unit 18, or recorded in the memory card 15 and read out by the PC 17.

The PC 17 executes the image processing section program as described above, and performs the image restoration processing on the image data supplied via the external interface section 18 or the image data read from the memory card 15. As described above, in the seventh embodiment, the PC 17 can perform the image restoration process realized in any of the above-described embodiments.

In the seventh embodiment, the image processing program as described above is executed by the PC 17 connected to the image input apparatus 1. However, such an image processing program is not executed by the PC 17, It may be executed by a remote server or the like connected via the Internet or the like.
That is, the PC 17 simply transfers the image data supplied from the image input device 1 to a server or the like capable of executing the above-described image processing program via the Internet or the like, and the image data subjected to the image restoration processing is transferred. Obtainable.

Further, in each of the above-described embodiments, an example of realizing high image quality by image restoration processing has been shown, but the present invention is not limited to image restoration processing, and high image quality is realized by other image processing. It is also possible to apply when doing.

[0193]

As described above, according to the inventions described in claims 1 to 36, it is possible to easily realize highly accurate image quality improvement for a multispectral image by hardware.

According to the thirty-seventh to thirty-ninth aspects of the present invention, it is possible to easily realize a highly accurate image quality improvement for a multispectral image by a computer.

[Brief description of drawings]

FIG. 1 is a functional block diagram of an image input device.

FIG. 2 is a diagram showing an array of color components of Bayer array type image data.

FIG. 3 is a diagram showing a configuration relating to implementation of image restoration processing in an image processing unit.

FIG. 4 is a diagram illustrating an operation of each unit in a luminance component generation processing unit.

FIG. 5 is a diagram showing a coefficient pattern.

FIG. 6 is a diagram showing an array of color information used when generating a luminance component.

FIG. 7 is a diagram showing a coefficient pattern.

FIG. 8 is a diagram showing a coefficient pattern.

FIG. 9 is a diagram showing a direction of strong similarity corresponding to the value of (HVs, DN).

FIG. 10 is a diagram showing a coefficient pattern.

FIG. 11 is a diagram showing an example of a bandpass filter.

FIG. 12 is a diagram showing an array of color information used when generating a luminance component.

FIG. 13 is a diagram illustrating an operation of each unit in the luminance component generation processing unit according to the second embodiment.

FIG. 14 is a diagram showing a coefficient pattern.

FIG. 15 is a diagram showing a coefficient pattern.

FIG. 16 is a diagram illustrating operations of a G component generation unit and a Mg component generation unit according to the third embodiment.

FIG. 17 is a diagram showing a coefficient pattern.

FIG. 18 is a diagram showing a positional relationship of color information used when calculating similarity.

FIG. 19 is a diagram showing a configuration relating to implementation of image restoration processing in the image processing unit.

FIG. 20 is a diagram illustrating an operation of each unit in the color difference component generation processing unit.

FIG. 21 is a diagram showing a positional relationship of color information used when calculating similarity.

FIG. 22 is a diagram showing a coefficient pattern.

FIG. 23 is a diagram showing an example of an interpolation filter.

FIG. 24 is a diagram showing a configuration relating to realization of image restoration processing.

[Fig. 25] Fig. 25 is a diagram for describing conversion from a honeycomb array type image to Bayer array type image data.

FIG. 26 is a diagram illustrating a conventional technique.

FIG. 27 is a diagram illustrating a conventional technique.

[Explanation of symbols]

1 Image input device 11 Image sensor 12 A / D converter 13 Image processing unit 14 Control unit 15 memory card 16 Memory card interface 17 PC (personal computer) 18 External interface section 19 display 20 printers 21 CD-ROM 100 Luminance component generation processing unit 110 R / B position luminance component generation unit 111, 210 vertical / horizontal similarity determination unit 112 diagonal similarity determination unit 113 G component generator 114 Mg component generator 115 Vertical and Horizontal Luminance Component Generation Unit 116 diagonal luminance component generation unit 117 Luminance component synthesizer 120 G position luminance component generation unit 130 Luminance plane correction unit 200 Color difference component generation processing unit 220 Crb component generator 230 Color difference component generator 240 Color difference interpolation processing unit 300 Color system conversion processor 400 45 degree rotation processing unit 500 45 degree reverse rotation processing unit

   ─────────────────────────────────────────────────── ─── Continued front page    F-term (reference) 5B057 CA01 CA08 CA12 CA16 CB01                       CB08 CB12 CB16 CE17 CE18                       CH01 CH11                 5C066 AA01 CA05 EC02 GA01 GA02                       GA05 HA03 JA03 KE01 KE05                       KM05                 5C077 MP08 PP32 PP34 PP37 PP46                       PQ12 PQ18 PQ22                 5C079 HB01 HB04 HB11 LB02 MA01                       MA11 NA29                 5C082 AA01 BA12 BA34 BA35 BA39                       BB15 BB53 BD09 CA12 CA42                       CA81 DA53 DA61 DA86 MM09                       MM10

Claims (39)

[Claims] 1. A multi-spectral image represented by a plurality of colors and having one color information in each of the pixels arranged in a two-dimensional square lattice pattern is different from the color system of the multi-spectral image. In an image processing device that performs image processing for generating a luminance component of a color system, a first direction coefficient pattern indicating a weighted addition coefficient for color information arranged in a first direction and a first direction coefficient pattern orthogonal to the first direction. A second direction coefficient pattern indicating a weighted addition coefficient for color information arranged in two directions, and a weighted addition for color information arranged in a third direction different from the first direction and the second direction. Recording means for recording a third directional coefficient pattern indicating the coefficient and a fourth directional coefficient pattern indicating the coefficient of the weighted addition for the color information arranged in the fourth direction orthogonal to the third direction, First direction clerk First pattern generating means for generating a first coefficient pattern group including at least three types of coefficient patterns using the pattern and the second direction coefficient pattern; the third direction coefficient pattern and the fourth direction coefficient Pattern, and second pattern generating means for generating a second coefficient pattern group including at least three types of coefficient patterns, and the first coefficient pattern group and the second coefficient pattern group are combined. A third pattern generating means for generating a third coefficient pattern group including at least nine types of coefficient patterns, and a coefficient pattern of any one of the third coefficient pattern group, An image processing apparatus, comprising: a luminance component generating unit that weights and adds color information to generate a luminance component. 2. The image processing apparatus according to claim 1, wherein the first pattern generation means includes two types of coefficient patterns, the first direction coefficient pattern and the second direction coefficient pattern, and the two types. And a coefficient pattern obtained by averaging the coefficient patterns of 1) to generate a first coefficient pattern group. 3. The image processing apparatus according to claim 1, wherein the second pattern generation unit includes two types of coefficient patterns, the third direction coefficient pattern and the fourth direction coefficient pattern, and the two types. And a coefficient pattern obtained by averaging the coefficient patterns of 1. 4. The image processing apparatus according to claim 1, wherein the third pattern generation unit includes at least three types of coefficient patterns forming the first coefficient pattern group, and the second coefficient pattern group. The image processing apparatus is characterized by generating at least nine kinds of coefficient patterns by combining and averaging at least three kinds of coefficient patterns constituting the above. 5. The image processing apparatus according to claim 1, wherein the similarity regarding the first direction and the similarity regarding the second direction are calculated, and the first direction is calculated based on the similarity. And a first similarity determination unit that determines the similarity between the second direction and at least three stages, and the first pattern generation unit can be determined by the first similarity determination unit. The first direction coefficient pattern and the second direction coefficient pattern may be selected according to the difference in similarity.
An image processing apparatus, wherein the first coefficient pattern group is generated by synthesizing with a direction coefficient pattern. 6. The image processing apparatus according to claim 1, wherein the similarity regarding the third direction and the similarity regarding the fourth direction are calculated, and the third direction is calculated based on the similarity. And a second similarity determining unit that determines the similarity between the second direction and the fourth direction in at least three stages, and the second pattern generating unit can be determined by the second similarity determining unit. The third direction coefficient pattern and the fourth direction coefficient pattern may be selected according to the difference in similarity.
An image processing apparatus, wherein the second coefficient pattern group is generated by synthesizing with a direction coefficient pattern. 7. The image processing apparatus according to claim 1, wherein the recording unit displays the multispectral image in a color system that includes first to third color components, and the first color component. Is higher than the spatial densities of the second color component and the third color component, the color information of the first color components arranged in the first direction is averaged as the first direction coefficient pattern. And a coefficient pattern for averaging color information of the first color components arranged in the second direction is recorded as the second direction coefficient pattern, and the third direction coefficient is recorded. As a pattern, second patterns arranged in the third direction
And a coefficient for weighted averaging the color information of the third color component is recorded, and as the fourth direction coefficient pattern, the fourth direction coefficient pattern is recorded.
An image processing apparatus for recording a coefficient for weighted averaging the color information of the second and third color components arranged in the direction. 8. The image processing apparatus according to claim 1, wherein the recording unit records a coefficient pattern that does not include a negative value. 9. The image processing apparatus according to claim 1, wherein the brightness component generating means generates a brightness component at the same pixel position as the multispectral image. 10. The image processing apparatus according to claim 1, wherein the recording unit uses the luminance component as a coefficient pattern forming at least one of the first coefficient pattern group and the second coefficient pattern group. An image processing apparatus, wherein a coefficient pattern showing a positive value of a weighted addition coefficient corresponding to color information of a generation target pixel position is recorded. 11. The image processing apparatus according to claim 1, wherein the recording unit includes color information of all color components that constitute a color system of the multispectral image centered on a luminance component generation target pixel. An image processing apparatus, which records a coefficient pattern for weighted addition of color information in the narrowest range. 12. The image processing apparatus according to claim 1, further comprising a correction unit that performs edge enhancement processing by a fixed filter on the luminance component generated by the luminance component generation unit. apparatus. 13. The image processing apparatus according to claim 1, further comprising a color difference component generation unit that generates a color difference component using color information of the multispectral image. 14. The image processing apparatus according to claim 13, wherein the color difference component generating means calculates a degree of similarity in at least two directions to determine similarity, and based on the determination result of the similarity, the color difference component. An image processing apparatus, wherein: 15. The image processing apparatus according to claim 14, wherein the similarity is an inter-color similarity composed of color information between different color components of the multispectral image. apparatus. 16. A color system different from the color system of the multi-spectral image for a multi-spectral image represented by a plurality of colors and having one color information in each pixel arranged in a two-dimensional grid pattern. In an image processing device that performs image processing for generating a brightness component of a system, a pixel connecting a brightness component generation target pixel and a nearest pixel arranged at a position closest to the brightness component generation target pixel n
₁ (n ₁ ≧ 2) pieces of the direction of each ordered n ₁ coefficients pattern indicating the coefficients of the weighted addition with respect to the color information, the to the luminance component subject pixels and luminance components subject pixels second proximity connecting the pixel n ₂ (n ₂ ≧ 2) pieces weighted n ₂ coefficients indicating the coefficients of the addition for each color information arranged in the direction of which is arranged in a position close to the next nearest pixel recording means for recording a pattern, using the n ₁ coefficients patterns, m ₁ (m _1> n ₁₎ a first pattern generating means for generating a first coefficient pattern group consisting of pieces of coefficient pattern When using the n ₂ coefficients patterns, m ₂ (m _1> n ₂₎ and the second pattern generating means for generating a second coefficient pattern group consisting of pieces of coefficient pattern, the first coefficient By combining the pattern group and the second coefficient pattern group, L ( L> m ₁ and L> m ₂ ) A third coefficient pattern group consisting of coefficient patterns
And a luminance component generation unit for generating the luminance component by weighted addition of color information of the multispectral image using any one of the coefficient patterns of the third coefficient pattern group. An image processing device characterized by the above. The image processing apparatus according to claim 17 claim 16, wherein the first pattern generating means, said n ₁ coefficients patterns, at least obtained by averaging between the n ₁ coefficients pattern An image processing apparatus, which generates a first coefficient pattern group consisting of one coefficient pattern. The image processing apparatus according to claim 18 claim 16, wherein the second pattern generating means, said n ₂ coefficients patterns, at least obtained by averaging between the n ₂ coefficients pattern An image processing apparatus, wherein a second coefficient pattern group including one coefficient pattern is generated. 19. The image processing apparatus according to claim 16, wherein the third pattern generation unit includes m ₁ coefficient patterns forming the first coefficient pattern group, and the second coefficient pattern group. The image processing apparatus is characterized in that m ₁ × m ₂ coefficient patterns are generated as the L coefficient patterns by averaging by combining the m ₂ coefficient patterns constituting the above. 20. The image processing apparatus according to claim 16, wherein the first pattern generation unit connects the luminance component generation target pixel and the closest pixel to n.
Depending on the similarity of the differences between _one direction, and combining the n ₁ coefficients pattern, an image processing apparatus and generates the first coefficient pattern group. 21. The image processing device according to claim 16, wherein the second pattern generation unit is configured to determine similarity between n ₂ directions connecting the luminance component generation target pixel and the second adjacent pixel. An image processing apparatus, wherein the n ₂ coefficient patterns are combined according to a difference to generate the second coefficient pattern group. 22. The image processing apparatus according to claim 16, wherein the recording unit records a coefficient pattern that does not include a negative value. 23. The image processing device according to claim 16, wherein the luminance component generation means generates a luminance component at the same pixel position as the multispectral image. 24. The image processing apparatus according to claim 16, wherein the recording unit uses the luminance component as a coefficient pattern that constitutes at least one of the first coefficient pattern group and the second coefficient pattern group. An image processing apparatus, wherein a coefficient pattern showing a positive value of a weighted addition coefficient corresponding to color information of a generation target pixel position is recorded. 25. The image processing apparatus according to claim 16, wherein the recording unit displays the multispectral image in a color system including first to Nth (N ≧ 3) color components, as the n ₁ coefficients patterns, record the coefficient patterns for performing weighted addition of color information of at least a first color component, the n as _two coefficient pattern, at least one color component of the second to n An image processing apparatus, which records a coefficient pattern for weighted addition of color information of components. 26. The image processing apparatus according to claim 16, further comprising a correction unit that performs edge enhancement processing by a fixed filter on the luminance component generated by the luminance component generation unit. apparatus. 27. The image processing apparatus according to claim 16, further comprising a color difference component generation unit that generates a color difference component using color information of the multispectral image. 28. The image processing apparatus according to claim 27, wherein the color difference component generating means calculates a degree of similarity in at least two directions to determine similarity, and a color difference component based on a result of the similarity determination. An image processing apparatus, wherein: 29. The image processing apparatus according to claim 28, wherein the degree of similarity is a degree of similarity between different colors configured by color information between different color components of the multispectral image. apparatus. 30. A multispectral image represented by first to nth (n ≧ 2) color components and arranged in a two-dimensional square lattice, in which one color information exists, In an image processing device that performs image processing for generating a luminance component of a color system different from the color system of a spectrum image, the color information of the multispectral image is directly referred to
Of the first direction, the similarity of the second direction orthogonal to the first direction, the similarity of the third direction different from the first direction and the second direction, and the third direction of A similarity calculation means for calculating a similarity in a fourth direction orthogonal to the direction, and a first direction coefficient indicating a weighted addition coefficient for color information of at least the first color component arranged in the first direction. A pattern, a second direction coefficient pattern indicating a coefficient of weighted addition for color information of at least a first color component arranged in the second direction, and at least a first color component other than the first color component arranged in the third direction. A third direction coefficient pattern indicating a weighted addition coefficient for color information and a fourth direction coefficient pattern indicating a weighted addition coefficient for color information other than at least the first color component arranged in the fourth direction are recorded. Image processing comprising: a recording unit; and a luminance component generating unit that generates a luminance component using the similarity calculated by the similarity calculating unit and the coefficient pattern recorded in the recording unit. apparatus. 31. The image processing apparatus according to claim 30, wherein at least one coefficient pattern of the first direction coefficient pattern and the second direction coefficient pattern, the third direction coefficient pattern, and the fourth direction coefficient. An image processing apparatus comprising: a synthesizing unit that synthesizes a coefficient pattern for generating the luminance component by using a pattern and at least one coefficient pattern. 32. The image processing apparatus according to claim 30, wherein the similarity calculation unit calculates the similarity regarding the first direction and the similarity regarding the second direction using predetermined color information. The image processing apparatus is characterized by calculating the similarity regarding the third direction and the similarity regarding the fourth direction independently of the two similarities by using predetermined color information. 33. The image processing apparatus according to claim 32, wherein the similarity calculation unit calculates the similarity in the third direction and the similarity in the fourth direction in the third direction. At least 1 constituting the coefficient pattern or the fourth direction coefficient pattern
An image processing apparatus characterized by using color information of three color components. 34. The image processing apparatus according to claim 30, further comprising a color difference component generating unit that generates a color difference component using the color information of the multispectral image. 35. The image processing apparatus according to claim 34, wherein the color difference component generating means calculates similarity in at least two directions to determine similarity, and the color difference component based on the similarity determination result. An image processing apparatus, wherein: 36. The image processing apparatus according to claim 35, wherein the degree of similarity is a degree of similarity between different colors configured by color information between different color components of the multispectral image. apparatus. 37. A multi-spectral image represented by a plurality of colors and having one color information in each pixel arranged in a two-dimensional square lattice pattern is different from the color system of the multi-spectral image. In an image processing program for causing a computer to realize image processing for generating a luminance component of a color system, a first direction coefficient pattern indicating a weighted addition coefficient for color information arranged in a first direction, and the first direction coefficient pattern. A second direction coefficient pattern indicating a weighted addition coefficient for color information arranged in a second direction orthogonal to the direction; and a second direction coefficient pattern arranged in a third direction different from the first direction and the second direction. A third directional coefficient pattern indicating a weighted addition coefficient for color information and a fourth directional coefficient pattern indicating a weighted addition coefficient for color information arranged in a fourth direction orthogonal to the third direction. A recording procedure for recording; a first pattern generation procedure for generating a first coefficient pattern group composed of at least three types of coefficient patterns using the first direction coefficient pattern and the second direction coefficient pattern; A second pattern generation procedure for generating a second coefficient pattern group composed of at least three types of coefficient patterns by using a third direction coefficient pattern and the fourth direction coefficient pattern; and the first coefficient pattern group. A third pattern generation procedure for combining the second coefficient pattern group to generate a third coefficient pattern group including at least nine kinds of coefficient patterns; and any one of the third coefficient pattern group. And a luminance component generating procedure for generating a luminance component by weighted addition of color information of the multispectral image using a coefficient pattern. Grams. 38. A color system different from the color system of the multispectral image for a multispectral image represented by a plurality of colors and having one color information in each pixel arranged in a two-dimensional grid pattern. In an image processing program for causing a computer to realize image processing for generating a brightness component of a system, a pixel connecting a brightness component generation target pixel and a nearest pixel arranged at a position closest to the brightness component generation target pixel n
₁ (n ₁ ≧ 2) pieces of the direction of each ordered n ₁ coefficients pattern indicating the coefficients of the weighted addition with respect to the color information, the to the luminance component subject pixels and luminance components subject pixels second proximity connecting the pixel n ₂ (n ₂ ≧ 2) pieces weighted n ₂ coefficients indicating the coefficients of the addition for each color information arranged in the direction of which is arranged in a position close to the next nearest pixel a recording procedure for recording the pattern, using the n ₁ coefficients pattern, the first pattern generation step of generating a first coefficient pattern group consisting of m ₁ (m _1> n ₁₎ pieces of coefficient pattern When, using said n ₂ coefficients patterns, m ₂ (m _1> n ₂₎ and the second pattern generation step of generating a second coefficient pattern group consisting of pieces of coefficient pattern, the first coefficient By combining the pattern group and the second coefficient pattern group, L ( L> m ₁ and L> m ₂ ) A third coefficient pattern group consisting of coefficient patterns
And a luminance component generation procedure for generating the luminance component by performing weighted addition of color information of the multispectral image using any one coefficient pattern of the third coefficient pattern group. An image processing program characterized by the above. 39. A multi-spectral image represented by first to n-th (n ≧ 2) color components and having one color information in each of pixels arranged in a two-dimensional square lattice, An image processing program for causing a computer to realize image processing for generating a luminance component of a color system different from the color system of a spectrum image, by directly referring to the color information of the multispectral image,
, A similarity regarding a second direction orthogonal to the first direction, a similarity regarding a third direction different from the first direction and the second direction, and the third A similarity calculation procedure for calculating a similarity with respect to a fourth direction orthogonal to the direction, and a first direction coefficient indicating a weighted addition coefficient for color information of at least the first color component arranged in the first direction. A pattern, a second direction coefficient pattern indicating a weighted addition coefficient for color information of at least the first color component arranged in the second direction, and at least a first color component other than the first color component arranged in the third direction. A third directional coefficient pattern indicating a weighted addition coefficient for color information and a fourth directional coefficient pattern indicating a weighted addition coefficient for color information other than at least the first color component arranged in the fourth direction are recorded. Image processing, comprising: a recording procedure; and a luminance component generation procedure for generating a luminance component using the similarity calculated by the similarity calculation procedure and the coefficient pattern recorded in the recording procedure. program.