JPH1013845A - Picture processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To execute optimum estimation with respect to optional structure by executing adaptive estimation depending on structure of a picture by using spatially correlated information concerning the local area of a picture signal to estimate. SOLUTION: Picture information from a CCD 3 is stored in a picture memory 7 by way of an A/D converter 5 and then stored in a picture information storage device 10 within a picture processor 19. In addition, arranging infomation of a color filter 2 stored in ROM 6 is stored in an information storage device for an image pickup device 9. Next, an estimated position deciding device 11 decides the position of the picture signal to estimate and an area extracting device 12 extracts the local area of a prescribed size involving this area. A calculator 14 calculates spatially correlated information concerning this local area and a picture signal estimating device 17 estimates a picture signal corresponding to the position of the picture signal obtained from the device 11. Thereby optimum estimation is executed with respect to optional structure by executing adaptive estimation depending on the structure of the picture by using spatial correlation information concerning the local area.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus for estimating an image signal based on spatial correlation information calculated for a local area.

[0002]

2. Description of the Related Art In recent years, an imaging system using a solid-state imaging device such as a CCD instead of a conventionally used silver halide film has been rapidly spreading. In such an imaging system, color image information is stored as color information of three colors or luminance and color difference information. For example, in an imaging system such as a digital camera, the principle is that three types of filters having different spectral sensitivities corresponding to three colors are generally arranged on the front surface of a CCD to acquire respective color signal information. To obtain signal information, 3 per pixel
Different types of sensors are required.

As described above, when an imaging system is realized, an optical system for splitting incident light from a subject and three CCs having three types of filters having different spectral sensitivities are arranged.
D, a three-plate type imaging device is used.
The use of a single CCD and a split optical system results in an expensive device.

[0004] Therefore, a single-panel digital camera in which color filters are appropriately arranged in each pixel of one CCD and image information having different color information is obtained in each pixel is widely used as an inexpensive device. In this case, image information having color information of three colors at each pixel is estimated by post-processing such as spatial interpolation processing.

The most basic method of color information interpolation in a single-chip type image pickup apparatus uses linear interpolation from pixel information in the vicinity of an interpolation pixel. This interpolation processing is performed slowly between pixels used for image information for interpolation. Since the processing assumes a case where the color of the image has been changed, a false color is generated in a region including many high spatial frequency components such as an edge portion of an image.
Several methods for estimating image information have been proposed to improve such problems.

[0006] First, by recognizing the edge of an image or a specific image structure and adaptively changing the processing method according to the image structure, it is possible to cope with a structure in which an estimation error generally increases. For example, in JP-A-6-303617,
Three types of interpolation methods are switched and used in accordance with the luminance change near the estimated pixel. Also, JP-A-61-501423
In the publication, the interpolation method is switched by matching a previously prepared pattern with the image near the estimated pixel.

[0007] An estimation method using correlation between color information of three colors has also been proposed. For example, JP-A-5-56446
In this publication, it is assumed that there is a high correlation between low spatial frequency components for each color, and estimated color information is obtained from acquired color information × (estimated color of low spatial frequency component / acquired color information of estimated pixel of low spatial frequency component). I'm asking. In addition, Japanese Unexamined Patent Publication No.
In Japanese Patent No. 502096, estimation is performed by linear interpolation from the vicinity of an estimated pixel according to a change in first color information sampled at a high spatial frequency.

[0008]

Any of the above-mentioned prior arts is an adaptive estimation method depending on an image structure.
An estimation method using spatial correlation information on a local region of an image signal to be estimated has not been proposed.

Therefore, the image processing apparatus of the present invention performs adaptive estimation depending on the image structure using the spatial correlation information on the local region of the image signal to be estimated, so that an image having an arbitrary structure can be obtained. An object of the present invention is to provide an image processing device capable of performing optimal estimation.

[0010]

In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention determines a position of an image signal to be estimated based on information on an imaging apparatus that generates an image signal. Estimated position determining means for extracting, a region extracting means for extracting a local area of a predetermined size including the position of the image signal obtained from the estimated position determining means, and a spatially related local area obtained from the area extracting means. Spatial correlation information calculating means for calculating correlation information, and an image signal corresponding to the position of the image signal obtained from the estimated position determining means based on the spatial correlation information calculated by the spatial correlation information calculating means Image signal estimating means for estimating.

Further, an image processing apparatus according to a second aspect of the present invention comprises:
In the image processing apparatus according to the first invention, a structure information extracting means for extracting structure information relating to a local region obtained from the area extracting means, and the spatial information extraction means based on the structure information extracted by the structure information extracting means. A changing unit for changing a method of calculating spatial correlation information by the correlation information calculating unit.

Further, an image processing apparatus according to a third aspect of the present invention comprises:
Estimated position determining means for determining a position of an image signal to be estimated based on information on an imaging device that generates an image signal composed of a plurality of color signals, and a position of the image signal obtained from the estimated position determining means. Region extracting means for extracting a local region having a predetermined size from an image signal including a plurality of color signals; spatial correlation information calculating means for calculating spatial correlation information on the local region obtained from the region extracting device; Color correlation information calculation means for calculating color correlation information relating to a local area obtained from the area extraction means; spatial correlation information calculated by the spatial correlation information calculation means; and color calculated from the color correlation information calculation means. Image signal estimating means for estimating an image signal corresponding to the position of the image signal obtained from the estimated position determining means based on the correlation information.

That is, the image processing apparatus according to the first aspect of the present invention first determines the position of the image signal to be estimated by the estimated position determining means based on the information on the imaging device that generates the image signal. A local area of a predetermined size including the position of the image signal obtained by the means is extracted by the area extracting means. Next, spatial correlation information on the local region obtained from the region extracting unit is calculated by the spatial correlation information calculating unit, and based on the calculated spatial correlation information, the image obtained from the estimated position determining unit is calculated. An image signal corresponding to the position of the signal is estimated by the image signal estimating means.

An image processing apparatus according to a second aspect of the present invention
In the image processing apparatus according to the first invention, the structural information relating to the local region obtained from the area extracting means is extracted by the structural information extracting means, and the spatial correlation information calculating means extracts the spatial information based on the extracted structural information. The method of calculating the statistical correlation information is changed by the changing means.

An image processing apparatus according to a third aspect of the present invention comprises:
First, a position of an image signal to be estimated is determined by an estimated position determining unit based on information on an imaging device that generates an image signal including a plurality of color signals, and a position of the image signal obtained from the estimated position determining unit is determined. Is extracted by an area extracting means from an image signal including a plurality of color signals. Next, the spatial correlation information regarding the local area obtained from the area extracting means is calculated by the spatial correlation information calculating means, and the color correlation information regarding the local area obtained from the area extracting means is calculated by the color correlation information calculating means. Based on the spatial correlation information calculated by the spatial correlation information calculating means and the color correlation information calculated by the color correlation information calculating means, the position of the image signal obtained from the estimated position determining means. Is estimated by the image signal estimating means.

[0016]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram illustrating an image processing system to which the first embodiment of the present invention is applied, and includes a single-panel digital camera 4 and an image processing device 19.
The single-panel digital camera 4 includes a lens 1 and a single-panel CC
D3 and three types of (R, G, B) color filters 2 having different spectral sensitivities arranged in a mosaic shape with respect to the CCD 3.
, An A / D converter 5, an image memory 7, and a ROM 6. Here, as shown schematically in FIG. 2, each of the R, G, and B color filters 2 has one R and B filter in a rectangular area of 2 × 2 pixels, and two G filters.
It has a structure in which unit structures included in pixels are periodically repeated and arranged.

The image processing device 19 includes an imaging device information storage device 9, an image information storage device 10, an estimated position determination device 11, an area extraction device 12, and a correlation range storage device 1.
3, a spatial correlation information calculation device 14, an estimation coefficient calculation device 15, an estimation coefficient range storage device 16, an image signal estimation device 17, and a corrected image storage device 18.

In the above configuration, image information obtained by capturing an image of a subject with the single-panel digital camera 4 is represented by C
After being output from the CD 3, it is converted into digital image information by the A / D converter 5 and stored in the image memory 7.
The image information stored in the image memory 7 is transmitted as an image signal 8 to the image information storage device 10 in the image processing device 19 and stored. Here, the image information stored in the image memory 7 is stored in the image information storage device 10 as image information in which each pixel has only color information corresponding to the filter arrangement of FIG.

At this time, information on the arrangement of the color filters 2 stored in the ROM 6 inside the single-plate digital camera 4 is also transmitted to and stored in the imaging device information storage device 9 in the image processing device 19.

The image processing device 19 converts image information having only color information corresponding to any one of the color filters 2 for the pixels R, G, and B captured by the single-panel digital camera 4 into information in the image. Then, the image information is corrected to image information having R, G, and B color information for all pixels by estimating only from information on the imaging device. The estimation of the color information is performed in the image information storage device 1 in the order of G, B, and R colors.
The processing is sequentially performed on the image information stored in “0”.

That is, the estimated position determining device 11
The position of the estimated pixel corresponding to the estimated color information of the image read from M6 and stored in the imaging device information storage device 9 is obtained. The estimated position is obtained from the information on the filter arrangement stored in the imaging device information storage device 9 at the position of the pixel where the estimated color information is missing. For example, if the estimated color information is G, the estimated position is the position of the pixel where the R or B color filter is arranged. The estimated positions obtained in the image for each estimated color information are all stored in the estimated position determining device 11. The region extraction device 12 obtains a region for calculating an autocorrelation function described later and stores it as a region extraction image in a region extraction storage device described later.

The region extraction by the region extracting device 12 is performed as follows. That is, in FIG. 3, the region extraction calculation device 25 calculates the estimated pixel position information 24 determined by the estimated position determination device 11 and the correlation range shape information 23 stored in the correlation range storage device 13 in advance. A predetermined region extraction image is extracted from the image information 20 stored in the image information storage device 10 and stored in the region extraction storage device 26.

The region extracted image at this time becomes image information corresponding to the correlation range shape of the image information stored in the image information storage device 10 centering on the estimated position. For pixels having no estimated color information in the region extraction image of the image information stored in the image information storage device 10, before extracting the region, the pixel information of the neighboring region having the estimated color information is used in advance. Estimated color information is obtained by an interpolation process such as simple linear interpolation, and an image is obtained in which all pixels in the region extraction image have the estimated color information. That is, in FIG. 3, in the image information 20 stored in the image information storage device 10, a pixel of a portion where a G component is missing is interpolated by an interpolation pixel (G ^* ) 22 from a neighboring pixel having G color information. Image 21 obtained by
In FIG. 3, if the coordinates of the estimated pixel position are (i, j) and the correlation range shape is a 3 × 3 rectangular area, the area extracted image is
Are stored in the region extraction storage device 26 as the image of the region shown in FIG.

The spatial correlation information calculator 14 shown in FIG.
Calculates the autocorrelation function of the region extraction image obtained by the region extraction device 12, and performs least square approximation to a function form stored in advance and outputs the function. That is, in FIG. 4, the autocorrelation function of the region extraction image obtained by the region extraction device 12 is calculated by the autocorrelation function calculation device 28 and is input to the least square coefficient calculation device 27. Further, the function form storage device 27 stores a function form that can approximate an autocorrelation function of a statistical image such as an exponential function represented by the following equation.

[0025]

(Equation 1)

Here, i and j are spatial coordinate values of the image, α,
β is a constant. The least-squares-coefficient operation device 27 calculates the coefficient α in the equation (1) such that the square error between the autocorrelation function calculated by the autocorrelation function operation device 28 and the function stored in the function storage device 27 is minimized. , Β. The correlation value calculation device 30 obtains the value of the correlation function from Expression (1) using the coefficients α and β calculated by the least square coefficient calculation device 27, and sequentially outputs the values.

The estimation coefficient calculation unit 15 in FIG. 1 calculates the coefficient C _ij of the image information estimation equation (2) based on the autoregressive model using the correlation coefficient value calculated by the spatial correlation information calculation unit 14. I do. This autoregressive model is
This is a model in which the estimated image information is represented by a linear sum of the neighboring image information, and the estimated value g (m, n) of the correct image information f (m, n) at the spatial coordinates (m, n) is

[0028]

(Equation 2) It is expressed as Here, D is the range (neighboring area) of the spatial coordinates of the image information f (m, n) used for estimating g (m, n), and c _ij is a coordinate value relatively from the spatial coordinates of the estimated image information. Image information f located at i and j in the mn direction, respectively
The estimation coefficients of (mi, nj) are shown. At this time, the square error Err between the estimated value g (m, n) and the correct image f (m, n) is

[0029]

(Equation 3) It is expressed as Here, A corresponds to the range of the region extraction image. The estimation coefficient c _ij for minimizing the square error Err is approximately obtained by partially differentiating Err with c _ij ,

[0030]

(Equation 4)

Here, R (ip, jp), R (p,
Since q) is a known value calculated by the spatial correlation information calculation device 14, equation (4) is a simultaneous equation in which the estimation coefficient c _ij is an unknown. The estimation coefficient calculation device 15 calculates the estimation coefficient c _ij from the relationship of the expression (4) using the information of the neighboring area D stored in the estimation coefficient range storage device 16.

The image signal estimating device 17 in FIG. 1 uses the estimated coefficient c _ij calculated by the estimated coefficient calculating device 15 to obtain image information f of the autoregressive model represented by the equation (2).
An estimated value g (m, n) of (m, n) is calculated.

The corrected image storage device 18 in FIG. 1 sequentially stores the estimated image information of each estimated position of the estimated color information sequentially calculated by the image signal estimating device 17. The processing from the determination of the region extraction image by the region extraction device 12 to the storage of the estimated image information in the corrected image storage device 18 is defined as G,
The color information of B and R is performed at all estimated positions, and all the pixels are stored in the corrected image storage device 18 as image information having RGB color information. The configuration of the first embodiment described above can be variously modified and changed. For example, the filters of a digital camera may be arranged in stripes,
A CMY complementary color system may be used in addition to the primary color system spectral characteristics. Further, the present invention can be applied to a filter having an arbitrary number of filters instead of three filters, such as RGB.

FIG. 5 is a diagram showing a configuration of an image processing system to which the second embodiment of the present invention is applied. In FIG. 5, the region extraction image extracted by the region extraction device 12 in the same manner as in the first embodiment is transferred from the region extraction storage device 26 in the region extraction device 12 to the structure information extraction device 3 in FIG.
Read to 1. The structure information extraction device 31 calculates the edge strength I of the region extraction image. The edge strength I is defined by the following equation, assuming that the region extraction image is a rectangular image of 3 × 3 pixels.

[0035]

(Equation 5)

Here, a to i represent the pixel values at the pixel positions of the region extraction image shown in FIG. 6, and X and Y correspond to the differential values in the x and y directions, respectively. The edge intensity I is normalized to 0 to 1 according to the density range of the image to be used.
The edge strength I can be calculated by appropriately redefining the edge strength I for an area extracted image having an arbitrary shape.

The spatial correlation information calculator 14 shown in FIG. 5 has a configuration as shown in FIG. Similarly to the first embodiment, the coefficients α and β of the function form of the correlation function are calculated by the least square coefficient calculating device 29. The least-squares coefficient correction device 32 calculates the coefficients α and β by using the edge strength I of the extraction area calculated by the structure information extraction device 31 and the correction coefficient storage device 33.
Are corrected to α 'and β' by the following equation using the correction coefficients P and Q stored in advance.

Α ′ = α (PI + 1) β ′ = β (QI + 1) (7) By the correction of equation (7), α and β become I = 1 at the edge portion, and are increased by P and Q%, respectively. A correlation value is calculated by the correlation value calculation device 30 using the corrected coefficients α ′ and β ′. Thereafter, estimated image information is obtained by the same processing as in the first embodiment, and is stored in the corrected image storage device 18.

The configuration of the second embodiment described above can be variously modified and changed. For example, it is also possible to calculate not only the edge strength but also the direction of the edge using the values of X and Y to correct the correlation function. Further, it is also possible to select a method of calculating the correlation function according to the strength, direction, and the like of the edge.

FIG. 8 is a diagram showing the configuration of an image processing system to which the third embodiment of the present invention is applied. In FIG. 8, the structure information extracting device 31 calculates the edge strength I of the region extracted image extracted by the region extracting device 12 by the same method as in the above-described second embodiment. The estimation method switching device 33 selects a method of estimating the image information at the estimation position from the value of the edge strength I calculated by the structure information calculation device 31. That is, when I> T, the first estimation method is selected as the estimation method. When I ≦ T, the second estimation method is selected. T is an experimentally predetermined constant.

The first estimating method is a method of estimating image information using the autocorrelation function information of a region extraction image based on the same autoregressive model as in the first embodiment. In the second estimation method, the simple linear interpolation device 35 estimates pixel information of an interpolation position as a linear sum of pixel information near a predetermined interpolation position. The estimated pixel information g (m, n) of the estimated position is 3 ×
The average value of the image information f (m, n) in the area No. 3 is calculated by the following equation.

[0042]

(Equation 6)

Here, E represents a 3 × 3 rectangular area other than (m, n) centered at (m, n). The estimated image information calculated by the simple linear interpolation device 35 according to the equation (8) is stored in the corrected image storage device 18.

The configuration of the third embodiment described above can be variously modified and changed. For example, the method of estimating the second image signal is not limited to a method based on linear interpolation, and any method can be used according to the purpose. The number of estimation methods that can be switched is not limited to two, and by preparing a plurality of estimation methods, a plurality of switching methods can be selected. Further, the switching method is not limited to the method based on the structural analysis, and can be manually set in the processing device.

FIG. 9 is a diagram showing the configuration of an image processing system to which the fourth embodiment of the present invention is applied. In FIG. 9, a region extraction image extracted by the region extraction device 12 by the same method as in the first embodiment is analyzed by the structure information extraction device 31.

FIG. 10 is a diagram showing the structure of the structure information extracting device 31. In FIG. 10, dozens of types of pattern images having the same size as the region extraction image are stored in the pattern storage device 38 in advance. The pattern matching calculation device 39 uses this pattern image to perform pattern matching with the region extraction image from the region extraction storage device 26 in the region extraction device 12. And the pattern matching calculation device 3
The similarity between the correlation area image and the pattern image calculated in 9 is calculated as an evaluation value by the similarity evaluation calculation device 40.

The memory card 36 in FIG. 9 is provided with estimation coefficients as table data corresponding to the types of pattern images. The estimation coefficient stored in the memory card 36 is a value calculated from an autocorrelation function calculated by a process similar to that of the above-described first embodiment with respect to the pattern image stored in the structure information extracting device 31 in advance. .

The estimated coefficient selecting device 37 reads out estimated coefficients corresponding to several types of pattern images similar to the region extracted image analyzed by the structure information extracting device 31 from the table data in the memory card 36, and compares the read estimated coefficients with similarity. The calculation is performed using the evaluation value calculated by the degree evaluation calculation device 40. The image signal estimating device 17 obtains estimated image information by using the estimated coefficients calculated by the estimated coefficient selecting device 37 in the same manner as in the first embodiment, and stores the estimated image information in the corrected image storage device 18.

The configuration of the above-described fourth embodiment can be variously modified and changed. For example, the memory card 36
The table data in the table may not give the estimation coefficients corresponding to the pattern of the region extraction image, but may give the estimation coefficients corresponding to the parameters of the function form of the correlation function. Further, the memory card 36 can be replaced according to the type of the image to be captured, or can store a plurality of different table data and select it according to the type of the image to be captured.

Hereinafter, a fifth embodiment of the present invention will be described.
Estimated position determination device 11 in the first embodiment described above
In the fifth embodiment, not only one pixel but also a 5 × 5 rectangular estimation area is determined. Determination of the estimation area
The processing is performed for the information of each of the RGB colors for the entire image so that the pixels included in the estimation area do not overlap.

The area extracting device 12 extracts a 10 × 10 rectangular area centered on the estimated area from the image information as an area extracted image. The spatial correlation information calculating device 14 and the estimation coefficient calculating device 15 perform the same processing as in the first embodiment, and the image signal estimating device 17 uses an internal control device for only pixels in which the estimated color information is missing in the estimated area. The image information is estimated using the estimation coefficients calculated by the estimation coefficient calculation device 15 and stored in the corrected image storage device 18.

The fifth embodiment can be variously modified and changed. For example, the estimation area is not a predetermined area, but an area in which the estimation coefficient can be considered to be constant by the same structure information extraction device 31 as in the second embodiment can be used as the estimation area. In this case, the region extraction image extracted by the region extraction device 12 is analyzed by the structure information extraction device 31, and the estimated position determination device 11 determines an estimated region. Subsequent processing is the same as in the above-described fifth embodiment.

FIG. 11 is a diagram showing the configuration of an image processing system to which the sixth embodiment of the present invention is applied. In FIG. 11, the structure information extracting device 31 creates a differential image of the region extracted image extracted by the region extracting device 12 by the same method as in the first embodiment described above. The correlation region extraction device 41 estimates a region where the value of the autocorrelation function can be regarded as constant, and newly extracts the region as a correlation region image. The region extraction image extracted by the region extraction device 12 is a predetermined rectangular region image. The correlation region extraction device 41 creates a differential image of the region extraction image extracted by the region extraction device 12, and estimates a region where the autocorrelation function can be regarded as constant from the similarity of the distribution of the differential image density. A rectangular area that includes the set of estimated positions is newly set as a correlation area image. Thereafter, image information is estimated using the correlation area image by the same method as in the first embodiment, and stored in the corrected image storage device 18.

The configuration of the sixth embodiment can be variously modified and changed. For example, the method of creating the correlation area image is not limited to the method based on the differential image, but it is also possible to actually calculate the correlation function of the extraction area image and determine the correlation area from the similarity.

FIG. 12 is a diagram showing the configuration of an image processing system to which the seventh embodiment of the present invention is applied. In FIG. 12, the estimated position selecting device 100 includes a center of an estimated region obtained by dividing an image into a predetermined rectangular region among the estimated positions determined by the estimated position determining device 11 in the same manner as in the first embodiment. Select only the position closest to the point. The function of the autocorrelation function at the selected estimated position selected by the estimated position selecting device 100 is calculated by the spatial correlation information calculating device 14. The above processing is performed for the entire image, and the coefficients of the function form of the correlation function at all the selected estimated positions are obtained and stored in the correlation coefficient storage device 42.

The spatial correlation information interpolating device 43 estimates the coefficients of the function form of the correlation function at all positions by interpolation using the stored coefficients of the function form of the correlation function at the selected estimated position. The interpolation calculation is performed on the basis of using correlation coefficients at three selected positions surrounding the estimated position. Image information is estimated for each estimated position at which the function-type coefficient of the correlation function has been obtained by the same method as in the first embodiment, and stored in the corrected image storage device 18.

The configuration of the seventh embodiment described above can be variously modified and changed. For example, the estimated position selecting device 100 does not select the selected estimated position from the predetermined rectangular area, but can assume that the correlation function is constant from the image analyzed by the structure information extracting device 31 similar to the second embodiment. It is also possible to estimate and determine the area. Also, instead of interpolating the coefficients of the function form of the correlation function at the selected estimated position from the coefficients of the function form of the correlation function, it is also possible to interpolate the estimated coefficients of the other positions from the estimated coefficients at the selected estimated position. It is possible.

FIG. 13 is a diagram showing the configuration of an image processing system to which the eighth embodiment of the present invention is applied. 13, an image processing device 65 of the present invention includes image information captured by a single-plate digital camera 44 having R, G, and B color filters arranged as shown in FIG.
A luminance signal is generated from the filter arrangement information of the single-panel digital camera 44 stored therein.

Image information g (m, n) captured by the single-panel digital camera 44 is transmitted to an image memory 47 in an image processing device 65 via a color image input terminal 46.
It is memorized. At the same time that the image information g (m, n) is stored in the image memory 47, the filter arrangement information of the single-panel digital camera 44 stored in advance in the computer 45 includes the color component mask H _c (m, n). (C = r, g, b)
The information is transmitted to and stored in the color component mask buffer 55 via the mask information input terminal 54 as related information.

The color component mask buffer 55 includes an R component mask buffer 55a, a G component mask buffer 55b, and a B component mask buffer 55c, and stores mask information of the corresponding color components. The color component mask H _c (m, n) has a value of 1 when the color filter at the pixel position (m, n) of the single-panel digital camera 44 is the color component c of the mask, and a value of 0 when the other color components are used. Have.

FIG. 14 shows an example of the color component masks H _r (m, n) and H _g (m, n) corresponding to the filter arrangement of FIG. At the same time as transmitting the mask information, the computer 4
5 stored in the luminance signal calculation coefficient β _c (c = r,
g, b) are transmitted and stored in the luminance weight buffer 56 in the image processing device 65 via the mask information input terminal 54. Luminance Y (m, n) is the luminance signal operation coefficient beta _c,
Image information f having R, G, B full color information at each pixel
_{Using c} (m, n) (c = r, g, b),

[0062]

(Equation 7) It becomes what is represented.

After the information transmission to the color component mask buffer 55 is completed, the block I / O section 48 controls the partial block image g _c (m, n) under the control of the control device 52 using the information of the color component mask buffer 55. ) (C = r, g, b) is written into the block buffer 49 from the image information g (mn) in the image memory 47. The block buffer 49 includes an R component block buffer 49a and a G component block buffer 49.
b and B component block buffers 49c. Each color component block buffer has g (m, n) other than pixels having color information corresponding to the color components of each color component block buffer.
Is stored as a color component block image g _c (m, n). The image information g (m, n) in the image memory is obtained by using a color component mask H _c (m, n) and a color component image f _c (m, n).

[0064]

(Equation 8) It is expressed as

By the above processing, information transmission from the single-panel digital camera 44 and the computer 45 to the image processing device 65 is completed. Thereafter, under the control of the control device 52, the luminance signal estimation processing is sequentially performed for each pixel. Done. The luminance signal Y (m, n) is estimated by the following expression obtained by expanding the AR model.

[0066]

(Equation 9)

Here, D is the range (neighborhood) of the spatial coordinates of the image information g (m, n) in the image memory used for estimating Y (m, n), and _cij is the spatial coordinate of the estimated image information. Represents the estimated coefficient of image information g (mi, nj) whose coordinate values are located at i and j in the m and n directions, respectively. At this time, the square error Err in the predetermined area A between the estimated value obtained by the equation (11) and the correct luminance component Y (m, n).
Is

[0068]

(Equation 10) Get.

From equation (14), c _kl is approximately obtained,
The luminance signal Y (m, n) is estimated from the equation (11). First, the control device 52 determines the position of an estimated pixel for luminance estimation.

[0070]

[Equation 11] Is calculated and stored. For example, a single-panel digital camera 4
4 R component mask H _r (m, n) and G component mask H _g
(M, n) if the rectangular region of 5 × 5 pixels as shown in FIG. 14, RG colors cross-correlation R ^h _rg (i, j) is as shown in FIG. 15, S _pq is, S _pq = ｛(-2, -1,4),
(-2,1,4), (-1, -2,4), (-1,0,
6), (-1, 2, 4), (0, -1, 6), (0,
(1,6), (1, -2,4), (1,0,6), (1,
2,4), (2, -1,4), (2,1,4)}. Further, the mask correlation calculation device 57 stores the number of values of each color component mask H _c (m, n) being 1, that is, the number α _c of pixels having a corresponding color filter. For example, FIG.
The R component mask H _r (m, n) and the G component mask H shown in FIG.
_{In the case of g} (m, n), α _r is 9 and α _g is 12.

The inter-color correlation calculation device 50 includes a partial inter-color correlation calculation unit 51, a correlation value interpolation unit 53, and a correlation function parameter storage unit 52. The partial color correlation calculating section 51 reads a partial image of the predetermined area A of each color component from each color component block buffer 49a, 49b, 49c under the control of the control device 52, and combines R, G, B without overlapping.

[0072]

(Equation 12) , A cross-correlation R _pq ′ (i, j) between partial images defined by the following equation is calculated.

[0073]

(Equation 13)

Here, g _p (m, n) and g _q (m−i,
n−j) has a lack of information at pixels that do not correspond to the color filter, and therefore, in general, a correlation value R _pq (i, j) defined by the following equation:

[0075]

[Equation 14] Does not match. Where f _p (m, n) and f _q (m−
i, n-j) are color information p, q for all pixels, respectively.
Represents image information having Therefore, the partial color correlation calculating section 51 calculates the set S calculated by the mask correlation calculating device 57.
corrected R _pq '(i, j) by using _pq, R _pq
An estimated value of (i, j) is calculated. Since the coordinates (i, j) of the points included in the set S _pq are coordinate values for which R ^h _pq (i, j) is not zero, the non-zero term g _p (m ' , N ′) g _q (m′−i, n′−j) whose value is the value of the term f _p (m ′, n ′) f _q (m '-I, n'-
j). However, such a term is limited to a term corresponding to (i, j) of the set S _pq with _respect to the total number of pixels of the block.

For example, in the case of the example shown in FIG. 15, (17)
The number of terms for which correct values can be obtained for the 25 terms in the equation is a maximum of 6
Term, and the other terms are zero. Therefore, the value R _pq ′ (i, j) obtained by the equation (17) and the correct correlation value R
The ratio between the mask correlation value R ^h _pq (i, j) and the correlation value R ₀ ^h _pq (i, j) between the masks whose ratio with _pq (i, j) is approximately 1 in the entire region. And R _pq (i,
estimate R of _{j) "pq (i, j} ) determined by the following _{formula. R" pq (i, j} ) = R 'pq (i, j) R 0 h pq (i, j) / R h pq ( i, j) (19) Also, using the image information g (m, n) in the image memory 47,

[0077]

(Equation 15) Is calculated and stored internally.

The correlation value interpolating unit 53 includes a partial color correlation calculating unit 5
The correlation function R ″ _pq (i, j) calculated and corrected in 1
And outputs as an approximate correlation function. R "
_{Since pq} (i, j) is an autocorrelation between images with missing pixel information, it has data only at discrete positions as shown in FIG. The correlation value interpolation unit 53 performs a least square approximation of the correlation function R ″ _pq (i, j) to a known function form using the approximate function form stored in the correlation function parameter storage unit 52 and the parameters. The obtained R ″ _pq (i, j) is stored in the correlation value interpolation unit 53.

The correction coefficient calculating device 64 includes the correlation value adding section 5
8, a correlation value buffer 59, and a correction coefficient calculator 60. The correlation value adder 58 calculates α _c calculated by the mask correlation calculator 57, the luminance signal operation coefficient β _c obtained from the luminance weight buffer 56, and the approximated R ″ _pq (obtained from the correlation value interpolator 53). The following equation is calculated and stored using (i, j).

[0080]

(Equation 16)

The correlation value buffer 59 stores the value of the expression (20) calculated by the partial color correlation calculation unit 51 and the value of the expression (21) calculated by the correlation value addition unit 58.
The correction coefficient calculating unit 60 calculates c _kl that satisfies the expression (14) using the values of the expressions (20) and (21) stored in the correlation value buffer 59 and stores it. Here, on the left side of the equation (14), an approximation of the following equation:

[0082]

[Equation 17] Was used.

The luminance component restoring device 61 calculates the image information f (m, n) stored in the image memory 47 and the correction coefficient calculator 60
The luminance component Y (m, n) of the estimated pixel is calculated from the equation (11) using the correction coefficient c _kl stored in the output image memory 62 and stored in the output image memory 62.

The above processing is performed for all the pixels. The corrected image can be read out to the outside via the luminance image output terminal 63. Note that the above-described embodiment can be variously modified and changed. For example, the estimation pixel is performed for each pixel. However, when it is determined that the approximation accuracy is good, it is also possible to obtain the luminance components of all the pixels of the block. As a method of calculating the correlation between individual colors, various correlation estimation methods can be used, and it is also possible to use foresight information. Although the luminance signal is estimated here, it is possible to restore not only the luminance but also any image information calculated by the linear sum of the color signals. In this example, the correction of an image captured by a single-plate mosaic color filter CCD has been described.

[0085]

(Equation 18) Accordingly, the present invention can be applied to, for example, correction of an image including chromatic aberration. Further, the cross-correlation between colors may be obtained by applying an appropriate function form based on information obtained from an observed image. If the statistical properties of the object are known in advance, it is effective to use such information.

The specific embodiments described above include inventions having the following configurations. Embodiments and effects corresponding to the respective inventions are as follows. (1) Estimated position determining means for determining a position of an image signal to be estimated based on information on an imaging device that generates an image signal, and a predetermined size including the position of the image signal obtained from the estimated position determining means Area extracting means for extracting a local area of the area, spatial correlation information calculating means for calculating spatial correlation information on the local area obtained from the area extracting means, and spatial spatial information calculated by the spatial correlation information calculating means. An image signal estimating means for estimating an image signal corresponding to the position of the image signal obtained from the estimated position determining means based on the correlation information. (2) The image processing apparatus according to configuration (1), wherein the estimated position determining unit determines a position of an image signal to be estimated based on arrangement information of a color filter of the imaging device. (3) The image processing device according to the configuration (1) or (2), wherein the region extracting unit extracts a block region of n × m pixels where n and m are integers of 2 or more. (4) The configuration according to any one of (1) to (3), wherein the spatial correlation information calculation unit calculates the spatial correlation information based on an autocorrelation function of the local region. Image processing device. (5) The configuration wherein the spatial correlation information calculating means calculates the spatial correlation information based on approximating an autocorrelation function of the local region with an exponential function.
The image processing device according to any one of (1) to (4). (6) The image signal estimating means includes an estimation coefficient calculating means for calculating an estimation coefficient for a local area obtained from the area extracting means based on the spatial correlation information calculated from the spatial correlation information calculating means. The image processing apparatus according to any one of the configurations (1) to (5), wherein the estimation coefficient calculation unit calculates the estimation coefficient of the local region based on an autoregressive model. (7) Structural information extracting means for extracting structural information on a local region obtained from the area extracting means, and spatial correlation by the spatial correlation information calculating means based on the structural information extracted by the structural information extracting means. The image processing apparatus according to configuration (1), further comprising: changing means for changing a method of calculating information. (8) The image processing apparatus according to configuration (7), wherein the structure information extracting unit extracts the structure information based on brightness information. (9) The image processing apparatus according to configuration (7), wherein the structure information extracting unit extracts the structure information based on a differentiation process and a pattern match. (10) Estimated position determining means for determining the position of the image signal to be estimated based on information on the imaging device that generates the image signal, and a local area of a predetermined size including the position obtained from the estimated position determining means. A region extracting means for extracting, a structure information extracting means for extracting structural information on a local region obtained from the region extracting means, and an estimation coefficient calculated from spatial correlation information on the local region obtained from the region extracting means. A first image signal estimating means for estimating an image signal corresponding to the position of the image signal obtained from the estimated position determining means, and a local area obtained from the area extracting means, The image signal corresponding to the position of the image signal is obtained by using another image signal corresponding to the position of the obtained image signal or an image signal located around the position. Second image signal estimating means for estimating the first image signal and the switching means for switching between the first image signal estimating means and the second image signal estimating means based on the structure information extracted by the structure information extracting means. An image processing apparatus characterized by the above-mentioned. (11) The second image signal estimating means is configured to: linearly interpolate, in a local area obtained from the area extracting means, an image signal at a position around the position of the image signal obtained from the estimated position determining means; The image processing apparatus according to configuration (10), wherein the estimation is performed by interpolation using a second-order or higher-order function. (12) The second image signal estimating means, in the local area obtained from the area extracting means, is correlated with another image signal corresponding to the position of the image signal obtained from the estimated position determining means, The image processing device according to configuration (10), wherein the image signal is estimated. (13) Estimated position determining means for determining the position of the image signal to be estimated based on information on the imaging device that generates the image signal, and a predetermined size including the position of the image signal obtained from the estimated position determining means Region extraction means for extracting a local region of the above, structure information extraction means for extracting structure information relating to the local region obtained from the region extraction device, and estimation in which estimation coefficients corresponding to a plurality of predetermined structure information are stored An image processing apparatus comprising: a coefficient storage unit; and an estimation coefficient selection unit that selects an estimation coefficient from the estimation coefficient storage unit based on the structure information extracted from the structure information extraction unit. (14) The image processing apparatus according to the configuration (13), wherein the estimation coefficient storage unit is formed of a replaceable recording medium such as a magnetic disk and a memory card. (15) A configuration further comprising control means for sharing and using the spatial correlation information calculated by the spatial correlation information calculating means at a plurality of positions obtained from the estimated position determining means ( Any one of 1) to (6)
The image processing device according to any one of the above. (16) Structural information extracting means for extracting structural information on the local area obtained from the area extracting means, and a local area obtained from the area extracting means based on the structural information extracted by the structural information extracting means. The image processing apparatus according to any one of configurations (1) to (6), further comprising: a second area extracting unit configured to extract a new local area from the image data. (17) The image processing apparatus according to the configuration (16), wherein the second region extracting unit extracts a region where the autocorrelation function can be regarded as approximately constant. (18) Estimated position determining means for determining the position of the image signal to be estimated based on information about the imaging device that generates the image signal, and the positions of the plurality of image signals obtained from the estimated position determining means are set at predetermined intervals. Estimated position re-extracting means for sparsely re-extracting the above, area extracting means for extracting a local area of a predetermined size including the position of the image signal obtained from the estimated position re-extracting means, and Spatial correlation information calculating means for calculating spatial correlation information of the local area corresponding to the position of the image signal re-extracted by the estimated position re-extracting means with respect to the obtained local area; A spatial phase interpolating spatial correlation information relating to the positions of all the image signals obtained from the estimated position determining means based on the spatial correlation information calculated from the information calculating means; Function information interpolating means, and image signal estimating means for estimating an image signal corresponding to the position of the image signal obtained from the estimated position determining means based on the spatial correlation information interpolated by the spatial correlation information interpolating means. An image processing apparatus comprising: (19) Estimated position determining means for determining the position of an image signal to be estimated based on information on an imaging device that generates an image signal composed of a plurality of color signals, and the image signal of the image signal obtained from the estimated position determining means. Region extracting means for extracting a local area of a predetermined size including a position from an image signal including a plurality of color signals, and spatial correlation information calculating means for calculating spatial correlation information on the local area obtained from the area extracting means A color correlation information calculating means for calculating color correlation information on a local area obtained from the area extracting means; a spatial correlation information calculated by the spatial correlation information calculating means; and a color correlation information calculated from the color correlation information calculating means. Image signal estimating means for estimating an image signal corresponding to the position of the image signal obtained from the estimated position determining means based on the obtained color correlation information. An image processing apparatus comprising: (20) A configuration characterized in that the color correlation information calculation means includes a foreseeable color correlation information holding means for holding foreseeable information on a spatial correlation of a plurality of color signals related to the local area of the predetermined size. An image processing apparatus according to 19). (21) The configuration wherein the color correlation information calculation means includes function approximation means for approximating spatial correlation information of a plurality of color signals relating to the local region of the predetermined size to a predetermined function represented by parameters. An image processing apparatus according to 19). (22) The image processing apparatus according to configuration (19), wherein the area extracting unit extracts an n × m block area where n and m are integers of 2 or more. (Embodiments of the Invention Corresponding to Configurations (1) to (6)) These inventions correspond to the first embodiment described above. That is, the imaging device in the configuration is the single-panel digital camera 4 of FIG.
In addition, the estimated position determining means is the estimated position determining device 11 of FIG.
The region extracting means is provided in the region extracting device 12 of FIG. 1, and the spatial correlation information calculating means is provided in the spatial correlation information calculating device 14 of FIG.
In addition, the estimation coefficient calculation means is the estimation coefficient calculation device 15 of FIG.
The image signal estimating means corresponds to the image signal estimating device 17 in FIG. Also, the arrangement information of the color filters of the imaging device corresponds to the color filter arrangement information on the CCD of FIG. 2 stored in the imaging device information storage device 9, and the block area of n × m pixels becomes a 3 × 3 block area. Corresponding.

In FIG. 1, image information captured by the single-panel digital camera 4 and stored in the internal image memory 7 is transmitted and stored in the image information storage device 10 in the image processing device 19. At the same time, the C of the single-panel digital camera 4 stored in the ROM 6 inside the single-panel digital camera 4
The arrangement information of the color filters 2 arranged on the CD 3 is transmitted and stored in the imaging device information storage device 9. From the filter arrangement information stored in the imaging device information storage device 9, all pixel positions of the image information in the image information storage device 10 corresponding to the color information to be estimated by the estimated position determination device 11 are calculated and stored. With respect to the stored position of one pixel, an area extraction image is extracted from the image information storage device 10 by the region extraction device 12 according to the region stored in the correlation range storage device 13. The spatial correlation information calculation device 14 calculates correlation information regarding the region extraction image, and the estimation coefficient calculation device 15 calculates the estimation coefficient of the estimation formula for estimating the image signal based on the model using the correlation information. The image signal estimating device 17 calculates an estimated value of the image signal using the estimation coefficient, and stores the estimated value in the corrected image storage device 18. Estimated position determination device 11
By performing the processing for all the estimated positions stored therein, each pixel in the entire image is stored in the corrected image storage device 18 as an image having RGB color information. (Effects of Configurations (1) to (6)) Since adaptive estimation depending on an image structure is performed using spatial correlation information on a local region of an image signal to be estimated, an image having an arbitrary structure can be obtained. To make an optimal estimation. For example, by using the autocorrelation information of the region extraction image near the estimated position and performing image information estimation by an autoregressive model, it is possible to automatically calculate an estimation coefficient that minimizes the estimation error, Thereby, image processing depending on an arbitrary image structure can be performed. (Embodiments of the Invention Corresponding to Configurations (7) to (9)) The second embodiment described above corresponds to this invention. The structural information extracting means in the configuration corresponds to the structural information extracting device 31 in FIG. 5, and the changing means corresponds to the least square coefficient correcting device 32 (FIG. 7) in the spatial correlation information calculating device 14. According to the present invention, the spatial correlation information calculating device 1 according to the image structure near the estimated position obtained by the analysis of the structural information extracting device 31
By correcting the correlation coefficient calculated in step 4, more accurate spatial correlation information is calculated. The structure information extraction device 31 analyzes the edge strength of the region extraction image extracted by the region extraction device 12. Spatial correlation information calculation device 1
4 corrects the correlation coefficient calculated by the least squares coefficient calculator 29 (FIG. 7) according to the edge strength of the region extraction image. In the correction, the coefficient increases as the edge strength increases. Hereinafter, image information is estimated and stored by the same processing as in the first embodiment. (Effects of Configurations (7) to (9)) The shape of the correlation function calculated by the spatial correlation information calculation device 14 cannot always be obtained accurately due to interpolation processing of the region extraction image or approximation of the function form. . Therefore, by correcting the shape of the correlation function in an edge region that is particularly difficult to estimate so as to obtain a good result empirically, more accurate estimation can be performed. (Embodiments of the Invention Corresponding to Configurations (10) to (12)) The third embodiment described above corresponds to this invention. The first image signal estimating means in the configuration is a means for estimating image information using the autocorrelation function information of the region extraction image based on the autoregressive model described above, and the second image signal estimating means is a simple linear interpolation device 35 The switching means corresponds to the means for estimating the pixel information of the interpolation position as a linear sum of the pixel information in the vicinity of the interpolation position determined in advance, and the switching means corresponds to the estimation method switching device 33 of FIG.

In the present invention, simple linear interpolation in a flat area and a method based on an autoregressive model in an edge area are selectively used in accordance with the image structure near the estimated position obtained by the analysis of the structure information extracting device 31. . Structure information extraction device 31
Analyzes the structure of the region extraction image extracted by the region extraction device 12 and calculates parameters used by the estimation method switching device 33 to select an estimation method. The estimation method switching device 33 uses the first image signal estimation means when the edge strength of the region extraction image is larger than a predetermined value from the result of the structural information analysis, and uses the second image signal estimation means when the edge strength is smaller than the predetermined value. Estimate the signal. (Effects of Configurations (10) to (12)) By switching the image signal estimation method according to the structure of the region extraction image, it is possible to perform image signal estimation with a small amount of calculation while maintaining estimation accuracy. For example, in an area where the image structure is easy to estimate, the estimation is performed by the simple linear interpolation device 35 having a relatively small amount of calculation, instead of using the estimation method based on the autoregressive model having a large amount of calculation. Further, by incorporating the information obtained by the structural information analysis device 31 into the processing correction of the first image signal estimation method, or by using the information as the second image signal estimation method, more accurate estimation becomes possible. (Embodiments of the Invention Corresponding to the Configurations (13) and (14)) The fourth embodiment described above corresponds to this invention. The estimated coefficient storage means in the configuration is stored in the memory card 36 of FIG.
The estimation coefficient selection means corresponds to the estimation coefficient selection device 37, respectively. In the present invention, in the first embodiment described above, the calculation of the estimation coefficient having a large amount of calculation is obtained in advance for a plurality of predetermined patterns, and a table of the image structure near the estimation position and the estimation coefficient is created and stored. The calculation is speeded up by using this table information when estimating the image information. The structure information extraction device 31 of FIG. 9 performs a pattern matching operation between the region extraction image and a predetermined pattern image stored in the pattern storage device 38, and evaluates the similarity. Then, an estimation coefficient corresponding to a pattern having a high degree of similarity is obtained from table data stored in the memory card 36, and the estimation coefficient is calculated by appropriate weighting based on the evaluation value of the degree of similarity. The image signal is estimated by the image signal estimating device 17 using the obtained estimation coefficients, and is stored in the corrected image storage device 18. (Effects of Configurations (13) and (14)) By storing the estimation coefficients corresponding to a plurality of predetermined structural information, the estimation coefficients can be calculated without performing the calculation for calculating the estimation coefficients from the region extraction image. Can be requested. In this way, by calculating a calculation part having a large amount of calculation only for a predetermined image in advance and storing it as table data, the amount of calculation can be reduced using the table data in actual processing. (Embodiments of the Invention Corresponding to Configurations (15) to (17)) The sixth embodiment described above corresponds to this invention.
The second region extracting means in the configuration corresponds to the correlation region extracting device 41 of FIG. According to the present invention, by estimating a region near the estimated position where the AR model is well established and determining the correlation region, highly accurate image information is efficiently estimated. FIG.
The structure information extracting device 31 creates a differential image of the region extracted image, and the correlation region extracting device 41 estimates a region where the differential value is almost constant as a region where the correlation value is also constant and sets it as a correlation region.
The autocorrelation function of the correlation area is used to calculate the spatial correlation information.
Then, the image signal is estimated and stored by the same processing as in the first embodiment. (Effects of Configurations (15) to (17)) Correlation region extraction device 4
By calculating the estimation coefficient in a region where the correlation can be considered to be constant, more accurate estimation can be performed. In addition, the calculation amount can be reduced by setting an appropriate correlation area in the calculation process of the estimation coefficient calculation requiring a large amount of calculation. (Embodiment of Invention Corresponding to Configuration (18)) The seventh embodiment described above corresponds to this invention. The estimated position re-extraction unit in the configuration corresponds to the estimated position selection device 100, and the spatial correlation information interpolation unit corresponds to the spatial correlation information interpolation device 43.

According to the present invention, the correlation coefficient of the estimated position is not actually obtained for each pixel as in the first embodiment described above, but the correlation is calculated from the correlation calculation only for an appropriate sample position. The correlation coefficient between the sample positions is obtained by interpolation from the correlation coefficient at the sample position. This is based on the presumption that the shape of the correlation function changes mostly spatially continuously. By calculating the correlation coefficient by interpolation, the amount of calculation can be reduced. The estimated position selecting device 100 selects the closest estimated position among a plurality of predetermined positions of the predetermined image information, and stores the selected estimated position as the selected position. The region extracting means calculates a function-type coefficient of the correlation function by the spatial correlation information calculating device 14 for all selected positions, and stores it in the correlation coefficient storage device 42. The spatial correlation information interpolating device 43 obtains the function-type coefficients of the correlation function at all the estimated positions in the image by interpolation from the function-type coefficients of the correlation function at the estimated positions stored in the correlation coefficient storage device 42. Thereafter, an image signal is obtained by the same processing as in the first embodiment, and is stored in the corrected image storage device 18. (Effect of Configuration (18)) A plurality of positions of the image signal to be estimated are sparsely re-extracted at predetermined intervals or more, and spatial correlation information of a local region corresponding only to the re-extracted positions is calculated.
The spatial correlation information for all positions is interpolated. That is, the coefficient of the function form of the correlation function is estimated by the estimated position selecting device 1.
It is possible to reduce the amount of calculation by obtaining only the selected position selected by 00 and calculating the coefficient of the function form of the correlation function at the other estimated position by interpolation from the coefficient of the function form of the correlation function at the selected position. it can. (Embodiments of the Invention Corresponding to the Configurations (19) to (22)) The eighth embodiment described above corresponds to the present invention.
The correction position determination means in the configuration is provided to the control device 52 of FIG.
The region extracting means corresponds to the control device 52 and the partial color correlation calculating section 51 in FIG. 13, and the color correlation information calculating means corresponds to the color correlation calculating device 50 in FIG. The correction coefficient calculation means corresponds to the correction coefficient calculation device 64 in FIG. The image signal correction means corresponds to the luminance component restoration device 61 in FIG. The look-ahead color correlation information holding unit corresponds to the correlation function parameter storage unit 52 in FIG.

In the present invention, as in the first embodiment, A
The luminance signal at the estimated position is estimated as a linear sum of surrounding image information by the R model. However, the luminance signal is R,
It is assumed to be represented as a known linear sum of the G and B signal values. The estimation coefficient for estimating the luminance signal requires not only the autocorrelation information of each color of RGB but also the cross-correlation information between the colors. Here, the cross-correlation between the colors which is approximately obtained is used.

In FIG. 13, image information captured by a single-panel digital camera 44 having a filter as shown in FIG. 2 is transmitted and stored in an image memory 47 in an image processing device 65 through a color image input terminal 46. Also,
The color component mask buffer 55 and the luminance weight respectively represent the information on the filter arrangement of the single-panel digital camera 44 and the luminance signal operation coefficients for representing the luminance signal by the linear sum of the R, G, and B color signals stored in the computer 45 in advance. The data is transmitted and stored in the buffer 56. 13 controls the block I / O unit 48 using the filter arrangement information stored in the color component mask buffer 55 to convert the partial block image from the image information stored in the image memory 47 into the block buffer 49. To record.

The transmission and storage of information from the single-panel digital camera 44 and the computer 45 are completed as described above. Thereafter, a luminance signal is estimated for each pixel and stored in the output image memory 62. The inter-color correlation calculation device 50 includes an image memory 47.
Cross-correlation necessary for calculating a luminance signal estimation coefficient using the image information in the block, the block image stored in the block buffer 49, and the correlation information of the filter arrangement calculated by the mask correlation calculator 57. Is calculated. The correction coefficient calculator 64 calculates the inter-color cross-correlation obtained by the inter-color correlation calculator 50, the luminance signal estimation coefficient stored in the luminance weight buffer 56, and the correlation information of the filter arrangement obtained by the mask correlation calculator 57. To calculate a correction coefficient. The luminance component restoring device 61 includes a correction coefficient calculating device 64
Calculate the luminance signal using the correction coefficient calculated by
It is stored in the output image memory 62. From the estimation of the luminance signal for all the pixels, storage in the output image memory 62 is performed.
Complete the process. The luminance signal image stored in the output image memory 62 is transmitted to an external display device, output device, or the like through the luminance image output terminal 63 under the control of the control device 52. (Effects of Configurations (19) to (22)) With respect to an image signal composed of a plurality of color signals, it is possible to estimate image information from all neighboring pixels by using a spatial correlation between the color signals. In addition, image information can be estimated with higher accuracy than when each color signal is independently recovered.

When a specific object is set as a subject, a priori knowledge regarding the spatial correlation between the respective color signals can be obtained in advance and effectively utilized. By switching the foresight information according to the type of the image signal, more accurate estimation can be performed.

The partial area obtained from the color signal in the predetermined area,
Alternatively, the accuracy of the approximate cross-correlation between the respective color signals differs depending on the region. By applying the general function form of cross-correlation between colors and the function form of cross-correlation between colors obtained according to the type of image to the cross-correlation estimated from image information, local accuracy of image information estimation accuracy is Can be suppressed. It is also possible to obtain the cross-correlation from the surrounding cross-correlation by interpolation at a position where the cross-correlation estimation accuracy is considered to be low, or to improve the accuracy by restricting the cross-correlation to have a specific property. It is possible.

[0095]

According to the present invention, adaptive estimation depending on an image structure is performed using spatial correlation information on a local region of an image signal to be estimated. This makes it possible to perform optimal estimation.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of an image processing system to which a first embodiment of the present invention is applied.

FIG. 2 is a diagram showing an arrangement of color filters on a CCD.

FIG. 3 is a diagram showing a configuration of an area extracting device shown in FIG. 1;

FIG. 4 is a diagram showing a configuration of a spatial correlation information calculation device shown in FIG.

FIG. 5 is a diagram illustrating a configuration of an image processing system to which a second embodiment of the present invention is applied.

FIG. 6 is a diagram showing an area extraction image for edge detection.

7 is a diagram showing a configuration of the spatial correlation information calculation device shown in FIG.

FIG. 8 is a diagram illustrating a configuration of an image processing system to which a third embodiment of the present invention is applied.

FIG. 9 is a diagram illustrating a configuration of an image processing system to which a fourth embodiment of the present invention is applied.

10 is a diagram showing a configuration of the structure information extracting device shown in FIG.

FIG. 11 is a diagram illustrating a configuration of an image processing system to which a sixth embodiment of the present invention is applied.

FIG. 12 is a diagram illustrating a configuration of an image processing system to which a seventh embodiment of the present invention is applied.

FIG. 13 is a diagram illustrating a configuration of an image processing system to which an eighth embodiment of the present invention is applied.

FIG. 14 is a diagram illustrating an example of a color component mask.

FIG. 15 is a diagram for explaining a cross-correlation between red and green component masks.

FIG. 16 is a diagram showing a data distribution of a cross-correlation function.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Lens, 2 ... Color filter, 3 ... CCD, 4 ... Single-panel digital camera, 5 ... A / D converter, 6 ... ROM, 7
... Image memory, 8 ... Image signal, 9 ... Imaging device information storage device, 10 ... Image information storage device, 11 ... Estimated position determination device, 12 ... Region extraction device, 13 ... Correlation range storage device, 1
4 ... Spatial correlation information calculation device, 15 ... Estimation coefficient calculation device, 16 ... Estimation coefficient range storage device, 17 ... Image signal estimation device, 18 ... Corrected image storage device.

Claims (3)

[Claims] 1. An estimated position determining means for determining a position of an image signal to be estimated based on information on an imaging device for generating an image signal, and a position of the image signal obtained from the estimated position determining means. Region extraction means for extracting a local region of a predetermined size; spatial correlation information calculation means for calculating spatial correlation information on the local region obtained from the region extraction means; An image processing apparatus comprising: an image signal estimating unit that estimates an image signal corresponding to a position of the image signal obtained from the estimated position determining unit based on spatial correlation information. 2. A structure information extracting means for extracting structure information relating to a local region obtained from the area extracting means, and based on the structure information extracted by the structure information extracting means.
2. The image processing apparatus according to claim 1, further comprising: changing means for changing a method of calculating spatial correlation information by said spatial correlation information calculating means. 3. Estimated position determining means for determining a position of an image signal to be estimated based on information on an image pickup device for generating an image signal composed of a plurality of color signals, and said image obtained from said estimated position determining means. Region extracting means for extracting a local region of a predetermined size including a signal position from an image signal containing a plurality of color signals; and spatial correlation information for calculating spatial correlation information on the local region obtained from the region extracting device Calculating means; color correlation information calculating means for calculating color correlation information on a local region obtained from the area extracting means; spatial correlation information calculated by the spatial correlation information calculating means; and the color correlation information calculating means Image signal estimating means for estimating an image signal corresponding to the position of the image signal obtained from the estimated position determining means based on the color correlation information calculated from The image processing apparatus characterized by comprising a.