JP4545206B2 - Image processing apparatus and recording medium storing image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus and a recording medium storing an image processing program.

  FIG. 22 is a diagram showing a typical electronic still camera system. Image data obtained by photographing a subject with the electronic still camera 804 shown in FIG. 22A is normally stored in a memory card 805 as shown in FIG. Further, a color printer 801 as shown in FIG. 22C can be connected by a connection cable to perform color printing of a small size of about A6 size.

  Further, the memory card 805 can be inserted into a docking station 802 as shown in FIG. 22D by being stored in a predetermined adapter, and the TV shown in FIG. 22E through the docking station 802. An image can be observed with the monitor 800. Further, by connecting an MO drive 803 as shown in FIG. 22 (f) to the docking station 802, image data can be stored in the MO disk 806 shown in FIG. 22 (g).

  On the other hand, image data obtained by the electronic still camera 804 can be transferred to a desktop personal computer 809 as shown in FIG. Further, the memory card 805 can be stored in a predetermined adapter and image data can be taken into the notebook computer 810. Furthermore, the image data on the MO disk 806 can be transferred to the notebook computer 810 via a predetermined MO drive. Compared with the TV monitor 800, the monitor of the desktop personal computer 809 and the liquid crystal screen of the notebook personal computer 810 can display with higher definition. Furthermore, it is also possible to connect a desktop personal computer 809 or notebook personal computer 810 to a color printer 811 larger than the color printer 801 as shown in FIG.

  In the electronic still camera system, the number of pixels of the electronic still camera is generally about 640 × 480 (about 300,000 pixels) to 1280 × 1024 (about 1.3 million pixels). On the other hand, about 300,000 pixels are required for TV monitors, about 1 million pixels for personal computer monitors, about 1.3 million pixels for 300 dpi A6 size printing, and about 5 million pixels for A4 size printing. In addition, even with an electronic still camera, a relative decrease in the number of pixels occurs due to shooting at 1/2 × 1/2 size of the number of pixels corresponding to the image quality mode or digital zooming. There are many cases where the number of pixels and the number of pixels required for output do not match.

  As the electronic still camera, an image pickup system using a single-plate, two-plate, or three-plate pixel shift type CCD is generally used. Regarding the technology for improving the resolution by the pixel shifting method, for example, “Image Input Technology Handbook” (published on March 31, 1992, first edition, edited by Yuji Kiuchi, published by Nikkan Kogyo Shimbun), pages 143 to 145, and Generally described on pages 259-260.

  In such an imaging system, one pixel is composed of a plurality of color signals, and at least one or more color signals are often missing depending on the position of the pixel.

  FIG. 23 shows an arrangement of cyan (Cy), magenta (Mg), yellow (Ye) and green (G) complementary color mosaic filters generally used in a single-plate imaging system. In FIG. 23, the luminance signals corresponding to the n and n + 1 lines in the even field are Ye, n and Ye, n + 1, respectively, and the color difference signals are Ce, n and Ce, n + 1. Similarly, the luminance signals corresponding to the n and n + 1 lines in the odd field are Yo, n, Yo, n + 1, and the color difference signals are Co, n, Co, n + 1. These signals are given by

Yo, n = Yo, n + 1 = Ye, n = Ye, n + 1 = 2R + 3G + 2B (1)
Co, n = Ce, n = 2R-G (2)
Co, n + 1 = Ce, n + 1 = 2B-G (3)
However, Cy, Mg, and Ye are represented by the following formula using G, red (R), and blue (B).

Cy = G + B (4)
Mg = R + B (5)
Ye = R + G (6)
As shown in equation (1), the luminance signal is generated on all lines of even and odd fields. On the other hand, as shown by the equations (2) and (3), the two color difference signals are generated only for each line, and the missing lines are compensated by linear interpolation. Thereafter, the three primary colors R, G, and B can be obtained by performing a matrix operation. In such a method, the color difference signal has only a half information amount with respect to the luminance signal, and an artifact called color moire occurs at the edge portion. In general, in order to reduce such color moire, a low-pass filter using a crystal filter is disposed on the front surface of the image sensor. However, a new problem arises that the resolution is lowered due to the insertion of the low-pass filter.

  On the other hand, a method for correcting a color difference signal using a luminance signal component is proposed instead of performing simple interpolation using only a color difference signal as described above. One is that the luminance signal Y is generated by linear interpolation, and the color difference signal C is supplemented by linear interpolation in a region where the luminance signal Y is little changed, and the luminance signal Y is expressed in the following (7) The restored color difference signal C ′ is obtained by transformation as shown in the equation.

C ′ = aY + b (7)
Here, a and b are constants.

In Japanese Patent Laid-Open No. 05-056446, the luminance signal Y is created by linear interpolation. For the color difference signal C, the luminance signal Y and the color difference signal C are processed by an electric circuit low-pass filter, and each low frequency component is processed. Gain Ylow and Clow. The color signal C ′ from which the omission has been recovered is obtained by the equation (8).

This corresponds to replacing the color difference signal with a corrected luminance signal. In the above prior art, the color difference signal is corrected on the basis of the luminance signal, but the luminance signal itself has only half the amount of information as compared to the three-plate type. Also in these methods, it is necessary to use a low-pass filter using a crystal filter in order to reduce color moire. For this reason, the resolution of the reference luminance signal is further reduced, and an image quality improvement comparable to that of the three-plate type cannot be realized.
"Image Input Technology Handbook" (published on March 31, 1992, first edition, edited by Yuji Kiuchi, published by Nikkan Kogyo Shimbun), pages 143 to 145, and pages 259 to 260 JP 05-056446 A

  In the prior art, the color difference signal is compensated based on linear interpolation or a luminance signal, and it is not possible to cope with the point that the missing color signal is restored with high accuracy. In particular, it is impossible to cope with the point that a signal that has been compensated for a missing color signal based on linear interpolation or a luminance signal is processed again to restore it with high accuracy.

  The present invention has been made paying attention to such a problem, and an object of the present invention is to provide an image processing apparatus capable of restoring a color signal with high accuracy even for a signal once subjected to processing such as linear interpolation, and the like. The object is to provide a recording medium storing an image processing program.

In order to achieve the above object, an image processing apparatus according to an aspect of the present invention is an image processing apparatus that processes an image signal obtained via a single-plate, two-plate, or three-plate pixel-shift imaging system. , conversion for converting a color signal missing of the image signal and the first recovery means for recovering at linear interpolation, the original image signal obtained by image signal restored by the first restoring means by the imaging system means and a second restoring means for the color signal recovers based on the color correlation between the color signals missing the converted image signal, converts the output to output the image signal restored by said second recovery means An output unit that performs processing .

  According to this aspect, it is possible to restore a color signal with high accuracy even for a signal once subjected to processing such as linear interpolation.

The output unit of the image processing apparatus may perform a direct output process for outputting the image signal recovered by the first recovery unit as it is, instead of the conversion output process.
The output unit of the image processing apparatus may perform the direct output processing by bypassing the processing by the conversion unit and the second recovery unit.
Further, a switching means for switching between the conversion output process and the direct output process may be further provided.
Further, the switching means of this image processing apparatus may automatically switch based on the number of pixels of the image signal at the time of photographing and the number of pixels required for the output medium, or the use of electronic zoom. Thereby, an appropriate image quality can be obtained in an appropriate processing time by automatic processing.

  Further, the switching by the switching means of the image processing apparatus may be performed manually. Thus, signal processing can be performed with priority given to either processing time or image quality based on the user's wishes.

In addition, the recording medium according to another aspect of the present invention provides a first recovery for recovering a color signal lacking an image signal obtained via a single-plate, two-plate, or three-plate pixel-shift imaging system by linear interpolation. processing and color between the first and conversion process for converting the image signal restored to the original image signal obtained by the imaging system by the recovery processing, each color signal the color signal to be missing the converted image signal storing a second recovery process for recovering on the basis of the correlation, the program including instructions for executing a switching process for switching between said second recovery process and the conversion process, to the computer.
According to this another aspect, it is possible to restore the color signal with high accuracy even for a signal once subjected to processing such as linear interpolation.

  According to the present invention, a color signal can be restored with high accuracy even for a signal once subjected to processing such as linear interpolation.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  In Japanese Patent Application No. 10-015325, the present applicant has proposed a method for recovering a color signal that is missing in high definition based on the color correlation between the color signals in the local region. This method can obtain an image comparable to the three-plate type when the local area is a uniform subject and has a single color correlation, but when there are a plurality of subjects and a plurality of color correlations. A false signal is generated. For this reason, in this Japanese Patent Application, a method of verifying reliability for each local region is employed, but the generation of false signals cannot be completely prevented. In addition, since the region with low reliability is switched to linear interpolation, if the priority is given to suppression of false signals, the effect of improving the image quality is reduced. Furthermore, it is difficult to realize high-speed processing for calculating the color correlation for each local region.

  Therefore, a method for overcoming such a problem will be described in detail below based on a specific embodiment.

(First embodiment)
FIG. 1 is a configuration diagram of a first embodiment of the present invention. In the first embodiment, it is assumed that the image processing apparatus of the present invention is configured by the electronic still camera 804 in the electronic still camera system of FIG. 22 and outputs an image-processed signal to the memory card 805 or the color printer 801. Yes.

  An input unit 101 using a single-plate CCD is connected to an R signal buffer 102, a G signal buffer 103, and a B signal buffer 104. The R signal buffer 102, the G signal buffer 103, and the B signal buffer 104 are connected to the neighboring region extraction unit 106 and the linear interpolation unit 116 via the process switching unit 105. The neighborhood area extraction unit 106 is connected to the divided image buffer 110 via the parameter calculation unit 107, the parameter buffer 108, and the image signal division unit 109.

  The divided image buffer 110 and the process switching unit 105 are connected to a uniform region extraction unit 111, and the uniform region extraction unit 111 is connected to the color correlation regression unit 113 and the missing pixel restoration unit 114 via the uniform region buffer 112. ing. The color correlation regression unit 113 is connected to the missing pixel restoration unit 114. The missing pixel restoration unit 114 is connected to an output unit 117 such as a memory card or a printer via a restored image buffer 115. The linear interpolation unit 116 is also connected to the output unit 117.

  A control unit 118 such as a microcomputer includes an input unit 101, a process switching unit 105, a neighborhood region extraction unit 106, an image signal division unit 109, a uniform region extraction unit 111, a color correlation regression unit 113, a missing pixel restoration unit 114, The linear interpolation unit 116 is connected.

  The operation of the above configuration will be described below. Here, a description will be given along the flow of signals. The three RGB signals from the input unit 101 are transferred to the R signal buffer 102, the G signal buffer 103, and the B signal buffer 104 under the control of the control unit 118. The color signals in the signal buffers 102, 103, and 104 are transferred to the neighboring region extraction unit 106 or the linear interpolation unit 116 via the processing switching unit 105 based on the control of the control unit 118. This selection can also be performed by a changeover switch (not shown), or automatic switching such as transferring to the neighborhood extraction unit 106 when the digital zoom is used and transferring to the linear interpolation unit 116 when the digital zoom is not used. It is also possible to do.

  When the color signal is transferred to the linear interpolation unit 116, the missing color signal is restored by a known linear interpolation, transferred to the output unit 117, and the process ends. When transferred to the neighboring area extraction unit 196, the signal is sequentially scanned in units of pixels, and at least one or more neighboring areas of a predetermined size including the current target pixel are extracted. The size to be extracted is determined based on the filter arrangement used in the input unit 101, and the number of areas to be extracted is determined in consideration of the processing speed and the image quality improvement effect. The parameter calculation unit 107 obtains a spectral gradient from the color signal existing in the neighboring region for each neighboring region, and classifies it into a class based on the sign of the gradient to obtain a parameter for region division. When one neighboring region is extracted, the class of the neighboring region is used as the parameter of the current target pixel. The parameters calculated by the parameter calculation unit 107 are transferred to the parameter buffer 108 and stored. The control unit 118 repeats the above process until the scanning of all pixels is completed. When scanning of all pixels is completed, classes for all pixels exist as parameters in the parameter buffer 108.

  Next, the control unit 118 causes the parameters on the parameter buffer 108 to be transferred to the image signal dividing unit 109. The image signal dividing unit 109 performs region separation for each class by performing known smoothing and labeling processing, and transfers the result to the divided image buffer 110. After the region division, the uniform region extraction unit 111 reads the RGB three signals corresponding to the individual regions from the processing switching unit 105 sequentially from the region division result on the divided image buffer 110 under the control of the control unit 118, and the uniform region buffer 112. Forward to.

  The color correlation regression unit 113 returns the color correlation of each color signal on the uniform region buffer 112 to a linear format, and transfers this linear format data to the missing pixel restoration unit 114. The missing pixel restoration unit 114 restores or restores the missing color signal based on each color signal on the uniform region buffer 112 and the linear data from the color correlation regression unit 113 and transfers the restored color signal to the restored image buffer 115. The control unit 118 repeats the above process until all the areas on the divided image buffer 110 are completed. When all the regions are completed, a complete image signal in which the missing signal is recovered exists on the restored image buffer 115, and this signal is output to the output unit 117.

  FIG. 2 is an explanatory diagram illustrating an example of a specific configuration of the input unit 101. A single-plate CCD 203 is arranged via a lens system 201 and a low-pass filter 202. The CCD 203 is, for example, a CCD having an R, G, B primary color filter arrangement shown in FIG. The image signal obtained by the CCD 203 is converted into RGB three signals via the A / D converter 204, the color separation circuit 205, the process circuits 206, 207, 208, and the matrix circuit 209, and the R signal buffer 102, G The signal is stored in the signal buffer 103 and the B signal buffer 104. The CCD 203 is connected to a CCD drive circuit 211 that is operated based on the clock generator 210.

  FIG. 3 is an explanatory diagram showing an example of a specific configuration of the filter arrangement in the CCD 203 of FIG. Here, there is a 2 × 2 size basic arrangement as shown in FIG. 3A, and this basic pattern is repeated repeatedly to fill all pixels on the CCD as shown in FIG. 3B. It has become. FIG. 3C is a diagram showing another basic arrangement of 2 × 4 size.

FIG. 4 is an explanatory diagram regarding region division based on the gradient of the spectrum in the neighborhood region extraction unit 106 and the parameter calculation unit 107. FIG. 4A shows an example of an input image. The upper area A is white and the lower area B is red. FIG. 4B is a diagram in which the intensities (I) are plotted against the three wavelengths (λ) in the A region and the B region. The A region is white, and the gradient of the spectral intensity with respect to the three wavelengths R, G, and B is almost equal to I _R (A) = I _G (A) = I _B (A), which is class 0.

Further, the B region is red, and the intensity of the R signal is large because the gradient of the spectrum intensity I _R (B)> I _G (B) = I _B (B), and G and B are almost equal and smaller than the intensity of the R signal. . This is class 4. FIG. 5 shows 13 combinations of R, G, and B signal gradients, to which classes are added when classification is impossible.

FIG. 4C shows an image when the input image of FIG. 4A is captured by a single-plate CCD having the filter arrangement shown in FIG. 3B. In order to obtain the gradient of the spectrum, RGB3 signals are required. For this reason, when a certain pixel is set as a target pixel, an area equal to the size of the basic arrangement of the filter is set as a neighborhood area, and a spectrum gradient is obtained from the neighborhood area. In this embodiment, a neighborhood area of 2 × 2 size is set. As shown in FIG. 4C, there can be four cases where there are four neighboring regions containing the target pixel. In the present embodiment, the gradient of the spectrum is obtained in all four neighboring regions. When a plurality of identical color signals are included in the neighborhood area, they are averaged. In FIG. 4C, the four neighboring regions are I _R (A) = I _G (A) = I _B (A), and are classified into class 0. If the classes are different in the four neighboring areas, they are classified into the largest number of classes, and if there are no largest number of classes, they are classified into the indefinite class 13.

  FIG. 4D shows a state in which classes 0 to 13 are assigned to each pixel as described above, and the classified images are output to the divided image buffer 110. Since the classification is performed based on the spectrum form, the same class has the same spectrum form, and the color correlation can be approximated by the same relational expression. The image signal dividing unit 109 performs region separation for each class by performing known smoothing and labeling processing on the classified images. FIG. 4E shows a state of being divided into three areas, and the result is transferred to the divided image buffer 110.

FIG. 6 is an explanatory diagram of the return of the color correlation to the line format in the color correlation regression unit 113. FIG. 6A shows an example of the input image. The image processed by the color correlation regression unit 113 is an image that is divided into regions that are uniform in terms of color correlation based on the spectral gradient in the parameter calculation unit 107. FIG. 6B shows an image when the input image of FIG. 6A is picked up by a single CCD having the filter arrangement shown in FIG. 3B. Hereinafter, the RGB3 signal is represented by S _i (i = R, G, B). The average of the S _i signal is AV_S _i , and the standard deviation is DEV_S _i . If a linear color correlation is established between two color signals S _i and S _j (j = R, G, B, and j is not equal to i), the linear form is

It is regressed with. FIG. 6C shows the regression of the color correlation between RG signals to a linear format. Similarly, regression to the linear format is performed between the signals G-B and RB. If the above linear format is obtained, the G signal can be recovered from the pixel where the R signal exists, and conversely, the R signal can be recovered from the pixel where the G signal exists. Recovery can be similarly performed between the R and B signals and between the G and B signals. By performing the above process on all regions in the divided image buffer 110, an image with all color signals restored on the restored image buffer 115 is obtained.

  In the present embodiment, processing is performed in hardware, but it is also possible to perform processing in software as shown in FIG.

  That is, in FIG. 7, first, in step S1, a single plate image signal from the input unit 101 is read. Next, in step S2, processing is selected based on whether or not a changeover switch (not shown) or electronic zoom is used. If linear interpolation is performed, the process branches to step S3. Otherwise, the process branches to step S5. In step S3, linear interpolation is performed to recover missing color signals. In step S4, the recovered color signal is output and the processing is completed.

Next, in step S5, region division is performed based on the spectral gradient. Detailed processing in step S5 will be described later. Next, in step S6, the region-divided image is scanned to individually extract individual regions, and the next stage processing is performed. Next, in step S7, the calculated R uniform region, G, an average AV_ S _i and the standard deviation DEV_ S _i of B color signals. Next, in step S8, the line format between RG, GB and RB is calculated based on the equation (9).

  In step S9, the missing color signal in the region is recovered based on the linear format. In step S10, the recovered color signal is output. Next, in step S11, it is determined whether or not scanning of all areas has been completed. If completed, the process is completed, and if not completed, the process branches to step S6.

  The region division in step S5 is performed as shown in FIG. First, in step S5-1, the image signal is scanned in units of pixels, and the next stage processing is performed. Next, in step S5-2, four 2 × 2 size neighboring areas containing the current pixel of interest are extracted. Next, in step S5-3, a class is obtained from the spectral gradient of the neighboring region based on FIG. Next, in step S5-4, it is determined whether or not the largest number of classes exist. If there is, the process branches to step S5-5, and if not, the process branches to step S5-6. In step S5-5, the largest number of classes are output. In step S5-6, 13 is output as the indefinite class.

  Next, in step S5-7, it is determined whether or not scanning of all pixels has been completed. If completed, the process branches to step S5-8, and if not completed, the process branches to step S5-1.

  Next, in step S5-8, smoothing is performed using a 3 × 3 size median filter. In step S5-9, the area is divided by labeling processing. Next, in step S5-10, an area divided image is output.

  As described above, a region having a single color correlation can be obtained by obtaining a spectral gradient from a neighboring region based on the filter arrangement and performing region division on the input image signal. The color correlation is calculated by regression to the linear format for each individual area, and the missing pixels are recovered. For this reason, it is possible to recover high-frequency components, which is impossible with conventional linear interpolation, and to generate a high-definition reproduced image.

  Further, since the signal is divided into uniform areas in advance, generation of false signals can be prevented. Furthermore, in the conventional example, a large number of regression calculations are required for each rectangular area, but it is only necessary to perform the regression calculation only for a larger area, so that the calculation time can be shortened. Since the gradient of the spectrum can be obtained from the magnitude relationship between the color signals, it can be performed at high speed and at low cost. Further, since it is possible to select restoration by linear interpolation of normal image quality as necessary, the processing speed can be further increased.

  In this embodiment, a 2 × 2 size filter arrangement is used as shown in FIG. 3A. However, the present invention is not limited to this and can be freely set. For example, the filter arrangement shown in FIG. In this case, the neighborhood area extraction unit 106 extracts a 2 × 4 size area. Further, it is not necessary to be limited to extracting four neighboring areas, and if it is desired to shorten the calculation time, for example, one can be performed. In this case, the parameter calculation unit 107 does not need to obtain the largest number of classes, and there are no indefinite classes. In the present embodiment, the compression processing is not shown, but when saving to a memory card or the like, a configuration in which a known compression unit such as JPEG is added before the output unit 117 is also possible.

(Second Embodiment)
The second embodiment of the present invention will be described below. The configuration of the second embodiment is basically the same as that of the first embodiment shown in FIGS. 1 and 2 except that the function of the parameter calculation unit 107 in the first embodiment is different.

  The operation of the second embodiment will be described below. This is basically the same as that of the first embodiment, and only different parts will be described below. FIG. 9 is an explanatory diagram showing an example of a specific configuration of the filter arrangement in the CCD 203 of FIG. In the present embodiment, unlike the first embodiment, Cy, Mg, Ye, and G complementary color filters are used. As shown in FIG. 9A, a basic arrangement of 2 × 4 size is used, and as shown in FIG. 9B, this basic pattern is repeated repeatedly to fill all pixels on the CCD. Yes. The color separation circuit 205, the process circuits 206, 207, 208, and the matrix circuit 209 in FIG. 2 are changed in accordance with the complementary color filter arrangement, and the R signal buffer 102 and the G signal buffer 103 in FIGS. The B signal buffer 104 is expanded to a 4-signal buffer for Cy, Mg, Ye, and G.

FIG. 10 is an explanatory diagram relating to region division based on the luminance signal in the neighborhood region extraction unit 106 and the parameter calculation unit 107. FIG. 10A shows an example of an input image. The upper area A is white and the lower area B is red. FIG. 10B shows an image when the input image of FIG. 10A is picked up by the single-plate CCD having the filter arrangement shown in FIG. 9B. In order to calculate the luminance signal from this single plate state, signals of four colors Cy, Mg, Ye, and G are required. For this reason, when a certain pixel is set as a target pixel, a neighborhood area of 2 × 2 size is set in the lower left direction. In the filter arrangement shown in FIG. 9B, if a 2 × 2 size neighborhood region is set in the lower left direction, signals of four colors Cy, Mg, Ye, and G always exist. The luminance signal Y is Y = Cy + Mg + Ye + G = 2R + 3G + 2B (10)
Given in. FIG. 10C shows a luminance signal calculated for each pixel based on the equation (10). FIG. 10D shows the edge intensity calculated by performing a known edge extraction process on this luminance signal. The result of binarization processing using a predetermined threshold, for example, 15 in this embodiment, is indicated by hatching with respect to this edge strength. FIG. 10E shows a result of performing region separation by performing a known labeling process based on the binarized pixels. This embodiment shows a state where the image is divided into four areas, and the result is transferred to the divided image buffer 110. The subsequent processing is performed in the same manner as in the first embodiment, and four signals of Cy, Mg, Ye, and G are recovered for each pixel and transferred to the recovered image buffer 115. Thereafter, three signals R, G, and B are calculated from the relationships shown in the equations (4) to (6) and output to the output unit 117.

  In the present embodiment, processing is performed in hardware, but it is also possible to perform processing in software as shown in FIG. The content is the same as that of the first embodiment shown in FIG. 7, and only step S5 is replaced with step S12. The area division in step S12 is performed as shown in FIG.

  That is, first, in step S12-1, the image signal is scanned pixel by pixel, and the next process is performed. Next, in step S12-2, a 2 × 2 size neighboring region containing the current pixel of interest is extracted. Next, in step S12-3, a luminance signal is calculated based on equation (10). Next, in step S12-4, it is determined whether or not scanning of all pixels has been completed. If completed, the process branches to step S12-5, and if not completed, the process branches to step S12-1. In step S12-5, edge extraction is performed. Next, in step S12-6, binarization processing is performed. In step S12-7, a labeling process is performed to divide the area. Next, in step S12-8, the region divided image is output.

  As described above, a region where a single color correlation is established can be obtained by obtaining the edge strength of the luminance signal from the neighboring region based on the filter arrangement for the input image signal and performing region division. The color correlation is calculated by regression to the linear format for each individual area, and the missing pixels are recovered. For this reason, it is possible to recover high-frequency components, which is impossible with conventional linear interpolation, and to generate a high-definition reproduced image.

  Further, since the signal is divided into uniform areas in advance, generation of false signals can be prevented. Furthermore, in the conventional example, a large number of regression calculations are required for each rectangular area, but it is only necessary to perform the regression calculation only for a larger area, so that the calculation time can be shortened. The calculation of the luminance signal can be realized only by the addition operation, and can be performed at high speed and at low cost. Further, since it is possible to select restoration by linear interpolation of normal image quality as necessary, the processing speed can be further increased.

  In this embodiment, a complementary color CCD is used, but the present invention can also be applied to a primary color CCD. For example, FIG. 13 shows a luminance signal calculation method when the filter arrangement shown in FIG. FIGS. 13A, 13B, and 13C show a 3 × 3 size neighborhood extraction unit when the target pixel is R, G, and B. FIG. When this neighborhood region is multiplied by the matrix coefficient shown in FIG. 13D, 4R + 8G + 4B is obtained in all cases, and this can be normalized and used as a luminance signal. It can also be applied to a two-plate or three-plate pixel shift type.

(Third embodiment)
The third embodiment of the present invention will be described below. FIG. 14 is a configuration diagram of the third embodiment of the present invention. In the third embodiment, the image processing apparatus of the present invention includes an electronic still camera 804 and a docking station 802 in the electronic still camera system of FIG. These two are separated, and the image signal obtained by the electronic still camera 804 is input to the docking station 802 via the memory card 805 and subjected to image processing, and the processed signal is connected to the docking station 802. The data is output to the color printer 801, the TV monitor 800, and the MO drive 803.

  The signal of the input unit 301 using the two-plate CCD in the electronic still camera 804 is transferred to the R signal buffer 302, the G signal buffer 303, and the B signal buffer 304, and the memory card 306 via the linear interpolation unit 305. Is output. On the other hand, the card reading unit 307 in the docking station is connected to the process switching unit 308. The process switching unit 308 is connected to the conversion unit 309 and the output unit 325. The conversion unit 309 receives a signal from the filter arrangement ROM 310 and is connected to the R signal buffer 311, the G signal buffer 312, and the B signal buffer 313. The R signal buffer 311, the G signal buffer 312, and the B signal buffer 313 are transferred in the order of the local region extracting unit 314, the color correlation regression unit 315, the parameter buffer 316, the local region dividing unit 317, and the divided image buffer 318. Is done. The signals from the divided image buffer 318 and the local region extraction unit 314 are transferred to the uniform region extraction unit 319, and the signal from the uniform region extraction unit 319 is restored to the color correlation regression unit 321 and the missing pixel restoration via the uniform region buffer 320. Transferred to the unit 322. The signal from the color correlation regression unit 321 is connected to the missing pixel restoration unit 322. A signal from the missing pixel restoration unit 322 is output to an output unit 325 such as a printer or a monitor via the restored image buffer 323 and the addition averaging unit 324. The control unit 326 such as a microcomputer includes a process switching unit 308, a local region extraction unit 314, a local region division unit 317, a uniform region extraction unit 319, a color correlation regression unit 321, a missing pixel restoration unit 322, and an averaging unit 324. Connected to.

  The operation of the third embodiment will be described below. The three RGB signals from the input unit 301 are transferred to the R signal buffer 302, the G signal buffer 303, and the B signal buffer 304, and the missing color signal is restored by the linear interpolation unit 305 and output to the memory card 306. Is done. The image signal on the memory card 306 is inserted into the card reading unit 307 in the docking station 802, and the image signal is transferred to the processing switching unit 308. The process switching unit 308 transfers the image signal to the conversion unit 309 or the output unit 325 based on the control of the control unit 326. This selection can be performed by a changeover switch (not shown). The number of pixels of the image signal is compared with the number of pixels of the output medium. If the number of pixels of the output medium such as a color printer is large, the conversion unit 309 is performed. When the number of pixels of an output medium such as a TV monitor is small, automatic switching such as transfer to the output unit 325 without processing is also possible.

When transferred to the conversion unit 309, the conversion unit 309 reads out the filter arrangement used in the imaging system from the filter arrangement ROM 310, and based on this filter arrangement information, the image signal on the memory card 306 can be obtained in the original imaging system. Each signal is transferred to the R signal buffer 311, the G signal buffer 312, and the B signal buffer 313. The local area extraction unit 314 sequentially scans the converted image signal in units of pixels, and extracts a local area having a predetermined size, for example, a 6 × 6 size, including the current target pixel. The color correlation regression unit 315 calculates a constant term of the color correlation in the local region according to the equation (9) in the first embodiment. Constant term is equivalent to (9) obtained by modifying as shown formula (11) formula _{(DEV_ S i / DEV_ S j} ) AV_ S j + AV_ S i.

  Since this constant term has a value close to 0 in a region where a single color correlation is established, region division can be performed based on this value. The color correlation regression unit 315 sets a small region of 2 × 2 size within the local region and sequentially scans this. The constant term is calculated between three sets of signals R−G, G−B, and R−B for each minute region, and the maximum value of the three constant terms is transferred to the parameter buffer 316. The controller 326 repeats the above process until the scanning in the local region is completed. When the scanning is completed, the parameter buffer 316 has a constant term for the pixels in the local region as a parameter.

  Next, the control unit 326 causes the parameters on the parameter buffer 316 to be transferred to the local region dividing unit 317. The local region dividing unit 317 performs binarization processing on the above parameters using a predetermined threshold value. As a result, an area where a single color correlation is established is divided into 0, and other boundary areas are divided into 1. Next, region separation is performed by performing a known labeling process, and the result is transferred to the divided image buffer 318. After the region division is completed, the uniform region extraction unit 319 reads the RGB three signals belonging to the same region as the current pixel of interest from the local region extraction unit 314 from the region division result on the divided image buffer 318 under the control of the control unit 326, and uniformly. Transfer to the area buffer 320. The color correlation regression unit 321 returns the color correlation of each color signal on the uniform region buffer 320 to a linear format, and transfers this linear format data to the missing pixel restoration unit 322. In the present embodiment, since a two-plate CCD is assumed, there are no missing pixels in the G signal. Therefore, in order to recover the color signal lacking the R and B signals, the regression to the linear format is performed between the two sets of signals RG and GB. The missing pixel restoration unit 322 restores the missing color signal based on each color signal on the uniform region buffer 320 and the linear data from the color correlation regression unit 321 and transfers it to the restored image buffer 323. The local area that is the basis of the recovery process is set while scanning the converted image signal in units of pixels. For this reason, an overlap according to the size of the local region occurs, and the recovered color signal also overlaps. In the present embodiment, it is assumed that the image is stored while being accumulated in the restored image buffer 323. The control unit 326 repeats the above process until the scanning of the image signal by the local region extraction unit 314 is completed. When all the pixels are completed, the addition averaging unit 324 averages the accumulated image signals on the restored image buffer 323 according to the number of integrations, and outputs the averaged image signal to the output unit 325.

  FIG. 15 is an explanatory diagram illustrating an example of a specific configuration of the input unit 301. A G signal low-pass filter 402 and a G signal CCD 404 and R and a B signal low pass filter 403 and an R and B signal CCD 405 are arranged via a lens system 401. The G signal CCD 404 has a G filter on all pixels, and the R and B signal CCDs 405 have R and B filters arranged in a checkered pattern. The electrical signal from the G signal CCD 404 is stored in the G signal buffer 304 via the A / D converter 406. The R and B signal CCDs 405 are transferred to the R signal buffer 302 and the B signal buffer 303 via the A / D converter 407 and the R / B separation circuit 408. The G signal CCD 404 and the R and B signal CCDs 405 are respectively connected to the G signal CCD driving circuit 410 and the R and B signal CCD driving circuits 411 operated based on the clock generator 409.

  FIG. 16 is an explanatory diagram regarding region division based on constant terms in the local region extraction unit 314 and the color correlation regression unit 315. FIG. 16A shows an example of an input image, in which the upper area A is white and the lower area B is red. FIGS. 16B and 16C show images of local regions when the input image of FIG. 16A is imaged by the two-plate CCD shown in FIG. The local area has a size of 6 × 6, for example. In this local region, three signals R, G, and B are required to calculate the constant term of the color correlation between the color signals. The color correlation regression unit 315 sets a small region of 2 × 2 size, and scans the local region with the upper left as the origin as shown in FIGS. 16B and 16C. If a minute region of 2 × 2 size is set, there are always three signals of R, G, and B. Within this minute area, three sets of color correlation constant terms are obtained for each minute area based on the equation (11), and the maximum value is selected. FIG. 16 (d) shows the selected constant term. Since the minute area is 2 × 2 size, a constant term corresponding to 5 × 5 size is obtained at this stage. The result of binarizing the constant term using a predetermined threshold, for example, 15 in the present embodiment, is indicated by hatching. FIG. 16E shows a result of performing region separation by performing a known labeling process based on the binarized pixels. In the present embodiment, the target pixel belongs to one label, and the uniform region extraction unit 319 extracts pixels belonging to one label.

  In this embodiment, processing is performed in hardware, but processing can be performed in software as shown in FIG.

  That is, first, in step S21, an image signal from the input unit 301 is read. Next, in step S22, a process is selected based on whether or not a changeover switch (not shown) or electronic zoom is used. In the case of linear interpolation, the process is completed. Otherwise, the process branches to step S23. In step S23, the original image signal obtained by the imaging system is converted. Next, in step S24, the original image signal is scanned pixel by pixel, and the next stage processing is performed. In step S25, a 6 × 6 size local region containing the current pixel of interest is extracted. In step S26, the local area is divided based on the constant term of the color correlation. Details of the processing in step S26 will be described later.

Next, in step S27, calculates R of the current pixel of interest and the same area, G, B average AV_ Si and the standard deviation DEV_ S _i of each of the color signals. In step S28, a line format between RG and GB is calculated based on the equation (9).

  In step S29, the missing color signal in the region is recovered or restored based on the line format. In step S30, the recovered color signals are output while being integrated. Next, in step S31, it is determined whether or not scanning of all areas has been completed. If completed, the process branches to step S32. If not, the process branches to step S24. In step S32, the accumulated color signals are averaged and output.

  The above-described local region division in step S26 is performed as shown in FIG.

  That is, first, in step S26-1, the local region is scanned in units of pixels and the next stage process is performed. Next, in step S26-2, a 2 × 2 size minute region is extracted. In step S26-3, RG, GB, and R-B3 sets of constant terms are calculated based on equation (11). Next, in step S26-4, the constant term of the maximum value is output. Next, in step S26-5, it is determined whether or not the scanning of the local region has been completed. If completed, the process branches to step S26-6, and if not, the process branches to step S26-1. In step S26-6, binarization processing is performed on the obtained constant term. Next, in step S26-7, the region is divided by a labeling process. In step S26-8, the same region as the current target pixel is output.

  As described above, in this embodiment, an image signal from which a missing color signal is recovered by normal linear interpolation is converted into an original original image signal obtained by the imaging system based on the filter arrangement of the imaging system. Thereafter, by obtaining a constant term when the color correlation is regressed in a local area of a predetermined size and performing area division, an area where a single color correlation is established is obtained. For this area, the color correlation is calculated by regressing to a linear form to recover missing pixels. For this reason, it is possible to recover high-frequency components, which is impossible with conventional linear interpolation, and to generate a high-definition reproduced image. Further, since the signal is divided into uniform areas in advance, generation of false signals can be prevented.

  Further, since the processing in the present embodiment can be performed separately from the electronic still camera, it can be applied universally to a conventional electronic still camera. Furthermore, since processing is performed in units of local areas, the amount of memory used is small and can be realized at low cost. In addition, since the processing can be bypassed as necessary, it is not necessary to perform useless processing.

  In this embodiment, the division in the local region is performed using the constant term, but it is not necessary to be limited to this. The spectral gradient in the first embodiment and the edge strength of the luminance signal in the second embodiment can also be used. On the contrary, the division by the constant term of this embodiment can be used also in the first and second embodiments. In this embodiment, processing is performed by a two-plate CCD, but the present invention can also be applied to a single-plate or three-plate pixel shift type. Furthermore, when using software, it is not necessary to use a dedicated docking station, and processing can be implemented on a general desktop or laptop computer. In the present embodiment, the maximum effect can be obtained for an image signal that has not been subjected to compression processing as shown in the above example. However, although the improvement effect is reduced, the present invention can also be applied to a compressed image signal. In this case, a compression unit may be added between the linear interpolation unit 305 and the memory card 306 in FIG. 14, and an expansion unit may be added between the card reading unit 307 and the process switching unit 308.

(Fourth embodiment)
The fourth embodiment of the present invention will be described below. The fourth embodiment is basically the same as the third embodiment shown in FIGS. 14 and 15 except that the function of the color correlation regression unit 315 in the third embodiment is different.

The operation of the fourth embodiment will be described below. This is basically the same as that of the third embodiment, and only different parts will be described below. FIG. 19 is an explanatory diagram relating to region division based on a line format error based on the maximum value / minimum value in the local region extraction unit 314 and the color correlation regression unit 315. FIG. 19A shows an example of an input image, which is a uniform region. Figure 19 (b) shows a process of regression on the linear equation of color correlation average AV_ S _i and the standard deviation DEV_ S _i between R-G signal based on the equation (9). Hereinafter, the maximum value in each signal is indicated by Max_S _i , and the minimum value is indicated by Min_S _i . When the maximum value or the minimum value is substituted into the line format shown in the equation (9), an equal sign is established in a uniform region as in this embodiment.

Here, if the errors on both sides are Err_max and Err_min,

It becomes. On the other hand, FIG. 19D shows a non-uniform region in which the upper region A is white and the lower region B is red. FIG. 19 (e) is a diagram showing a line format of color correlation between RG signals of the entire region and a line format of individual color correlations of region A and region B based on equation (9). . The linear format regressed over the entire region is influenced by the individual characteristics in the regions A and B and does not show a correct color correlation. If the maximum value or the minimum value is substituted into such a line format, the equal sign will not hold.

Therefore, it is possible to perform region division using the errors Err_max and Err_min.

  The color correlation regression unit 315 sets, for example, a small region of 3 × 3 size with respect to the local region extracted by the local region extraction unit 314, and sequentially scans this. The size of the minute area is adjusted by the filter arrangement of the imaging system to be used. The two sets of errors are calculated among the three sets of signals R−G, G−B, and R−B for each minute region, and the maximum value of these errors is transferred to the parameter buffer 316. The controller 326 repeats the above process until the scanning in the local region is completed. Thereafter, the parameters on the parameter buffer 316 are transferred to the local area dividing unit 317 as in the third embodiment. The local region dividing unit 317 performs binarization processing on the above parameters using a predetermined threshold value. As a result, an area where a single color correlation is established is divided into 0, and other boundary areas are divided into 1. Next, region separation is performed by performing a known labeling process, and the result is transferred to the divided image buffer 318.

  In the present embodiment, processing is performed in hardware, but it is also possible to perform processing in software as shown in FIG. The content is the same as that of the third embodiment shown in FIG. 20, and only step S26 is replaced with step S33.

  The area division in step S33 is performed as shown in FIG.

  That is, first, in step S33-1, the local region is scanned in units of pixels and the next stage process is performed. Next, in step S33-2, a 3 × 3 small area is extracted. Next, in step S33-3, the color correlation of each color signal is returned to a linear format based on equation (9). Next, in step S33-4, an error due to the maximum value / minimum value is calculated based on the equation (13). In step S33-5, the maximum error value is output. Next, in step S33-6, it is determined whether or not scanning of all pixels has been completed. If completed, the process branches to step S33-7, and if not completed, the process branches to step S33-1. Next, binarization processing is performed in step S33-7. In step S33-8, a labeling process is performed to divide the region. Next, in step S33-9, the region divided image is output.

  As described above, an area in which a single color correlation is established can be obtained by obtaining an error when the color correlation is regressed in a local area of a predetermined size and performing area division. For this area, the color correlation is calculated by regressing to a linear form to recover missing pixels. For this reason, it is possible to recover high-frequency components, which was impossible with conventional linear interpolation, and to generate a high-definition reproduced image. Further, since the signal is divided into uniform areas in advance, generation of false signals can be prevented. Since the processing of this embodiment can be performed separately from the electronic still camera, it can be applied to general electronic still cameras. Furthermore, since processing is performed in units of local areas, the amount of memory used is small and can be realized at low cost. In addition, since the processing can be bypassed as necessary, it is not necessary to perform useless processing.

  Note that the division by error of this embodiment can also be used in the first and second embodiments.

  The invention having the following configuration is extracted from the specific embodiment described above.

1. An image processing apparatus having a single-plate, two-plate, or three-plate pixel-shift imaging system, which sequentially scans image signals in units of pixels and parameters for segmentation from at least one neighboring region including the current pixel of interest Between the color signal existing in the uniform area, the parameter calculating means for calculating the image signal, the image signal dividing means for dividing the image signal into uniform areas where a single color correlation is established based on the calculated parameters A regression means for regressing the color correlation as a linear format; and a first recovery means for recovering a missing color signal based on the color signal existing in the uniform region and the linear format. An image processing apparatus.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the first embodiment shown in FIGS. 1 to 8 and the second embodiment shown in FIGS. 1, 2, and 9 to 13. The parameter calculation means in the configuration corresponds to the neighborhood region extraction unit 106 and parameter calculation unit 107 shown in FIG. The image signal dividing means in the configuration corresponds to the image signal dividing unit 109 and the uniform region extracting unit 111 shown in FIG. The regression means in the configuration corresponds to the color correlation regression unit 113 shown in FIG. The recovery means in the configuration corresponds to the missing pixel restoration unit 114 shown in FIG.

  In a preferred application example of the image processing apparatus of the present invention, a neighborhood area of a predetermined size is extracted by the neighborhood area extraction section 106 with respect to the signal from the process switching section 105 shown in FIG. 1, and the parameter calculation section 107 and the image signal division section 109 are extracted. And the color correlation regression unit 113 and the missing pixel restoration unit 114 recover the missing color signal based on the line format obtained by regressing the color correlation shown in FIG. An image processing apparatus transferred to

(Function)
The image signal is divided in advance into uniform regions where a single color correlation is established, and the missing color signal is restored for each region based on the color correlation.

(effect)
It is possible to provide an image processing apparatus that can restore missing color signals with high accuracy and high speed.

2.
1. The parameter calculation means obtains the gradient of the spectral spectrum from the color signal existing in the neighboring region, and calculates a parameter for region division from the magnitude relationship of the gradient.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the first embodiment shown in FIGS. The parameter calculation means in the configuration corresponds to the neighborhood region extraction unit 106 and parameter calculation unit 107 shown in FIG.

  In a preferred application example of the image processing apparatus of the present invention, a neighborhood area of a predetermined size is extracted by the neighborhood area extraction section 106 with respect to the signal from the process switching section 105 shown in FIG. 1, and the parameter calculation section 107 and the image signal division section 109 are extracted. 4 is an image processing apparatus in which the region is divided based on the gradient of the spectrum in the nearby region shown in FIG.

(Function)
In the restoration by the color correlation, the image signal is divided into uniform areas from the gradient of the spectrum, and the missing color signal is recovered based on the color correlation for each area.

(effect)
It is possible to provide an image processing apparatus capable of reducing false colors generated at edges and color boundary portions without reducing the resolution.

3.
1. The parameter calculation means obtains a luminance signal from the color signal existing in the neighboring area, and calculates a parameter for area division from the edge intensity of the luminance signal.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the second embodiment shown in FIGS. 1, 2, and 9 to 13. The parameter calculation means in the configuration corresponds to the neighborhood region extraction unit 106 and parameter calculation unit 107 shown in FIG.

  A preferred application example of the image processing apparatus of the present invention is that a neighborhood area of a predetermined size is extracted by the neighborhood area extraction section 106 with respect to the signal from the process switching section 105 shown in FIG. This is an image processing device in which region division is performed based on the edge strength of a luminance signal in a neighboring region.

(Function)
In the restoration by the color correlation, the image signal is divided into uniform areas from the edge intensity of the luminance signal, and the missing color signals are restored based on the color correlation for each area.

(effect)
2. This is the same as the effect.

4).
1. The parameter calculation means regresses the color correlation between the color signals existing in the neighboring area as a line format, and calculates a parameter for area division from the constant term of the line format.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the third embodiment shown in FIGS. The parameter calculation means in the configuration corresponds to the color correlation regression unit 315 shown in FIG.

  A preferred application example of the image processing apparatus according to the present invention is that a local region having a predetermined size is extracted by the local region extracting unit 314 with respect to the signal from the processing switching unit 308 shown in FIG. An image processing apparatus in which region division is performed based on a linear constant term of a color correlation in a local region shown.

(Function)
The restoration based on the color correlation is performed by dividing the image signal into uniform regions based on the linear constant terms of the color correlation, and recovering missing color signals based on the color correlation for each region.

(effect)
2. This is the same as the effect.

5).
1. The parameter calculation means regresses the color correlation between the color signals existing in the neighboring region as a linear format, and substitutes the maximum value and the minimum value of the color signal used for the regression of the linear format. A parameter for area division is calculated from the error.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the fourth embodiment shown in FIGS. 14, 15, 19, and 20. The parameter calculation means in the configuration corresponds to the color correlation regression unit 315 shown in FIG.

  In a preferred application example of the image processing apparatus according to the present invention, a local area of a predetermined size is extracted by the local area extracting unit 314 with respect to the signal from the processing switching unit 308 shown in FIG. This is an image processing apparatus in which region segmentation is performed based on an error when a maximum value and a minimum value are substituted into the color correlation line format shown.

(Function)
Restoration by color correlation is performed by dividing the image signal into uniform areas based on the error when the maximum and minimum values are assigned to the color correlation line format, and recovering missing color signals based on the color correlation for each area. To do.

(Effect) 2. This is the same as the effect.

6).
1. And a switching means for switching between the second recovery means for recovering the missing color signal of the image signal captured by the imaging system by linear interpolation, the first recovery means, and the second recovery means. Are further provided.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the first embodiment shown in FIGS. 1 to 8 and the second embodiment shown in FIGS. 1, 2, and 9 to 13. The second recovery means in the configuration corresponds to the linear interpolation unit 116 shown in FIG. The first recovery means in the configuration includes the neighborhood region extraction unit 106, parameter calculation unit 107, image signal division unit 109, uniform region extraction unit 111, color correlation regression unit 113, and missing pixel restoration unit 114 shown in FIG. Applicable. The switching means in the configuration corresponds to the process switching unit 105 shown in FIG.

  In the image processing apparatus of the present invention, the image signal from the input unit 101 shown in FIGS. 1, 2, and 3 is stored in the R signal buffer 102, the G signal buffer 103, and the B signal buffer 104, and the processing is switched. When the process leading to the linear interpolation 116 or the missing pixel restoration unit 114 is selected by the unit 105 and the linear interpolation 116 is selected, the missing color signal is recovered by the linear interpolation and transferred to the output unit 117, and the missing pixel When the process leading to the restoration unit 114 is selected, the color correlation regression unit 113 and the missing pixel restoration unit 114 restore the missing color signal based on the linear format obtained by regressing the color correlation shown in FIG. An image processing apparatus to be transferred.

(Function)
A means for recovering the missing color signal based on the color correlation and a means for recovering the missing color signal based on the linear interpolation are provided, and the two recovery means are switched.

(effect)
An image processing apparatus capable of obtaining an appropriate image quality in an appropriate processing time can be provided.

7).
6). The switching means automatically performs switching based on the number of pixels of the image signal at the time of imaging and the number of pixels required for the output medium, or whether or not the electronic zoom is used.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the first embodiment shown in FIGS. 1 to 8 and the second embodiment shown in FIGS. 1, 2, and 9 to 13. The switching means in the configuration corresponds to the process switching unit 105 shown in FIG.

  In a preferred application example of the image processing apparatus of the present invention, the image signal from the input unit 101 shown in FIGS. 1, 2, and 3 is stored in the R signal buffer 102, the G signal buffer 103, and the B signal buffer 104. In this image processing apparatus, the process switching unit 105 selects the process leading to the linear interpolation 116 or the missing pixel restoration unit 114.

(Function)
Means for recovering the missing color signal based on the color correlation and means for recovering the missing color signal based on the linear interpolation. Switching is performed automatically based on the number of required pixels or the presence / absence of electronic zoom.

  (Effect) It is possible to provide an image processing apparatus capable of obtaining an appropriate image quality in an appropriate processing time by automatic processing.

8).
6). The switching by the switching means is manual.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the first embodiment shown in FIGS. 1 to 8 and the second embodiment shown in FIGS. 1, 2, and 9 to 13. The switching means in the configuration corresponds to the process switching unit 105 shown in FIG.

  In a preferred application example of the image processing apparatus of the present invention, the image signal from the input unit 101 shown in FIGS. 1, 2, and 3 is stored in the R signal buffer 102, the G signal buffer 103, and the B signal buffer 104. In this image processing apparatus, the process switching unit 105 selects the process leading to the linear interpolation 116 or the missing pixel restoration unit 114.

(Function)
A means for recovering the missing color signal based on the color correlation and a means for recovering the missing color signal based on the linear interpolation are provided, and the two recovery means are manually switched.

(effect)
It is possible to provide an image processing apparatus capable of performing signal processing by giving priority to either processing time or image quality based on a user's desire.

9.
An image processing apparatus having a single-plate, two-plate, or three-plate pixel-shift imaging system, and a local region extraction unit that sequentially scans image signals in units of pixels and extracts a local region including the current pixel of interest; A plurality of micro regions within the local region, parameter calculating means for calculating parameters for region segmentation from each micro region, and a single color correlation between the local regions based on the calculated parameters. A local area dividing means for dividing into a uniform area, and a color signal belonging to the same area as the current target pixel is selected based on the uniform area in the local area, and the color correlation between the color signals is expressed in a linear format And selecting a color signal belonging to the same region as the current pixel of interest based on the uniform region within the local region, and selecting the current pixel of interest based on the color signal and the line format The image processing apparatus characterized by comprising: a first recovery means for recovering a color signal, the the missing in the region equal.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the third embodiment shown in FIGS. 14 to 18 and the fourth embodiment shown in FIGS. 14, 15, and 19 to 21. The local area extracting means in the configuration corresponds to the local area extracting unit 314 shown in FIG. The parameter calculation means in the configuration corresponds to the color correlation regression unit 315 shown in FIG. The local area dividing means in the configuration corresponds to the local area dividing unit 317 shown in FIG. The selective regression means in the configuration corresponds to the uniform region extraction unit 319 and the color correlation regression unit 321 shown in FIG. The first recovery means in the configuration corresponds to the missing pixel restoration unit 322 and the addition averaging unit 324 shown in FIG.

  A preferred application example of the image processing apparatus of the present invention is that a local region of a predetermined size is extracted by the local region extracting unit 314 with respect to the signal from the processing switching unit 308 shown in FIG. 14, and the color correlation regression unit 315 and the local region dividing unit are extracted. In 317, the area is divided, and based on the area division, the color correlation regression unit 321 and the missing pixel restoration unit 322 recover the missing color signal based on the line format in which the color correlation is regressed. In this image processing apparatus, the recovered color signals are averaged and transferred to the output unit 325.

(Function)
Local regions are sequentially extracted from the image signal, divided into uniform regions where a single color correlation is established within the local region, and missing color signals in the same uniform region as the current pixel of interest are converted into color correlations. Recover based.

(effect)
It is possible to provide an image processing apparatus capable of restoring missing color signals with high accuracy and at low cost.

10.
9. The parameter calculation means obtains the gradient of the spectral spectrum from the color signal existing in the minute region, and calculates a parameter for region division from the magnitude relationship of the gradient.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the first embodiment shown in FIGS. The parameter calculation means in the configuration corresponds to the neighborhood region extraction unit 106 and parameter calculation unit 107 shown in FIG.

  In a preferred application example of the image processing apparatus of the present invention, a neighborhood area of a predetermined size is extracted by the neighborhood area extraction section 106 with respect to the signal from the process switching section 105 shown in FIG. 1, and the parameter calculation section 107 and the image signal division section 109 are extracted. 4 is an image processing apparatus in which the region is divided based on the gradient of the spectrum in the nearby region shown in FIG.

(Function)
In the restoration by the color correlation, the image signal is divided into uniform areas from the gradient of the spectrum, and the missing color signal is recovered based on the color correlation for each area.

(effect)
It is possible to provide an image processing apparatus capable of reducing false colors generated at edges and color boundary portions without reducing the resolution.

11.
9. The parameter calculation means obtains a luminance signal from the color signal existing in the minute area, and calculates a parameter for area division from the edge intensity of the luminance signal.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the second embodiment shown in FIGS. 1, 2, and 9 to 13. The parameter calculation means in the configuration corresponds to the neighborhood region extraction unit 106 and parameter calculation unit 107 shown in FIG.

  A preferred application example of the image processing apparatus of the present invention is that a neighborhood area of a predetermined size is extracted by the neighborhood area extraction section 106 with respect to the signal from the process switching section 105 shown in FIG. This is an image processing device in which region division is performed based on the edge strength of a luminance signal in a neighboring region.

(Function)
In the restoration by the color correlation, the image signal is divided into uniform areas from the edge intensity of the luminance signal, and the missing color signals are restored based on the color correlation for each area.

(effect)
It is possible to provide an image processing apparatus capable of reducing false colors generated at edges and color boundary portions without reducing the resolution.

12
9. The parameter calculation means regresses the color correlation between the color signals existing in the minute area as a line format, and calculates a parameter for area division from the constant term of the line format.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the third embodiment shown in FIGS. The parameter calculation means in the configuration corresponds to the color correlation regression unit 315 shown in FIG.

  A preferred application example of the image processing apparatus according to the present invention is that a local region having a predetermined size is extracted by the local region extracting unit 314 with respect to the signal from the processing switching unit 308 shown in FIG. The image processing apparatus is configured to divide a region based on a linear constant term of a color correlation in a local region shown.

(Function)
The restoration based on the color correlation is performed by dividing the image signal into uniform regions based on the linear constant terms of the color correlation, and recovering missing color signals based on the color correlation for each region.

(effect)
It is possible to provide an image processing apparatus capable of reducing false colors generated at edges and color boundary portions without reducing the resolution.

13.
9. The parameter calculation means regresses the color correlation between the color signals existing in the minute area as a linear format, and substitutes the maximum value and the minimum value of the color signal used for the regression of the linear format. A parameter for area division is calculated from the error.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the fourth embodiment shown in FIGS. 14, 15, and 19 to 21. The parameter calculation means in the configuration corresponds to the color correlation regression unit 315 shown in FIG.

  In a preferred application example of the image processing apparatus according to the present invention, a local area of a predetermined size is extracted by the local area extracting unit 314 with respect to the signal from the processing switching unit 308 shown in FIG. This is an image processing apparatus in which region segmentation is performed based on an error when a maximum value and a minimum value are substituted into the color correlation line format shown.

(Function)
Restoration by color correlation is performed by dividing the image signal into uniform areas based on the error when the maximum and minimum values are assigned to the color correlation line format, and recovering missing color signals based on the color correlation for each area. To do.

(effect)
It is possible to provide an image processing apparatus capable of reducing false colors generated at edges and color boundary portions without reducing the resolution.

14
9. And a switching means for switching between the second recovery means for recovering the missing color signal of the image signal captured by the imaging system by linear interpolation, the first recovery means, and the second recovery means. Are further provided.

(Corresponding Embodiment of the Invention), (Action), (Effect) It is the same.

15.
14 The switching means automatically performs switching based on the number of pixels of the image signal at the time of imaging and the number of pixels required for the output medium, or whether or not the electronic zoom is used.

  (Corresponding Embodiment of the Invention), (Operation), (Effect) It is the same.

16.
14 The switching by the switching means is manual.

(Corresponding Embodiment of the Invention), (Action), (Effect) It is the same.

17.
An image processing apparatus having a single-plate, two-plate, or three-plate pixel shifting imaging system, and first recovery means for recovering a missing color signal of an image signal captured by the imaging system by linear interpolation, and Conversion means for converting the image signal recovered by the first recovery means into an original image signal obtained by the imaging system, and a color signal lacking the converted image signal based on a color correlation between the color signals An image processing apparatus comprising: a second recovery unit that recovers; and a switching unit that switches between the conversion unit and the second recovery unit.

(Corresponding Embodiment of the Invention)
The embodiment relating to the present invention corresponds at least to the third embodiment shown in FIGS. 14 to 18 and the fourth embodiment shown in FIGS. 14, 15, and 19 to 21. The first recovery means in the configuration corresponds to the linear interpolation unit 305 shown in FIG. The converting means in the configuration corresponds to the converting unit 309 shown in FIG. The second recovery means in the configuration includes a local region extraction unit 314, a color correlation regression unit 315, a local region segmentation unit 317, a uniform region extraction unit 319, a color correlation regression unit 321 and a missing pixel restoration unit 322 shown in FIG. Corresponds to the addition averaging unit 324. The switching means in the configuration corresponds to the process switching unit 308 shown in FIG.

  In a preferred application example of the image processing apparatus of the present invention, the image signal from the input unit 301 shown in FIGS. 14 and 15 is stored in the R signal buffer 302, the G signal buffer 303, and the B signal buffer 304, and is linear. The missing color signal is restored by the interpolation unit 305 and output to the memory card 306. The image signal on the memory card is read by the card reading unit 307, the process switching unit 308 selects the bypass process or the process leading to the addition averaging unit 324, and when the bypass process is selected, the image on the memory card is selected. When the signal is transferred as it is to the output unit 325 and the process leading to the addition averaging unit 324 is selected, a missing color signal is generated based on the linear format obtained by regressing the color correlation by the color correlation regression unit 321 and the missing pixel restoration unit 322. The image processing apparatus is recovered and transferred to the output unit 325.

(Function)
Means for recovering missing color signals based on linear interpolation, and converting the recovered color signals into original image signals based on information of the imaging system, and then recovering missing color signals based on color correlation from the image signals The latter recovery means can be bypassed.

(effect)
It is possible to provide an image processing apparatus that can restore a color signal with high accuracy even for a signal that has been subjected to processing such as linear interpolation.

18.
17. The switching means automatically performs switching based on the number of pixels of the image signal at the time of imaging and the number of pixels required for the output medium, or whether or not the electronic zoom is used.

(Corresponding Embodiment of the Invention), (Operation), (Effect) It is the same.

19. The switching by the switching means is manual.

(Corresponding Embodiment of the Invention), (Action), (Effect) It is the same.

20. Image signals obtained by single-plate, two-plate, or three-plate pixel shifting imaging systems are scanned sequentially in units of pixels, and parameters for segmentation are calculated from at least one neighboring region that includes the current pixel of interest. Parameter calculation processing, image signal division processing for dividing the image signal into uniform areas based on the calculated parameters, and color correlation between color signals existing in the uniform area A program including instructions for causing a computer to execute a regression process for regressing the relationship as a linear format and a recovery process for recovering a missing color signal based on the color signal existing in the uniform area and the linear format is stored. A computer-readable recording medium.

(Corresponding Embodiment of the Invention), (Action), (Effect) are as follows. It is the same.

21. A local region extraction process that sequentially scans the image signal obtained by imaging with a single-plate, two-plate, or three-plate pixel-shift imaging system in units of pixels and extracts a local region including the current pixel of interest, and the extracted local A single color correlation is established for the local region based on the parameter calculation processing for setting a plurality of micro regions in the region and calculating parameters for region division from each micro region, and the calculated parameters. Local area division processing to divide into uniform areas, and select a color signal belonging to the same area as the current pixel of interest based on the uniform area within the local area, and return the color correlation between the color signals as a linear format And selecting a color signal belonging to the same region as the current pixel of interest based on the uniform region within the local region, and an area equivalent to the current pixel of interest based on the color signal and the line format. Missing for selection recovery process to recover the color signals, storing a program including instructions for causing a computer to execute the computer-readable recording medium in.

(Corresponding Embodiment of the Invention), (Action), (Effect) It is the same.

22. A first recovery process for recovering a missing color signal of an image signal obtained by imaging with a single-plate, two-plate, or three-plate pixel shifting imaging system by linear interpolation, and the first recovery process A conversion process for converting an image signal into an original image signal obtained by the imaging system; a second recovery process for recovering a color signal lacking in the converted image signal based on a color correlation between the color signals; A computer-readable recording medium storing a program including instructions for causing a computer to execute a switching process for switching between the conversion process and the second recovery process.

(Corresponding Embodiment of the Invention), (Action), (Effect) are 17. It is the same.

It is a block diagram of 1st Embodiment of this invention. It is explanatory drawing of a single plate type CCD input part. It is a filter arrangement | positioning figure in 1st Embodiment of this invention. It is explanatory drawing of the area division based on a spectrum gradient. It is a figure which shows the correspondence of the gradient of a spectrum, and a class. It is explanatory drawing of a return to the linear form of color correlation. It is a flowchart (the 1) for demonstrating the effect | action of 1st Embodiment of this invention. It is a flowchart (the 2) for demonstrating the effect | action of 1st Embodiment of this invention. It is a filter arrangement | positioning figure in 2nd Embodiment of this invention. It is explanatory drawing of the area division based on a luminance signal. It is a flowchart (the 1) for demonstrating the effect | action of 2nd Embodiment of this invention. It is a flowchart (the 2) for demonstrating the effect | action of 2nd Embodiment of this invention. It is explanatory drawing of the luminance signal calculation of a primary color filter. It is a block diagram of 3rd Embodiment of this invention. It is explanatory drawing of a two-plate CCD input part. It is explanatory drawing of the area | region division based on a constant term. It is a flowchart (the 1) for demonstrating the effect | action of 3rd Embodiment of this invention. It is a flowchart (the 2) for demonstrating the effect | action of 3rd Embodiment of this invention. It is explanatory drawing of verification of the line format by the maximum value and the minimum value. It is a flowchart (the 1) for demonstrating the effect | action of 4th Embodiment of this invention. It is a flowchart (the 2) for demonstrating the effect | action of 4th Embodiment of this invention. It is a figure which shows the structural example of an electronic still camera system. It is explanatory drawing of the filter arrangement | positioning of a single plate-type image sensor.

Explanation of symbols

101 ... input section,
102 ... R signal buffer,
103 ... G signal buffer,
104... B signal buffer,
105 ... process switching part,
106... Neighborhood region extraction unit,
107 ... parameter calculation unit,
108: Buffer for parameters,
109 ... Image signal dividing unit,
110: Buffer for divided images,
111 ... uniform region extraction unit,
112 ... Uniform area buffer,
113 ... Color correlation regression unit,
114... Missing pixel restoration unit,
115 ... Restored image buffer,
116: linear interpolation unit,
117: Output unit.

Claims (7)

An image processing apparatus that processes an image signal obtained via a single-plate, two-plate, or three-plate pixel shifting imaging system,
First recovery means for recovering the missing color signal of the image signal by linear interpolation;
Converting means for converting the image signal restored by the first restoring means to the original image signal obtained by the imaging system,
A second recovery means for recovering for based the missing color signal of the converted image signal to the color correlation between the color signals,
An image processing apparatus comprising: an output unit that performs a conversion output process for outputting the image signal recovered by the second recovery unit . The image processing apparatus according to claim 1 , wherein the output unit performs a direct output process in which the image signal recovered by the first recovery unit is output as it is instead of the conversion output process . The image processing apparatus according to claim 2 , wherein the output unit performs the direct output processing by bypassing processing by the conversion unit and the second recovery unit . The image processing apparatus according to claim 2, further comprising a switching unit that switches between the conversion output process and the direct output process. 5. The image processing apparatus according to claim 4 , wherein the switching unit automatically performs switching based on the number of pixels of the image signal at the time of photographing and the number of pixels required for the output medium, or whether or not the electronic zoom is used. . The image processing apparatus according to claim 4 , wherein the switching by the switching unit is performed manually. A first recovery process for recovering a missing color signal of an image signal obtained through a single-plate, two-plate, or three-plate pixel-shift imaging system by linear interpolation;
And conversion processing for converting the image signal restored by the first restoration processing to the original image signal obtained by the imaging system,
A second recovery process for recovering a missing color signal of the converted image signal based on a color correlation between the color signals;
A switching process for switching between the conversion processing and the second recovery process,
The computer-readable recording medium which stored the program containing the instruction | indication which makes a computer perform.

Images