JP2011223221A - Photographing device and image integration program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a high dynamic range even when using an ordinary imaging element for video production.SOLUTION: A photographing device 1 includes a single-plate RGB imaging element 7 and a monochrome imaging element 8, which photograph a subject through a half-mirror 6, and an arithmetic processing unit 3. The arithmetic processing unit 3 includes: brightness calibration means 20 for demosaicking a color photographed image 7a of two photographed images 7a and 8a that are obtained by photographing a color chart in advance by the single-plate RGB imaging element 7 and monochrome imaging element 8, generating a brightness value Yof the color photographed image 7a through a 3×1 color brightness conversion matrix K from an average of RGB values of full resolution, and optimizing the color brightness conversion matrix K so that the brightness value Yand a brightness value Yof the monochrome photographed image 8a match each other; and integrated image generating means 30 for generating an integrated image Iin which color information of the image photographed by the single-plate RGB imaging element 7 is integrated based upon the optimized color brightness conversion matrix K when the subject Ois photographed.

Description

  The present invention relates to an imaging technique and an image processing technique, and more particularly to a video processing technique used for video production.

  In the production of video works used in broadcasting, etc., the degree of freedom of video processing required for post-production (also referred to as post-production) performed after shooting a predetermined subject should be sufficient to accommodate the creator's imagination. ing.

  For example, if there is a photographing apparatus capable of acquiring a real image with high accuracy based on video processing represented by video composition or the like and it can be used, the degree of freedom in video processing can be improved. However, such an ideal photographing apparatus is not used for image processing because of cost and difficulty in obtaining. On the other hand, an image pickup device (image pickup element) used in a normal photographing apparatus is generally designed and manufactured based on the vicinity of quality limited by the broadcasting or communication method of that era. Therefore, it is realistic to use such a normal imaging device also from the viewpoint of cost or availability.

  However, the accuracy of broadcasting equipment does not satisfy the imagination of creators. At present, for example, broadcasting equipment with a degree of accuracy specified by, for example, a broadcasting standard, and broadcasting equipment with a slightly higher precision than such a precision are generally mixed and used. Therefore, when video processing represented by video synthesis or the like is performed using such a general broadcasting device, the image may be deteriorated or uncomfortable. For example, when an image is enlarged by image processing in processing after photographing, color blur occurs and sharpness is impaired.

  Therefore, as an imaging apparatus capable of improving the degree of freedom of image processing, for example, a device that can acquire a wider dynamic range and is low in cost with respect to the image format of the final image work is desired. However, in terms of cost, the degree of freedom in video processing is a major factor that affects the value of video works, so there is also a need to obtain a more effective effect with a slight increase in costs.

  In the future, it is expected that video formats with higher resolution than the current situation will become widespread. In this case, the production side of the video format at the same time also supports video processing of the high resolution video format. In other words, when the video is processed on the production side, it is expected that a video material having a higher resolution and a higher dynamic range is required for the video format.

  Conventionally, devices for photographing a high dynamic range include a device that is devised for a single element, a device that devises a photographing method, and the like. Among these, devices that have been devised for a single device include those that have some nonlinear characteristics with respect to the relationship between incident light and the amount of charge accumulated in the device. However, in the present situation, there is no high-resolution element that gives the element some kind of non-linear characteristics due to the limitation of the degree of integration.

Further, among devices that shoot a high dynamic range, a device that devises a shooting method includes a method of integrating images shot multiple times (see Non-Patent Documents 1 and 2).
The method described in Non-Patent Document 1 is a method of acquiring a high dynamic range image using a plurality of images acquired by changing the exposure time by changing the shutter speed. Further, the method described in Non-Patent Document 2 is a method of integrating a plurality of images acquired by changing the exposure time into a high dynamic range image.

Keita Saito and two others, “Examination of continuity correction and spatial correction methods in wide dynamic range compression”, Journal of the Institute of Image Information and Television Engineers, Vol.61, No.3, p.325-331, 2007 Paul E. Debevec and Jitendra Malik, “Recovering High Dynamic Range Radiance Maps from Photographs”, Proceedings of the SIGGRAPH’97 Conference ”, p.369-378, August 1997

  However, among conventional apparatuses that capture a high dynamic range, an image capturing apparatus that integrates images that have been captured multiple times captures images at different times, making it difficult to handle moving images. Or even if it corresponds to a moving image, since the unnatural outline arises in the outline of the fast-moving thing, it is not suitable for image production. For example, when the technique of Non-Patent Document 1 is applied to a moving image, there is a problem that motion blur becomes unnatural because the integrated high dynamic range image has an unexposed time. In addition, the method of Non-Patent Document 2 has difficulty in dealing with moving images, and has a complication such as brightness calibration in post-processing.

  The present invention has been made in view of the above problems, and it is an object of the present invention to provide an imaging apparatus and an image integration program that obtain a high dynamic range even when a normal imaging element is used in video production. .

  In order to solve the above-described problem, an imaging apparatus according to claim 1 of the present invention includes one or more single-plate color imaging devices having RGB color filters and different spectral characteristics, and a monochrome imaging device having no color filter. Are arranged in a positional relationship in which the same photographing area is photographed with the same viewpoint and the same angle of view through the spectroscopic means, and each photographed image photographed using the color imaging device and the monochrome imaging device is 1 And an image integration unit that performs an integration process. The image integration unit includes a luminance calibration unit and an integrated image generation unit.

  According to such a configuration, the photographing apparatus photographs one subject in the same manner with one or more color imaging devices and a monochrome imaging device via a spectral unit such as a prism or a half mirror. Here, the color imaging device is composed of, for example, a single-plate RGB imaging device having an on-chip color filter such as a Bayer array, and the monochrome imaging device is, for example, a monochrome imaging device on which only a clear on-chip lens is mounted. Consists of In addition, the image integration unit of the photographing apparatus uses the luminance calibration unit to color the RGB values of the RGB plane generated by the demosaic that interpolates the RGB pixels of the color image obtained by photographing the color test photographing target with the color imaging device. The luminance value of the pixel in the color image is generated by conversion using the luminance conversion matrix, and the luminance value of the pixel in the monochrome photographic image obtained by photographing the color test photographing target with the monochrome imaging device simultaneously with the color imaging device The color luminance conversion matrix is obtained by optimizing so that the difference from the luminance value of the pixel in the color image is minimized. Here, for example, the process of generating the luminance value of the color image may be performed while photographing. Then, the image integration unit of the photographing apparatus performs the RGB image RGB value generated by demosaicing from the color image of the photographed image obtained by photographing the photographing object by the color imaging device and the monochrome imaging device by the integrated image generating unit. Among the luminance values converted by multiplying the optimized color luminance conversion matrix by the color space conversion matrix that maps to the color space, the luminance value of the region that is not saturated in the monochrome captured image In addition, the integrated luminance and the color information in the color space are obtained by selecting a luminance value in a region that is saturated in the monochrome captured image and in a region that is not saturated in the converted luminance value. This is the color information of the integrated image obtained by integrating the captured images into one, and the integrated luminance is the luminance information. To generate a focus image.

  In addition, since the photographing apparatus includes a color imaging device and a monochrome imaging device, one image can be obtained while interpolating information that can be obtained on one of them, but not on the other. Can do. For example, the resolution of a captured image obtained by a color imaging device having a Bayer array can be interpolated from the captured image obtained by a monochrome imaging device. Moreover, color information can be interpolated from the color imaging device for the image captured by the monochrome imaging device. Also, if one imaging device is saturated from the difference in the amount of incident light to each imaging device, information such as the luminance value of that part can be interpolated from the other imaging device. In this way, the imaging apparatus uses a plurality of relatively low-cost general imaging devices available at that time, and combines both images with high accuracy while compensating for the advantages and disadvantages of both. Therefore, it is possible to acquire a highly accurate video with a higher dynamic range and higher resolution than before. Furthermore, it is possible to secure the degree of freedom in the processing work that occurs in the production process of the video work while suppressing the cost increase to a degree that the number of general imaging devices is slightly increased.

  Further, according to this configuration, in the photographing apparatus, the integrated image generating unit multiplies the RGB value of the RGB plane obtained by demosaicing the photographed color image by the color space conversion matrix. That is, the integrated image generation unit uses color information obtained by demosaic processing of a color image as color information of each captured image captured by a color imaging device and a monochrome imaging device. Here, the color space conversion matrix temporarily maps the RGB values of the pixels of the color image into a special color space. The color space conversion matrix is a 3 × 3 matrix, and the color luminance conversion matrix is a matrix having 3 × 1 elements because it converts RGB values into luminance. The color space conversion matrix includes a color luminance conversion matrix in order to handle the integrated luminance value and the color information at the same time. The integrated luminance and color information in the color space are obtained by the color space conversion matrix. In this way, the integrated image obtained by the photographing apparatus is subjected to video processing in the post-production.

  According to a second aspect of the present invention, there is provided the photographing apparatus according to the first aspect, wherein the luminance calibration means uses a color chart as the color test photographing target for the color imaging device and the monochrome imaging device. The color luminance conversion matrix is optimized using the color chart image photographed in Step 1, and the optimized color luminance conversion matrix is held in the storage means.

  According to this horizontal composition, when the photographing apparatus is used many times under similar conditions, for example, in an indoor studio, the luminance of each imaging device is once calibrated using a color chart before the actual photographing. Then, after that, it is not necessary to calibrate the brightness every time the image is taken. If the brightness of each imaging device is calibrated, a highly dynamic and high resolution integrated image under the same conditions can be acquired each time the subject is photographed with a color imaging device and a monochrome imaging device. .

  According to a third aspect of the present invention, there is provided the photographing apparatus according to the first aspect, wherein the luminance calibration means sets the color test photographing object updated according to an external environment as the color test photographing object. The color luminance conversion matrix is optimized using each captured image captured by the imaging device and the monochrome imaging device, and the optimized color luminance conversion matrix is updated in accordance with the color test imaging target while being stored in the storage unit. I decided to keep it.

  According to this horizontal composition, for example, when a picture to be photographed has changed, when moving to a place with different illumination conditions indoors, when used with movement from indoor to outdoor, When using the camera, each shooting is performed using the normal shooting video shot with each imaging device before the actual shooting, depending on the external environment that changes from time to time due to differences in lighting conditions, weather, etc. By calibrating the luminance of the image each time, the optimized color luminance conversion matrix can be updated. Here, for example, the color luminance conversion matrix may be updated in units of frames. According to this photographing apparatus, when a subject is photographed with a color imaging device and a monochrome imaging device, a high dynamic and high resolution integrated image corresponding to each photographing environment can be acquired each time.

  According to a fourth aspect of the present invention, in the photographing apparatus according to any one of the first to third aspects, the brightness calibration means is the color imaging device and the monochrome imaging device. In order to minimize the difference between the luminance value of the pixel in the monochrome image and the luminance value of the pixel in the color image with respect to the remaining pixels by thinning out the pixels from each captured image according to a predetermined rule. Then, the color luminance conversion matrix is optimized.

  According to this horizontal composition, the image capturing apparatus optimizes the color luminance conversion matrix using each partial image without using all the pixels of the image captured by each image capturing device. Can be done. In particular, for example, when the color luminance conversion matrix is updated in units of frames, the calculation cost can be greatly reduced. Here, the rule for thinning out the pixels is not particularly limited as long as the positions of the pixels of the image captured by each imaging device are the same. For example, sampling is performed every other pixel or pixels are randomly selected. Can be.

  According to a fifth aspect of the present invention, in the photographing apparatus according to any one of the first to fourth aspects, the image integration unit further includes a position calibration unit. .

  According to this horizontal composition, in the photographing apparatus, the position calibration means of the image integration means is a color photographed by the color imaging device based on a plurality of photographed images obtained by photographing the position test photographing object with the respective imaging devices. By converting the coordinates of the feature points of the image by a projection matrix, a converted coordinate value converted into the coordinates in the monochrome image is generated, and the position test imaging target is imaged by the monochrome imaging device simultaneously with the color imaging device. The projection matrix is optimized and determined so that the difference between the coordinates of the feature point of the monochrome photographed image and the converted coordinate value in the monochrome photograph image corresponding to the feature point of the color image is minimized. Actually, depending on the installation accuracy of each imaging device with respect to the spectroscopic means such as a prism or a half mirror, the pixel sampling position may differ. According to such horizontal composition, in the photographing apparatus, the position calibration unit can also calibrate a minute positional deviation in pixel units that cannot be calibrated even by mechanical alignment by image processing. Therefore, the photographing apparatus can acquire a highly dynamic and high resolution integrated image with high accuracy.

  According to a sixth aspect of the present invention, in the photographing apparatus according to any one of the first to fifth aspects, a plurality of the color imaging devices are disposed, and each of the color imaging devices is A second color image other than the first color image used when the luminance calibration unit obtains the color luminance conversion matrix among the respective captured images of the color imaging device. From the RGB color value of the RGB plane generated by demosaicing similarly to the first color image, the luminance value of the second color image is generated by converting the RGB value by the second color luminance conversion matrix, and the second color image The second color luminance conversion matrix is obtained by optimizing the second color luminance conversion matrix so as to minimize the difference between the luminance value of the first color captured image and the luminance value of the first color captured image that has already been optimized. Image generation means is a matrix that includes the optimized second color luminance conversion matrix for the RGB values of the RGB plane generated by demosaicing from the second color image, and that maps to the color space. Among the luminance values converted by multiplying by the color space conversion matrix, the luminance value of the region that is not saturated in the first color image and the region that is saturated in the first color image are converted in the luminance value. By selecting a luminance value that is in a region that is not saturated among the luminance values, the integrated luminance and the color information in the color space are obtained, and this is obtained as a second integrated image obtained by integrating the captured images into one. The second integrated image is further generated using the color information and the integrated luminance as the luminance information.

  According to this horizontal composition, since the photographing apparatus includes a plurality of color imaging devices having color filters having different spectral sensitivities, one color is determined from the difference in the amount of incident light to each of the plurality of color imaging devices. If only the imaging device is saturated, information such as the luminance value of the part can be interpolated from other color imaging devices. Therefore, the photographing apparatus can acquire a higher dynamic and higher resolution integrated image with high accuracy.

  According to a seventh aspect of the present invention, in the photographing apparatus according to any one of the first to fifth aspects, the integrated image generation means is the color imaging device and the monochrome imaging device. When generating an integrated image obtained by integrating photographed images obtained by photographing the photographing objects, processing according to a luminance range of a color image obtained by photographing the photographing object with the color imaging device is performed.

  According to this horizontal composition, in the photographing apparatus, the integrated image generating means uses a first threshold that is predetermined as an upper limit of the luminance of the monochrome imaging device in a color image obtained by photographing the photographing object with the color imaging device. For pixels in a large luminance area, the RGB values of the RGB plane generated by demosaicing from the color image are obtained. Here, the first threshold value is, for example, a luminance value when the monochrome imaging device is saturated.

  Further, according to this horizontal composition, in the photographing apparatus, the integrated image generation unit is configured to perform the color photographing on the photographing target for pixels in a luminance area smaller than the second threshold that is smaller than the first threshold. The color image captured by the device is demosaiced again, and the RGB value of the RGB plane generated after interpolation using the gradient of the luminance value of the monochrome captured image captured by the monochrome imaging device is calculated in the demosaic. . Here, the second threshold value is, for example, about 90% of the first threshold value. If the demosaic is performed on the pixels in the luminance region smaller than the second threshold by the same method as the conventional method, it is difficult to ensure the quality because of insufficient sampling. However, according to such a configuration, since the integrated image generation unit uses the gradient of the luminance value of the monochrome photographed image, sufficient sampling is possible. As a result, it is possible to prevent color blurring and a decrease in fineness and to ensure quality.

Further, according to this horizontal composition, in the photographing apparatus, the integrated image generation unit has already calculated the pixels in the luminance region that are greater than the second threshold and the pixels in the luminance region that are greater than or equal to the first threshold. The calculated RGB values and the RGB values already calculated for the pixels in the luminance region smaller than the second threshold value are weighted and averaged according to the distance between the luminance of the pixel and the first and second threshold values. The integrated image generation means calculates the RGB value of the integrated image by the above-described processes corresponding to the luminance of each pixel.
Then, the integrated image obtained by the integrated image generating means performing processing according to the luminance range of the photographed color image is subjected to video processing in the post-production. For example, even if an image is enlarged by image processing in order to synthesize a video image of a live-action and CG, color blur hardly occurs and sharpness is not impaired.

  According to an eighth aspect of the present invention, in the photographing apparatus according to the sixth aspect, the integrated image generating unit photographs the photographing object with the plurality of color imaging devices and monochrome imaging devices, respectively. When an integrated image obtained by integrating the photographed images is generated, processing according to the luminance range of the second color image obtained by photographing the photographing object is performed.

  According to this horizontal composition, in the photographing apparatus, the integrated image generating means has a luminance greater than a third threshold value that is predetermined as an upper limit of the luminance of the first color image in the second color image obtained by photographing the photographing object. For the pixels in the region, the RGB value of the RGB plane generated by demosaicing from the second color image is obtained. Here, the third threshold value is, for example, a luminance value when the color imaging device that captured the first color image is saturated with the converted luminance.

  Further, according to this horizontal composition, in the photographing apparatus, the integrated image generation unit re-demosaics the second color image again for pixels in a luminance area smaller than the fourth threshold value that is smaller than the third threshold value. In this demosaic, the RGB value of the RGB plane generated after interpolation using the gradient of the luminance value of the integrated image is calculated. Here, the fourth threshold value is, for example, about 90% of the third threshold value.

Further, according to this horizontal composition, in the photographing apparatus, the integrated image generation unit has already calculated the pixels in the luminance region that is greater than or equal to the fourth threshold and the luminance region that is greater than or equal to the third threshold. The calculated RGB values and the RGB values already calculated for the pixels in the luminance region smaller than the fourth threshold value are weighted and averaged according to the distance between the luminance of the pixel and the third and fourth threshold values. The integrated image generation means calculates the RGB value of the integrated image by the above-described processes corresponding to the luminance of each pixel.
The integrated image obtained by the integrated image generating means performing processing according to the luminance range of the captured second color image is subjected to video processing in the post-production.

  According to a ninth aspect of the present invention, there is provided an image integration program in which one or more single-plate color imaging devices having different color characteristics having RGB color filters and a monochrome imaging device having no color filter have spectral means. In order to perform a process of integrating the captured images captured using the imaging units arranged in a positional relationship in which the same imaging area is captured with the same viewpoint and the same angle of view, a luminance calibration is performed. And a program for functioning as integrated image generation means.

  According to this configuration, the image integration program uses the luminance calibration means to calculate the RGB values of the RGB plane generated by the demosaic that interpolates the RGB pixels of the color image obtained by photographing the color test photographing target with the color imaging device. The luminance value of the pixel in the color image is generated by conversion using the color luminance conversion matrix, and the luminance value of the pixel in the monochrome photographic image obtained by photographing the color test photographing target with the monochrome imaging device simultaneously with the color imaging device The color luminance conversion matrix is obtained by optimizing so that a difference from the luminance value of the pixel in the color image is minimized. Then, the image integration program uses the integrated image generation means for the RGB values of the RGB plane generated by demosaicing from the color images of the captured images obtained by capturing the imaging objects with the color imaging device and the monochrome imaging device, respectively. Among luminance values converted by multiplying a color space conversion matrix that is an optimized color luminance conversion matrix and mapped to a color space, luminance values of regions that are not saturated in the monochrome captured image, and By obtaining a luminance value in a region that is saturated in the monochrome captured image and in a region that is not saturated in the converted luminance value, an integrated luminance and color information in the color space are obtained, and this is obtained. The color information of the integrated image is obtained by integrating the captured images into one, and the integrated luminance is integrated as the luminance information. To generate the image.

  According to the invention described in claim 1 or claim 9, the image capturing apparatus or the image integration program compensates the advantages and disadvantages of both images captured by using a plurality of general image capturing devices. Are combined into a single image with high accuracy, so that a highly accurate image with a higher dynamic range and higher resolution than before can be obtained at low cost. Accordingly, the degree of freedom of video processing can be expanded in video production.

  According to the second aspect of the present invention, once the luminance of each imaging device is calibrated, the imaging apparatus can perform high dynamic and high performance under the same conditions each time the subject is captured by each imaging device. An integrated image of resolution can be acquired.

  According to the third aspect of the present invention, the photographing apparatus can acquire a high dynamic and high resolution integrated image corresponding to each photographing environment each time the subject is photographed by each imaging device. .

  According to the invention described in claim 4, the image capturing apparatus optimizes the color luminance conversion matrix using each partial image without using all the pixels of the image captured by each image capturing device. Can be performed at high speed.

  According to the fifth aspect of the present invention, since the imaging apparatus associates pixels between captured images captured by each imaging device, it is possible to acquire a high dynamic and high resolution integrated image with high accuracy.

  According to the invention described in claim 6, even when only one color imaging device is saturated, the imaging apparatus can interpolate information such as the luminance value of the part from other color imaging devices. Highly dynamic and high resolution integrated images can be acquired with high accuracy

  According to the seventh aspect of the present invention, when the image capturing apparatus generates an integrated image in which color information of captured images captured by the respective image capturing devices is integrated, a pixel with low luminance is subjected to color image demosaicing. Since the gradient of the luminance value of the monochrome photographed image is used, sufficient sampling is possible. As a result, it is possible to prevent color blurring and a reduction in fineness, and to acquire an integrated image that ensures quality.

  According to the eighth aspect of the present invention, when the photographing apparatus generates an integrated image obtained by integrating the color information of the photographed images photographed by the respective color imaging devices, the second color image is obtained with respect to pixels having low luminance. In the demosaic, the gradient of the luminance value of the integrated image is used, so that sufficient sampling is possible.

1A and 1B are configuration diagrams of an imaging apparatus according to an embodiment of the present invention, where FIG. 1A is a block diagram illustrating an overall configuration, and FIG. 1B illustrates a configuration example of an imaging unit. It is a block diagram which concerns on the brightness | luminance integration of the image integration part of the imaging device which concerns on embodiment of this invention. It is a block diagram which concerns on pixel position integration of the image integration part of the imaging device which concerns on embodiment of this invention. It is a block diagram which concerns on integration of the color information of the image integration part of the imaging device which concerns on embodiment of this invention. It is explanatory drawing which shows the demosaic process performed with the brightness | luminance calibration means shown in FIG. It is explanatory drawing of the brightness | luminance integration process performed by the brightness | luminance calibration means shown in FIG. It is explanatory drawing which shows the specific example of the brightness | luminance integration process using a color chart, Comprising: (a) is a picked-up image of a monochrome image sensor, (b) is a picked-up image of a single-plate RGB image sensor, (c) is by optimization The graph which shows a brightness | luminance matching result is shown, respectively. It is a block diagram which shows the structural example of the image integration part of the imaging device which concerns on embodiment of this invention. It is explanatory drawing which shows the demosaic process performed by the 2nd demosaicing process means shown in FIG. It is explanatory drawing of the image integration process performed by the image integration part shown in FIG. It is explanatory drawing which shows the flow of the image processing of the imaging device which concerns on embodiment of this invention.

  An embodiment for carrying out an imaging apparatus of the present invention will be described in detail with reference to the drawings. Below, for convenience of explanation, 1. 1. Outline and configuration of photographing apparatus 2. Outline and configuration of image integration unit 3. Detailed configuration of the image integration unit 4. luminance integration processing; 5. Color information integration process Each chapter of the flow of operation of the photographing apparatus will be described sequentially.

(1. Overview and configuration of the imaging device)
1A and 1B are configuration diagrams of an imaging apparatus according to an embodiment of the present invention, where FIG. 1A is a block diagram illustrating an overall configuration, and FIG. 1B illustrates a configuration example of an imaging unit.
The photographing apparatus 1 includes an optical system and an image sensor, a computing device such as a GPU (Graphics Processing Unit) used in an FPGA (Field Programmable Gate Array) or a PC (Personal Computer), and a storage such as a memory or a hard disk. The computer includes a computer and an interface device that transmits and receives various types of information to and from the outside, and a program installed in the computer. That is, the photographing apparatus 1 is a camera system that is realized by the hardware device and software cooperating to control the above hardware resources by a program. As shown in FIG. Unit 2, arithmetic processing unit 3, storage unit 4, and post-production storage unit 5. The photographing apparatus 1 has a function of calibrating and integrating a plurality of image sensors.

The imaging unit 2 includes a single-plate RGB imaging device (color imaging device) 7 and a monochrome imaging device as a plurality of imaging devices arranged so that the same imaging area can be imaged with the same viewpoint and the same angle of view by a coaxial optical system. (Monochrome imaging device) 8.
The single-plate RGB imaging device 7 is composed of, for example, a general CCD (Charge Coupled Device) imaging device, and has RGB on-chip color filters by a Bayer array or the like. The monochrome image pickup device 8 is composed of, for example, a general CCD image pickup device, has no color filter, and is mounted with only a clear on-chip lens.

  Each imaging device is installed so as to have a conjugate positional relationship with respect to one lens L via a half mirror (spectral means) 6. Here, the conjugate means that the same imaging area is arranged so that it can be taken with the same viewpoint and the same angle of view by the coaxial optical system. In addition, when the size and the number of pixels of each imaging device are the same, the distance from the half mirror (spectral means) 6 becomes equal. In addition, the distance from the half mirror (spectral means) 6 is adjusted so that the same subject can be displayed in the same manner even if the size and the number of pixels of each imaging device are different.

  In addition, in FIG.1 (b), while showing the side view of an optical system, the entrance plane of the single-plate RGB image pick-up element 7 and the monochrome image pick-up element 8 was illustrated with the pixel of 4x4, but actually one side is It is about several thousand pixels, and the vertical and horizontal directions are not necessarily the same. In the single-plate RGB imaging device 7, R (red) is indicated by hatching, G (green) is indicated by dots, and B (blue) is indicated by horizontal stripes.

In the following description, it is assumed that the imaging unit 2 includes a total of two imaging devices, one color imaging device and one monochrome imaging device, as shown in FIG.
However, a plurality of color imaging devices having different spectral characteristics and a single monochrome imaging device may be provided.
Regardless of the number of imaging devices, instead of using the half mirror 6 as the spectroscopic means, the incident light may be separated by a prism that splits the visible light to about 1: 1.
Furthermore, it is not limited to the use of a half mirror or prism with a light distribution ratio of 1: 1, and the color image pickup device side is reduced by setting the light distribution ratio to 2: 1, 10: 1 or the like. It is good also as allocating. Thus, when the light distribution ratio is set to 2: 1, 10: 1, or the like, the apparent dynamic range can be expanded.

Further, depending on the shooting scene, the spectrum may be asymmetrically distributed instead of 1: 1. In this case, since the monochrome image sensor 8 side receives more light than the single-plate RGB image sensor 7, color information is less likely to be saturated even for a bright subject. It is desirable to do so. This ratio should be considered in the final incident light quantity to each pixel of each imaging device. This is because there is an influence of the attenuation amount of the on-chip color filter on the single-plate RGB imaging device 7 side.
The quarter wavelength plate B is not necessary when the half mirror 6 does not have polarization characteristics, but when the half mirror 6 has polarization characteristics, the quarter wavelength plate B is disposed before or behind the lens L. A wave plate B is installed. In FIG. 1B, as an example, a quarter-wave plate B is installed in front of the lens L.

The arithmetic processing unit 3 is realized by, for example, one GPU, and includes an image integration unit 11 and a preview processing unit 12 as illustrated in FIG.
The image integration unit (image integration unit) 11 combines each captured video imaged using the single-plate RGB imaging device 7 and the monochrome imaging device 8 based on the image integration program stored in the storage unit. The process to be integrated is performed.

Image integration unit 11, before generating the integrated image I _t which integrates each captured image I _o the subject of the shot for production into one with each imaging device, previously the subject of testing experimental photographed The color luminance conversion matrix is optimized using the luminance of each captured image. The brightness obtained by this optimization is held as an integrated brightness value Y _T. Integrated image I _t is a processing result of the image integration unit 11 is accumulated in a recording medium such as a hard disk of the storage unit 4 as image information for image processing. Details of the image integration unit 11 will be described later.

Preview processing section 12 has a function of displaying the photographed integrated image I _t and (preview function), a video processing function to store the post-production for the storage unit 5 is subjected to processing in the integrated image I _t.

The preview function is a process that can be performed by a normal camera. For example, the preview function is a function that performs plenary processing, gamma correction processing, knee processing, white clip processing, and edge enhancement processing. Thus, at photographers and director shooting, you can see the integrated image I _t which is displayed on the monitor 13 in real time. The monitor 13 includes, for example, a CRT (Cathode Ray Tube), a liquid crystal display (LCD), an EL (Electronic Luminescence), and the like.

  The video processing function is a function that performs, for example, non-linear tone mapping processing, unlike processing that can be performed by a normal camera. The non-linear tone mapping is for corresponding to the dynamic range of the monitor (display device) 13. Normally, when there is no device that displays an image with a wide dynamic range as it is, it is necessary to adjust the dynamic range (tone mapping) according to the display device. Various non-linear tone mapping processes have been proposed in the past, and this embodiment does not limit the method. As an example, a logarithmic average value of image luminance can be obtained, and curve correction can be performed such that a bright portion is compressed and a dark portion is extended based on this value.

The storage unit 4 includes a storage device such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and a hard disk. For example, the captured image I _O , the integrated luminance value Y _T, and the integrated image I _T. I decided to remember.
The captured image I _O indicates a captured image captured using each imaging device.
The integrated luminance value Y _T is a luminance value calculated by the image integration unit 11 and indicates a luminance value obtained by integrating the luminance of each captured image to be tested.
Integrated image I _T show images also integrated color information in addition to the luminance image integration unit 11.

Post-production for the storage unit 5 is for storing the image subjected to processing in the integrated image I _t at the preview process unit 12, for example, and the like typical hard disk.

(2. Outline and configuration of image integration unit)
Details of the image integration unit 11 of the photographing apparatus 1 shown in FIG. 1 will be described with reference to FIG. 2 shows only the image integration unit 11 of the photographing apparatus 1 shown in FIG. The process in which the image integration unit 11 integrates images from a plurality of imaging devices can be mainly divided into a calibration stage and a synthesis stage (integration stage).

  As shown in FIG. 2, the image integration unit 11 includes a luminance calibration unit 20 that mainly performs image processing at the calibration stage, and an integrated image generation unit 30 that performs image processing at the integration stage.

As a premise, in the photographing apparatus 1, a color chart (color test photographing target) O ₁ is photographed in advance in order to perform correlation correction of luminance with the single-plate RGB imaging element 7 and the monochrome imaging element 8. The color chart O ₁ includes, for example, a Macbeth chart which is a color checker in which 24 color areas are arranged as illustrated in FIG. The color chart image photographed at this time is denoted as photographed image 1. The luminance calibration unit 20 optimizes the color luminance conversion matrix using the captured image 1 and holds the integrated luminance value generated by the optimized color luminance conversion matrix in the storage unit 4. Thereafter, in the photographing apparatus 1, the subject (the actual photographing target) O ₂ is photographed by the single-plate RGB imaging element 7 and the monochrome imaging element 8. The captured image at this time is referred to as captured image 2. The integrated image generation means 30 performs image processing at the integration stage based on the captured image 2. Hereinafter, first, the image processing at the calibration stage by the image integration unit 11 will be described.

(3. Detailed configuration of the image integration unit)
Here, 3-1. Outline of calibration related to luminance correlation calibration, 3-2. Luminance calibration means, 3-3. Integrated image generating means, 3-4. First modification of image integration unit, 3-5. This will be described in each section of the second modification of the image integration unit.

<3-1. Outline of calibration related to luminance correlation calibration>
Usually, there are many conversion matrices for converting from RGB to another color space such as luminance color difference. However, this embodiment aims at matching not the apparent luminance but the luminance of the pixel of the captured image by the single-plate RGB imaging element 7 and the luminance of the pixel of the captured image by the monochrome imaging element 8. Therefore, a 3 × 1 color luminance conversion matrix [a b c] is defined, and the RGB luminance values of the pixels of the image captured by the single-plate RGB imaging device 7 are multiplied by the color luminance conversion matrix [a b c]. Thus, the luminance value is obtained, and the color luminance conversion matrix [a b c] is calculated and used by optimization.

<3-2. Brightness calibration means>
The luminance calibration means 20 shown in FIG. 2 has RGB values (hereinafter referred to as full resolution) of RGB planes generated by demosaic that interpolate the RGB pixels of the color image obtained by photographing the color chart O ₁ with the single-plate RGB imaging device 7. RGB values) are converted by the color luminance conversion matrix [a b c] to generate luminance values of the pixels of the color image, and the color chart O ₁ is photographed by the monochrome image sensor 8 simultaneously with the single-plate RGB image sensor 7. The color luminance conversion matrix [a b c] is optimized so as to minimize the difference between the luminance value of the captured image (monochrome captured image) and the luminance value of the color image.

  For this purpose, the luminance calibration unit 20 includes a demosaicing processing unit 21, an average RGB value calculation unit 22, an average luminance value calculation unit 23, and a color luminance conversion matrix optimization processing unit 24, as shown in FIG. And.

The demosaicing processing means 21 is an RGB value (hereinafter referred to as a full-resolution RGB value) of an RGB plane generated by demosaicing that interpolates each RGB pixel of a color image obtained by photographing the color chart O ₁ with the single-plate RGB imaging device 7. Is what you want. The full resolution RGB value obtained by photographing the color chart O ₁ is hereinafter referred to as RGB value 1. This RGB value 1 is used by the average RGB value calculation means 22.

  Here, a general demosaic process performed by the demosaicing processing means 21 will be described with reference to FIG. The demosaicing processing means 21 first decomposes the captured image 7a of the single-plate RGB image sensor into RAW data having a coarse G plane 101, R plane 102, and B plane 103. In the Bayer array, a group of two G pixels, one R pixel, and one B pixel is used as a unit.

  The demosaicing processing means 21 generates a full resolution G image 111 by interpolating the G plane 101 using the edge direction of the image. An adaptive color plane interpolation method (ACPI: Advanced Color Plane Interpolation) is generally used as an interpolation method using the edge direction of an image.

The demosaicing processing unit 21 performs interpolation by using the edge direction of the image and the gradient of the full resolution G image 111 for the R plane 102 to generate a full resolution R image 122. .
Similarly, the demosaicing processing means 21 interpolates the B plane 103 by using the edge direction of the image and using the gradient of the full resolution G image 111, and the full resolution B image 123. Is generated.

  The demosaicing processing means 21 generates a full resolution G image 121, a full resolution R image 122, and a full resolution B image 123 as full resolution RGB values (RGB value 1) by the above processing. Note that the full resolution G image 111 is the same as the full resolution G image 121, and in order to clarify that the full resolution G image 111 is an image used for interpolation of the R plane and the B plane, The sign is changed.

Returning to FIG. 2, the description of the configuration of the luminance calibration means 20 will be continued.
Average RGB value calculating means 22 based on the RGB values of the full resolution obtained by photographing a color chart O ₁ (RGB value 1), for each color area of the color chart O _1, the average value of the luminance of R, the luminance of G A set of the average value and the average value of the luminance of B is obtained as an RGB average luminance value [R _c G _c B _c ] and is converted by the color luminance conversion matrix [a b c] to thereby obtain the color image. and it generates a luminance value Y _c of the pixel.
The average RGB value calculating means 22 calculates the 3 × 1 color luminance conversion matrix [a b c for the RGB average luminance value [R _c G _c B _c ] according to the equation (1) for each color area of the color image. ] over to generate a luminance value Y _c. This luminance value Y _c is used by the color luminance conversion matrix optimization processing means 24.

The average luminance value calculating means 23 obtains the average value Y _g of the luminance values of the pixels for each color region of the color chart O ₁ based on the monochrome photographed image photographed by the monochrome image sensor 8. This luminance value Y _g is used by the color luminance conversion matrix optimization processing means 24.

The color luminance conversion matrix optimization processing means 24 calculates a sum of differences between the luminance value Y _{ci of} the i-th color region calculated from the color image and the luminance value Y _gi of the i-th color region of the monochrome photographed image. As an evaluation function, a 3 × 1 color luminance conversion matrix [a b c] is optimized and obtained by using, for example, the Markert method. The color luminance conversion matrix optimization processing unit 24 calculates the value of the function of Expression (2) so that the evaluation function is minimized, and optimizes the 3 × 1 color luminance conversion matrix [a b c]. To do.

When the color luminance conversion matrix [a b c] is optimized, the luminance value Y _c calculated from the color image is substantially equal to the luminance value Y _{g of the} color area of the monochrome photographed image. The luminance value at this time is stored in the storage unit 4 as the integrated luminance value Y _T. Details of the integrated luminance value Y _T will be described later. Here, the optimized color luminance conversion matrix [a b c] is referred to as color luminance conversion matrix K anew as shown in Equation (3). This color luminance conversion matrix K is also stored in the storage unit 4.

In the above description, the color test photographing target to be photographed in advance is described as the color chart O ₁ , but the color test photographing target is not limited to the color chart. For example, it can be calculated in the same manner by minimizing the difference between the RGB luminance values of each pixel and the luminance value of the monochrome element in a generally photographed image.

  In this case, the luminance calibration unit 20 captures a general captured image (color test imaging target) updated according to the external environment with the single-plate RGB imaging device 7 and the monochrome imaging device 8 each time. Is used to optimize the 3 × 1 color luminance conversion matrix [a b c], and the optimized color luminance conversion matrix K is updated according to a general captured image (color test imaging target) captured each time. However, it is held in the storage unit 4.

  Also, the brightness calibration means 20 thins out pixels according to a predetermined rule from each photographed image photographed by the single-plate RGB image sensor 7 and the monochrome image sensor 8, and the remaining pixels are targeted for the monochrome photographed image. The color luminance conversion matrix K = [a b c] may be optimized so that the difference between the luminance value and the luminance value of the color image is minimized. Here, the rule of thinning out pixels is, for example, a rule of sampling every other pixel or randomly selecting a pixel. Note that the rules are not particularly limited as long as the positions of pixels to be thinned by the imaging devices are the same.

<3-3. Integrated image generation means>
The integrated image generating means 30 converts the RGB values of the RGB plane generated by demosaicing from the color images of the captured images obtained by capturing the subject (the actual imaging target) O ₂ with the single-plate RGB imaging device 7 and the monochrome imaging device 8. On the other hand, a luminance value that is not saturated with the monochrome captured image is selected from the luminance values converted by multiplying the color space conversion matrix C including the color luminance conversion matrix K optimized by the luminance calibration means 20. It obtains the color information in the integrated luminance and color space by which the, the color information of the integrated image obtained by integrating the respective captured video into one, and is intended to generate an integrated image I _t integrated luminance as the luminance information . Here, the brightness value to be selected from the converted brightness value will be described in detail later, but it is converted in the brightness value of the area that is not saturated in the monochrome captured image and the area that is saturated in the monochrome captured image. In the luminance value, the luminance value in an unsaturated region is shown. In the following, for simplicity, the integrated image generating means 30 uses the color information of the color image obtained by photographing the subject O ₂ with the single-plate RGB imaging device 7 as it is, and uses the luminance information of the color image as the integrated luminance value Y. _The integrated image _IT is generated by replacing with _T.

In the present embodiment, the integrated image generation unit 30 includes a color space mapping unit 31 and a color information calculation unit 32.
The color space mapping means 31 is a full-resolution RGB value obtained by demosaicing a color image obtained by photographing the subject (the actual photographing target) O ₂ with the single-plate RGB imaging device 7 by the demosaicing processing means 21 of the luminance calibration means 20. Is multiplied by a color space conversion matrix C shown in Expression (4).

The color space conversion matrix C shown in Expression (4) is a matrix that maps a full resolution RGB value obtained by demosaicizing a color image to the color space, and includes a color luminance conversion matrix K = [a b c]. is there. d, e, f, g, h and i are predetermined constants. This color space conversion matrix C is a YC _{b Cr} conversion matrix. The color space mapping unit 31 maps the RGB average luminance value [R _c G _c B _c ] into a special color space by performing the conversion represented by equation (5).

Here, C ₁ and C ₂ indicate color difference information. Y _c is a luminance value due to the color image, but in the special color space of the mapping destination, since the luminance is integrated, it has the same meaning as the integrated luminance value Y _T here. ing. As will be described later, when the luminance region is divided into three, the integrated luminance value Y _T may be related to a color image or a monochrome photographed image.

The color information calculation unit 32 obtains the RGB value [R _t G _t B _t ] of the integrated image I _T by multiplying the matrix mapped to the special color space by the inverse matrix C ⁻¹ of the color space conversion matrix C. Is. Here, it mapped to a special color space matrix includes a luminance value Y _t. Here, the luminance value Y _t indicates only that the luminance is integrated in the special color space of the mapping destination, and has the same meaning as the integrated luminance value Y _T. C ₁ ′ and C ₂ ′ indicate color difference information of a special color space in which luminance is integrated.

As described above, the integrated image generating means 30 once maps the RGB values of the pixels of the color image into a special color space, and then multiplies the inverse matrix C ⁻¹ of this color space conversion matrix C to obtain the RGB of the integrated image. Find the value. The reason for multiplying the inverse matrix in this way is that, in the calculation of each pixel, the luminance value Y _{t on} the right side of the equation (6) takes into account the luminance value of the region that is not saturated in the monochrome captured image, and the integrated luminance This is because the luminance value Y _c of the matrix on the left side of Expression (5) once mapped and generated in order to obtain Details of such a case will be described later. By performing the conversion twice, the RGB value of the integrated image can be accurately obtained while simultaneously treating the luminance value and the color information. If the luminance value Y _{t on} the right side of Equation (6) is the same as the luminance value Y _c of the matrix on the left side of Equation (5), it is not necessary to multiply the inverse matrix C ⁻¹ , and therefore color information The calculating means 32 may not be provided. This corresponds to a case where the captured image has only luminance values that are not saturated.

<3-4. First Modification of Image Integration Unit>
Here, a first modification of the image integration unit will be described with reference to FIG. The image integration unit 11A shown in FIG. 3 is different from the image integration unit 11 shown in FIG. 2 in that it further includes a position calibration means 40 (see FIG. 3) as an option at the calibration stage. FIG. 3 shows only the image integration unit 11 of the photographing apparatus 1 shown in FIG.

As a premise, in order to correct the geometric installation error with the single-plate RGB imaging device 7 and the monochrome imaging device 8 in the imaging device 1, a pattern (position test imaging target) O ₃ is imaged in advance. The pattern O ₃ is composed of, for example, a checkered pattern as illustrated in FIG. Among the captured images of the pattern O ₃ , for example, 25 intersections 41 in a checkered pattern are detected as candidates for each corner. The captured image at this time is referred to as a captured image 3. The position calibration means 40 mainly performs image processing relating to the photographed image 3.

The position calibration means 40 shown in FIG. 3 optimizes a projection matrix for matching the positions of the respective pixels based on a plurality of captured images obtained by capturing the pattern O ₃ with each image capturing device, thereby capturing each image. Pixels are associated between captured images of devices.

The position calibration means 40 detects the corner coordinates of each captured image by image processing (for example, a Harris operator). That is, the position calibration unit 40 corresponds to the RGB coordinate (X _i , Y _i , Z _i ) corresponding to the i-th corner of the color image and the corresponding monochrome captured image of the monochrome captured image at the same i-th corner. The coordinates at (x _i , y _i , z _i ) are detected.

  The position calibration unit 40 corrects the geometric installation error as follows. That is, the position calibration means 40 corresponds to the coordinate value obtained by multiplying the coordinate corresponding to the corner of one of the color image and the monochrome photographed image by the 3 × 3 projection matrix and the corner of the other image. The projection matrix is optimized so that the coordinates to be made are equal.

In this modification, the position calibration means 40 performs this optimization using a marker method. Here, the projection matrix is derived from a matrix [X _i Y _i Z _i ] indicating RGB coordinates corresponding to the corners of the color image, and a matrix [x _i y indicating coordinates in the monochrome captured images corresponding to the corners of the monochrome captured image. _i z _i ] (conversion coordinate value). Further, as the geometric installation error of each imaging device, the following distance d is obtained at each corner of each captured image. This distance d indicates the distance between the coordinate value (conversion coordinate value) obtained by multiplying the RGB coordinates and the projection matrix at the corner and the coordinate value in the monochrome photographed image at that corner. Then, an evaluation function was defined so that the sum of distances obtained at each corner was minimized. As the initial value of the projection matrix, an average of coordinates of each corner of the color image, that is, a centroid position, and a value that matches the centroid position of the monochrome photographed image with the centroid position are used. The projection matrix P optimized by the position calibration means 40 is stored in the storage unit 4. Hereinafter, the pixel values are referred to by converting coordinates through the projection matrix P.

<3-5. Second Modification of Image Integration Unit>
Here, a second modification of the image integration unit will be described with reference to FIG. The image integration unit 11B illustrated in FIG. 4 is different from the image integration unit 11 illustrated in FIG. 2 in that the image integration unit 11B further includes a white balance processing unit 50 as an integration stage option. FIG. 4 shows only the image integration unit 11 of the photographing apparatus 1 shown in FIG. 1, and among the block components shown in FIG. 2, the block configuration not directly related to the description in FIG. Illustration of elements was omitted.
The white balance processing means 50 shown in FIG. 4 accurately displays the white color of the photographed image 2 obtained by photographing the subject (the actual photographing object) O ₂ with the photographing single-plate RGB imaging device 7 as in the conventional camera. A white balance process for correcting the white balance is performed. The white balance can be adjusted by a general method.

(4. Luminance integration processing)
Here, regarding the luminance integration processing performed by the luminance calibration means shown in FIG. Overview of luminance integration processing, 4-2. This will be described in each section of a specific example of luminance integration processing.
<4-1. Overview of luminance integration processing>
FIG. 6 is an explanatory diagram of luminance integration processing performed by the luminance calibration means shown in FIG.
The horizontal axis of the graph in FIG. 6 indicates the amount of incident light that enters each imaging device, and the vertical axis of the graph indicates the luminance value of each imaging device.

The gradient of the luminance value of the monochrome imaging element 8 increases linearly up to the incident light quantity Q ₁ and is saturated thereafter, as indicated by the thick solid line 201. That is, the monochrome image pickup device 8 reaches the saturation luminance value S _th1 at the incident light quantity Q ₁ .

On the other hand, the gradient of the RGB value [R _c G _c B _c ] of the single-plate RGB image pickup device 7 increases linearly without being saturated with the incident light quantity Q ₁ , as indicated by the thin solid line of reference numeral 202. _{In FIG. 3} , even when the saturation luminance value S _th1 of the monochrome image pickup element 8 is reached, it increases linearly. In the graph, regarding the gradient of the RGB value, the luminance of (R _c + G _c + B _c ) / 3 is shown.

The gradient of the converted luminance obtained by converting the transposed matrix of the RGB values [R _c G _c B _c ] of the single-plate RGB image sensor 7 with the optimized matrix [a b c] according to the above-described equation (1) is This is indicated by a solid line 203.

The gradient of the luminance after conversion is approximately twice as large as the gradient of the solid line 202 as indicated by the solid line 203. In a more detailed experiment, the saturation luminance value derived from the single-plate RGB imaging element 7 was improved to about 1.5 to 2.5 times the saturation luminance value S _th1 of the monochrome imaging element 8. Therefore, the dynamic range is improved by 3.5 to 8 dB in the integrated luminance value Y _T. This is 75.5 to 80 dB in consideration of the dynamic range (72 dB) of the image sensor. That is, HDR (High Dynamic Range) can be achieved.

As shown in FIG. 6, the monochrome image pickup device 8 has higher brightness if the amount of incident light is the same. Further, the monochrome image pickup device 8 has a larger amount of incident light on each image pickup device. Therefore, the image integration unit 11 uses the monochrome imaging element 8 for a dark region. Since it is used for a dark region, the luminance (Y _g ) of the monochrome image sensor 8 is likely to be saturated. Therefore, the image integration unit 11 replaces an area where the luminance (Y _g ) of the monochrome image sensor 8 is saturated with an optimized luminance value. This is the integrated luminance value Y _T. Incidentally, if the captured image just areas where the luminance of the monochrome image pickup device 8 is saturated, it is possible to use a Y _c shown as an integrated luminance value Y _T, the equation (1).

As a more preferred embodiment, the luminance value (Y _g ) of the monochrome imaging device 8 is used for a relatively low amount of incident light, and Y _c shown in the above equation (1) for a relatively high amount of incident light. Can be used. The configuration for this will be described later. In this case, in the vicinity of the saturated luminance value S _th1, in order to smoothly connect the slope, the luminance region of the integrated luminance value as the first threshold saturation luminance value S _th1, predetermined value S _sub only small saturation level than this S _th2 is divided into three with the second threshold value. The saturation level S _th2 is a luminance value corresponding to the incident light quantity Q ₂ . In the present embodiment, the saturation level S _th2 is about 10% of the saturation luminance value S _th1 .

In the following, a luminance region is assumed in which the graph of FIG. 6 is divided into three by broken lines parallel to the horizontal axis. Here, the regions having high luminance are referred to as region α, region β, and region γ.
The region α is a luminance region corresponding to the incident light amount from the incident light amount Q ₁ to the incident light amount Q ₃ , and the integrated luminance value Y _T is represented by Y _c shown in the above equation (1).
The region γ is a luminance region corresponding to the incident light amount from the incident light amount 0 to the incident light amount Q ₂ , and the integrated luminance value Y _T is represented by the luminance value Y _g of the monochrome imaging element 8.
The region β is a luminance region corresponding to the incident light amount from the incident light amount Q ₂ to the incident light amount Q ₁ , and the integrated luminance value Y _T is the Y _c shown in the above equation (1) and the luminance value Y _g . It is expressed as a weighted average according to the distance from the two thresholds. The weighted average luminance value is denoted as Y _t .

<4-2. Specific example of luminance integration processing>
FIG. 7 is an explanatory diagram showing a specific example of luminance integration processing using a color chart, where (a) is a captured image of a monochrome image sensor, (b) is a captured image of a single-plate RGB image sensor, and (c). Respectively show graphs showing the results of luminance matching by optimization.
Here, an image sensor having the following specifications was used.
8.9M≈4000 × 2000 (pixels), 12 bits As the interface between the image sensor and the calculation PC constituting the calculation processing unit 3, simple hardware capable of converting from LVDS to PCI Express was used. The signal from the image sensor is directly input to the computing PC without any processing, and this is transferred to the memory on the graphics card in the computing PC, and processed with floating point 32 bits by the GPU on the graphics card. went. Note that all the image processing after luminance integration was also performed on the graphics card.

In the graph of FIG. 7C, the horizontal axis indicates each color area of the color chart, and the vertical axis indicates the luminance ratio of the color image pixel to the luminance value of the captured image pixel of the monochrome imaging element 8.
The luminance matching result indicated by a diamond in this graph is a comparative example, and is a luminance Y matched with the transmission primary color defined for the 1080 system in the BTA S-001 standard.
Y = [0.2126 0.7152 0.0722] [RGB] ^T Equation (7)
The BTA S-001 standard is described in “standard name“ 1125/60 high definition television system studio standard (ARIB BTA-001 C 1.0 version) ”, Japan Radio Industry Association.

The luminance obtained by simply converting the RGB value [R _c G _c B _c ] of the single-plate RGB image sensor 7 in accordance with the normal HDTV (high vision) standard is the monochrome image sensor 8 as shown in the graph of FIG. It can be seen that this is far from the luminance value “1.000” of the pixel in the photographed image of FIG. 5, and further varies greatly depending on the color of the color chart. This is because the normal HDTV (high vision) standard does not exactly match the luminance of pixels in a monochrome photographed image due to spectral characteristics such as subject reflectance, light source, and color filter.

On the other hand, the luminance matching result indicated by a square in the graph of FIG. 7C is an integrated luminance value Y _T integrated by the imaging apparatus 1 of the present embodiment. The luminance area was divided into areas α, β, and γ. As shown in the graph of FIG. 7C, the integrated luminance value Y _T obtained by optimizing the coefficients for R, G, and B is the luminance of the pixel in the captured image of the monochrome imaging element 8 over each color area. The values agree very well, and it can be seen that the luminance of the pixels in the color image and the luminance of the pixels in the monochrome photographed image are integrated.

(5. Color information integration processing)
The image integration unit 11C illustrated in FIG. 8 includes an integrated image generation unit 30C in order to perform processing by dividing a luminance region into regions α, β, and γ. FIG. 9 shows only the image integration unit 11 of the photographing apparatus 1 shown in FIG. 1, and among the block components shown in FIG. 2, the block configuration not directly related to the description in FIG. Illustration of elements was omitted.
The integrated image generation unit 30C includes a color space mapping unit 31C, a color information calculation unit 32C, an integrated luminance determination unit 33, a second demosaicing processing unit 34, a second color space mapping unit 35, and a weighted average unit 36. And.

This integrated image generating means 30C is premised on the following operations. Since these operations are the same as the processing described in the calibration stage, description thereof will be omitted as appropriate.
In the photographing apparatus 1, when the subject (the actual photographing target) O ₂ is photographed by the single-plate RGB imaging element 7 and the monochrome imaging element 8, the average RGB value calculating means 22 (see FIG. 2) of the luminance calibration means 20 is obtained. An RGB average luminance value [R _c G _c B _c ] is obtained based on a full resolution RGB value (RGB value 2) obtained by photographing the subject O ₂ , and this is converted by a color luminance conversion matrix [a b c]. Thus, the luminance value Y _c of the pixel in the color image of the subject O ₂ is generated.
Further, the average brightness value calculating means 23 (see FIG. 2) of the brightness calibration means 20 obtains the brightness value Y _g of the pixel in the monochrome captured image obtained by capturing the subject O ₂ . The luminance value Y _c of the pixel in the color image of the subject O ₂ and the luminance value Y _g of the pixel in the monochrome photographed image are sent to the integrated luminance determination unit 33.

The color space mapping unit 31C shown in FIG. 8 is the same as the color space mapping unit 31 shown in FIG. 2, except that only those having an integrated luminance value in the α region are processed. For this reason, the luminance value Y _c converted by the conversion expressed by the above equation (5) is attributed to the color image in the integrated luminance value Y _T. Further, since the color difference information C ₁ and C ₂ converted by the conversion expressed by the above equation (5) is associated with the luminance value due to this color image, the luminance value Y _c , the color difference information C ₁ and C 2 ₂ is output to the color information calculation means 32C as adjusted color information.

The integrated luminance discrimination means 33 receives the luminance value Y _c of the color image of the subject O ₂ and the luminance value Y _g of the monochrome photographed image, and refers to the integrated luminance value Y _T generated by the luminance calibration means 20 to The luminance value input from the imaging device has a region where the monochrome image sensor 8 is saturated (α region), a region where the monochrome image sensor 8 is not saturated (γ region), and an intermediate region (β region) between them. It is determined which of the following is true.

When the integrated luminance determining unit 33 determines that the luminance value input from each imaging device is the α region, the integrated luminance determining unit 33 outputs an operation instruction to the color space mapping unit 31C.
The integrated luminance determining unit 33 outputs an operation instruction to the second color space mapping unit 35 when determining that the luminance value input from each imaging device is the γ region.
The integrated luminance determining unit 33 outputs an operation instruction to the second demosaicing processing unit 34 when determining that the luminance value input from each imaging device is in the β region.

The second demosaicing processing unit 34 performs demosaic processing again on the pixels in the unsaturated region of the color image of the subject O ₂ imaged by the single-plate RGB imaging device 7 based on the operation instruction from the integrated luminance determination unit 33. Is to be applied. The full-resolution RGB values that have been demosaiced again are used by the second color space mapping means 35.

  Here, demosaic processing performed by the second demosaicing processing means 34 will be described with reference to FIG. The second demosaicing processing means 34 first decomposes the captured image 7a of the single-plate RGB image sensor into RAW data having a coarse G plane 101, R plane 102, and B plane 103.

  The second demosaicing processing means 34 interpolates the G plane 101 by using the edge direction of the image and by using the gradient of the captured image 8a of the monochrome image pickup device, thereby obtaining a full resolution G image. 321 is generated.

Similarly, the second demosaicing processing means 34 performs interpolation on the R plane 102 by using the edge direction of the image and by using the gradient of the captured image 8a of the monochrome image sensor. A resolution R image 322 is generated.
Similarly, the second demosaicing processing means 34 performs interpolation on the B plane 103 by using the edge direction of the image and using the gradient of the captured image 8a of the monochrome image sensor. A full resolution B image 323 is generated.

The second demosaicing processing means 34 generates the full resolution G image 321, the full resolution R image 322, and the full resolution B image 323 as the full resolution RGB values of the color image of the subject O ₂ by the above processing.

Returning to FIG. 8, the description of the configuration of the integrated image generating means 30C will be continued.
The second color space mapping means 35 is the same as the color space mapping means 31C, except that only those having an integrated luminance value in the γ region are processed. For this reason, the luminance value Y _c converted by the conversion expressed by the above equation (5) is attributed to the monochrome photographed image in the integrated luminance value Y _T. Further, since the color difference information C ₁ and C ₂ converted by the conversion expressed by the above equation (5) is associated with the luminance value caused by the monochrome photographed image, the luminance value Y _c , the color difference information C ₁ , C ₂ is output to the color information calculation unit 32C as adjusted color information.

The weighted average means 36 adjusts color information in a matrix obtained by mapping the RGB average luminance values [R _c G _c B _c ] generated by the first demosaic process by the color space mapping means 31C into a special color space; The adjusted color information in the matrix in which the RGB average luminance value [R _c G _c B _c ] generated by the second demosaic process by the second color space mapping unit 35 is mapped to a special color space is obtained from two threshold values. Color information adjusted for the β region is generated by weighted averaging according to the distance. For this reason, the adjusted color information of the β region is caused by both the color image and the monochrome photographed image. Luminance value Y _t in the adjusted color information, of the integrated luminance value Y _T, the one named after both color images and monochrome photographic image. The luminance value Y _t and the averaged color difference information C ₁ and C ₂ are respectively output to the color information calculation unit 32C as adjusted color information.

The color information calculation unit 32C is the same as the color information calculation unit 32 shown in FIG. 2, but in addition to the color space mapping unit 31C, a matrix input from the second color space mapping unit 35 and the weighted average unit 36 is also used. The difference is that the inverse matrix C ⁻¹ of the color space conversion matrix C is multiplied.

  Here, the integration of the adjusted color information executed by the color information calculation unit 32C will be described with reference to FIG. FIG. 10 is an explanatory diagram similar to the explanatory diagram shown in FIG. 6, but in the following, it is assumed that the incident light quantity region is obtained by dividing the explanatory diagram of FIG. 10 into three along the vertical axis.

A relatively large region where the amount of incident light is Q ₁ or more on the horizontal axis in FIG. 10 corresponds to the region α of the luminance region. In the case of this region, the color information calculation unit 32C uses the matrix information input from the color space mapping unit 31C to provide color information corresponding to the first demosaic process of the single-plate RGB imaging device 7 (one imaging device). The color information associated with the pixel) is transferred as the color information of the pixel.

In addition, a relatively small region where the amount of incident light is Q ₂ or less on the horizontal axis in FIG. 10 corresponds to the region γ of the luminance region. In the case of this region, the color information calculation unit 32C uses the matrix information input from the second color space mapping unit 35 to use the color information corresponding to the second demosaic process of the single-plate RGB image sensor 7 (monochrome image sensor). And the color information associated with the two imaging devices of the RGB elements) is transferred as the color information of the pixel.

Further, the intermediate region where the incident light quantity is Q ₂ or more and Q ₁ or less on the horizontal axis in FIG. 10 corresponds to the region β of the luminance region. In the case of this region, the color information calculation unit 32C uses the matrix information input from the weighted average unit 36 and transfers the color information after performing a weighted average according to the distance from the two threshold values Q ₂ and Q _1. . The distance from the two threshold values Q ₂ and Q ₁ corresponds to the distance from the saturation level S _th2 and the saturation luminance value S _{th1 in the} luminance value.

(6. Flow of operation of the photographing device)
Next, the operation flow of the photographing apparatus 1 shown in FIG. 1 will be described. For example, it is preferable to perform the following mechanical alignment before the calibration stage of image processing. In this mechanical alignment, as a rough adjustment (coarse adjustment), for example, a checkerboard like the pattern shown in FIG. 3 is photographed by the photographing apparatus 1 and visually observed, the single-plate RGB imaging element 7 and the monochrome imaging element 8. Are adjusted with respective micrometers (6 degrees of freedom) arranged in advance. Subsequently, as a calibration stage of image processing, the position calibration means 40 (see FIG. 3) performs fine adjustment (fine adjustment). Further, as a calibration stage of image processing, the brightness calibration means 20 (see FIG. 2) generates an integrated brightness value Y _T. The position calibration may be performed before the brightness calibration or may be performed later.

  In the following, image processing after coarse adjustment and fine adjustment and luminance calibration in the photographing apparatus 1 will be described with reference to FIG. 11 (refer to FIGS. 1, 2, 4 and 8 as appropriate). Note that the luminance area is divided into areas α, β, and γ for processing.

First, in the photographing apparatus 1, the subject (the actual photographing target) O ₂ is photographed by the single-plate RGB imaging element 7 and the monochrome imaging element 8.
The luminance calibration unit 20 of the image integration unit 11 performs demosaicing of the color image of the subject O ₂ by ACPI or the like using the demosaicing processing unit 21 (step S1). Note that the full resolution RGB value generated in step S1 is less accurate than the full resolution RGB value generated in step S5.

Then, the luminance calibration unit 20 converts the RGB value of the full resolution of the subject O ₂ by the average RGB value calculation unit 22 using the optimized color luminance conversion matrix K = [a b c]. A luminance value Y _{c of the} image is generated (step S2).

  The luminance information of the color image generated in step S2 can be used to interpolate the luminance information of the region where the amount of incident light incident on the single-plate RGB imaging device 7 is relatively small with respect to the monochrome imaging device 8. Information. On the other hand, the luminance information of the monochrome photographed image 8 a photographed by the monochrome image sensor 8 is obtained by interpolating the luminance information of a region where the incident light amount incident on the monochrome image sensor 8 is relatively large with respect to the single-plate RGB image sensor 7. It is.

The integrated image generating means 30C receives the luminance value Y _c of the color image of the subject O ₂ and the luminance value Y _g of the monochrome photographed image, and integrates the luminance information (step S3). Then, the integrated image generating unit 30C refers to the integrated luminance value Y _T created in advance by the integrated luminance determining unit 33, and the luminance value input from each imaging device is any of the α region, the γ region, and the β region. (Step S4). When it is determined that the luminance value corresponds to the γ region, the second demosaicing processing unit 34 performs demosaic processing again on pixels in the unsaturated region of the color image of the subject O ₂ imaged by the single-plate RGB image sensor 7. (Step S5). Note that the full resolution RGB value generated in step S5 uses the gradient of the captured image 8a of the monochrome image sensor, and therefore is more accurate than the full resolution RGB value generated in step S1. It becomes color information.

Subsequent to step S4 or step S5, the color information calculation unit 32C corresponds to each of the luminance value areas by using color information associated with the captured image of one imaging device or two imaging devices. The color is transferred to the pixel (step S6). Thus, the integrated image I _t is generated and stored in the storage unit 4.

Specifically, when it is determined that the luminance value corresponds to the α region, the color information calculation unit 32C is generated by the first demosaic process of the color image of the subject O ₂ imaged by the single-plate RGB image sensor 7. Based on the obtained color information (color information associated with one image pickup device), the color is transferred to a pixel whose luminance value corresponds to the α region.

If it is determined that the luminance value corresponds to the γ region, the color information calculation unit 32C performs the second demosaic process (two imaging devices) on the color image of the subject O ₂ captured by the single-plate RGB imaging element 7. The color information is transferred to a pixel whose luminance value corresponds to the γ region.

  Further, when it is determined that the luminance value corresponds to the β region, the color information calculation unit 32C determines the color information of the pixel determined to correspond to the α region and the color of the pixel determined to correspond to the γ region. The color is transferred to a pixel whose luminance value corresponds to the β region by color information obtained by weighted averaging the information.

Following step S6, the preview processing unit 12 of the imaging apparatus 1, the video processing function reads the integrated image I _t from the storage unit 4, performs non-linear tone mapping in accordance with the monitor 13 (step S6). The video on which this nonlinear tone mapping has been performed is stored in the post-production storage unit 5 or output to the monitor 13.

By photographing using the photographing apparatus 1 of the present embodiment, a high dynamic range and high definition video can be acquired. Thus, typically, or imaging device saturation, even in the scene, such as a region that can not be visually recognized or to underexposure occurs, integrated image I _t is less likely to saturate, underexposure difficult.

  In addition, due to the expansion of the dynamic range, no significant image quality degradation was observed compared to conventional hardware processing with limited accuracy or low accuracy, despite multi-stage image processing.

In addition, when performing demosaic processing, it has been difficult to ensure quality because the conventional method is insufficient sampling. However, in the imaging apparatus 1 of the present embodiment, the luminance of the monochrome captured image is about the γ region of the luminance region. to use the value Y _g original luminance value of RGB elements, when performing demosaic process again, by utilizing the gradient of the luminance values Y _g monochrome photographic images, to prevent deterioration of color bleeding or fineness Can do. Therefore, even if the brightness or color is changed by image processing in the post-shooting process, the pseudo contour is hardly deteriorated, and even if enlarged, the color blur does not occur and the sharpness is not impaired. Therefore, according to the imaging device 1, the degree of freedom of video processing can be expanded.

  In the embodiment, all tone mapping, including tone mapping, can be performed in real time by a single GPU mounted on one PC. That is, the photographing apparatus 1 of the present embodiment can be easily realized in hardware, and can be expected to be reduced in cost. Further, if the computing performance by the PC is improved, the color luminance conversion matrix K can be automatically updated in real time during shooting of a normal subject.

  As mentioned above, although this invention was demonstrated based on this embodiment, this invention is not limited to this. For example, the photographing apparatus may include two color imaging devices having different spectral characteristics and one monochrome imaging device. In this case, as the first phase, first, an integrated luminance value is generated as described above for one color imaging device and one monochrome imaging device, and the images are integrated. Here, one of the two color imaging devices (first color imaging device) and the monochrome imaging device are to be integrated.

  Next, as the second phase, the second color imaging device is replaced with the monochrome imaging device in the first phase, and the replaced second color imaging device and the first color imaging device are integrated. As a result, an integrated luminance value is generated in the same manner as in the first phase, and the images are integrated. In the case of the second phase, in the graph of FIG. 6, the second integrated luminance value can be generated and the images can be integrated by the same method as in the first phase by replacing as follows.

That is, in the replacement in the graph of FIG. 6, a thick solid line denoted by reference numeral 201 represents the gradient of the luminance value of the first color imaging device. On the other hand, a thin solid line denoted by reference numeral 202 represents the gradient of the luminance value of the second color imaging device. A solid line denoted by reference numeral 203 represents a gradient of converted luminance obtained by converting the luminance value of the second color imaging device using the optimized second color luminance conversion matrix. Further, the saturation luminance value S _th1 is referred to as a third threshold for convenience, and the saturation level S _th2 is referred to as a fourth threshold for convenience. In a range smaller than the fourth threshold, the luminance gradient of the integrated image integrated in the first phase is used in the demosaic process. Note that the second color imaging device may have a gentler or steeper luminance gradient than the first color imaging device. Further, if the spectral characteristics are different from each other, three or more color imaging devices can be similarly integrated.

  Further, for example, the photographing apparatus 1 shown in FIG. 1 can be realized by operating a general computer by an image integration program that functions as each unit of the image integration unit 11 described above. This program (image integration program) can be distributed via a communication line, or can be distributed by writing on a recording medium such as a CD-ROM.

  In the photographing apparatus 1, the image integration unit 11 may include the luminance calibration unit 20, the integrated image generation unit 30, the position calibration unit 40, and the white balance processing unit 50 as the best mode. .

  Further, in the integrated image generating means 30C shown in FIG. 8, the color information calculating means 32C is information on the entire image that has been processed for each pixel by the color space mapping means 31C, the second color space mapping means 35, and the weighted averaging means 36. Is used to perform the process of multiplying the inverse matrix for each pixel in the image, but is not limited to this. For example, the color information calculating unit 32C is excluded from the integrated image generating unit, and the color space mapping unit 31C, the second color space mapping unit 35, and the weighted average unit 36 are set according to the luminance of a predetermined pixel selected in the captured image. A similar effect can be achieved by performing any one of the processes and storing the information of the pixel and the integrated luminance and color information as the processing result of the pixel in association with each other.

  The present invention can be used not only in the video production industry but also in a wide range, for example, in an industry where it is necessary to obtain a monitoring video, and is particularly suitable for use in creating relatively high-quality video content.

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Imaging part 3 Arithmetic processing part 4 Memory | storage part 5 Posto storage part 6 Half mirror (spectral means)
7 Single-plate RGB image sensor 8 Monochrome image sensor 11, 11A, 11B, 11C Image integration unit (image integration means)
12 preview process unit 13 monitor 20 brightness calibration means 21 demosaicing processing means 22 average RGB value calculation means 23 average brightness value calculation means 24 color brightness conversion matrix optimization processing means 30, 30C integrated image generation means 31, 31C color space Mapping means 32, 32C Color information calculation means 33 Integrated luminance discrimination means 34 Second demosaicing processing means 35 Second color space mapping means 36 Weighted averaging means 40 Position calibration means 50 White balance processing means B 1/4 wavelength plate L lens O ₁ color chart (for color test shooting)
O ₂ subject (target for actual shooting)
O ₃ pattern (target for position test photography)

Claims (9)

Position where one or more single-plate color imaging devices having different spectral characteristics with RGB color filters and a monochrome imaging device without color filters capture the same imaging area with the same viewpoint and the same angle of view via the spectral means An imaging apparatus comprising: an imaging unit arranged in a relationship; and an image integration unit that performs a process of integrating each captured image captured using the color imaging device and the monochrome imaging device into one,
The image integration means includes
The luminance value of the pixel in the color image is obtained by converting the RGB value of the RGB plane generated by the demosaic that interpolates the RGB pixels of the color image obtained by photographing the color test photographing target with the color imaging device by the color luminance conversion matrix. And the difference between the luminance value of the pixel in the monochrome photographic image obtained by photographing the color test photographing target with the monochrome imaging device simultaneously with the color imaging device and the luminance value of the pixel in the color image is minimized. Brightness calibration means for optimizing the color brightness conversion matrix;
The matrix including the optimized color luminance conversion matrix for the RGB values of the RGB plane generated by demosaicing from the color images of the captured images captured by the color imaging device and the monochrome imaging device. Among the luminance values converted by multiplying by the color space conversion matrix that maps to the color space, the luminance value of the region that is not saturated in the monochrome captured image, and the region that is saturated in the monochrome captured image An integrated luminance and color information in the color space are obtained by selecting a luminance value in a region that is not saturated in the converted luminance value, and this is obtained as an integrated image obtained by integrating the captured images into one. And an integrated image generation unit configured to generate an integrated image using color information and the integrated luminance as luminance information. Shadow devices. The brightness calibration means includes
The color luminance conversion matrix is optimized using a color chart image obtained by photographing a color chart as the color test photographing target with the color imaging device and the monochrome imaging device, and the optimized color luminance conversion matrix is held in a storage unit. The imaging device according to claim 1. The brightness calibration means includes
The color luminance conversion matrix is optimized by using each of the color imaging device and the monochrome imaging device for each color test imaging object that has been updated according to an external environment, and the color luminance conversion matrix is optimized. The imaging apparatus according to claim 1, wherein the matrix is held in a storage unit while being updated in accordance with the color test imaging target. The brightness calibration means includes
From each captured image captured by the color imaging device and the monochrome imaging device, pixels are thinned out according to a predetermined rule, and the luminance value of the pixel in the monochrome captured image and the luminance of the pixel in the color image are set for the remaining pixels. The photographing apparatus according to any one of claims 1 to 3, wherein the color luminance conversion matrix is obtained by optimization so that a difference from the value is minimized. The image integration means includes
Based on a plurality of photographed images obtained by photographing the position test photographing object with the respective imaging devices, the coordinates of the feature points of the color image photographed with the color imaging device are converted into the coordinates in the monochrome photographed image by transforming with the projection matrix. A converted coordinate value is generated, and at the same time as the color imaging device, the position test imaging target is captured by the monochrome imaging device and the coordinates of the feature point of the monochrome captured image and the monochrome corresponding to the feature point of the color image 5. The apparatus according to claim 1, further comprising a position calibration unit that optimizes the projection matrix so as to minimize a difference from the converted coordinate value in a captured image. Shooting device. A plurality of the color imaging devices are provided, and each color imaging device has a color filter having different spectral sensitivity from each other,
The brightness calibration means includes
Of each captured image of the color imaging device, an RGB plane generated by demosaicing in the same manner as the first color image from a second color image other than the first color image used when obtaining the color luminance conversion matrix. The RGB value is converted by the second color luminance conversion matrix to generate the luminance value of the second color image, and the luminance value of the second color image and the luminance of the first color captured image already optimized are generated. The second color luminance conversion matrix is optimized so as to minimize a difference from the value,
The integrated image generation means includes
A second color space conversion matrix that is a matrix including the optimized second color luminance conversion matrix with respect to the RGB values of the RGB plane generated by demosaicing from the second color image, and that maps to the color space Among the luminance values converted by multiplying by the luminance value of the region not saturated in the first color image and the converted luminance value in the region saturated in the first color image. By selecting a luminance value in a region that is not saturated, the integrated luminance and the color information in the color space are obtained, and this is used as the color information of the second integrated image in which the captured images are integrated into one, and The imaging apparatus according to claim 1, further generating a second integrated image using the integrated luminance as luminance information. The integrated image generation means includes
When generating an integrated image obtained by integrating photographed images obtained by photographing the photographing objects with the color imaging device and the monochrome imaging device,
In a color image obtained by photographing the photographing object with the color imaging device, for pixels in a luminance region that is larger than a first threshold that is predetermined as an upper limit of the luminance of the monochrome imaging device,
Obtain the RGB value of the RGB plane generated by demosaicing from the color image,
For pixels in the luminance region smaller than the second predetermined threshold value smaller than the first threshold value,
A color image obtained by photographing the photographing object with the color imaging device is demosaiced again, and the demosaic image is generated after interpolation using the gradient of the luminance value of the monochrome photographed image photographed with the monochrome imaging device. Calculate the RGB value of the RGB plane,
For pixels in the luminance region that are greater than or equal to the second threshold and less than or equal to the first threshold,
The RGB value already calculated for the pixel in the luminance region greater than the first threshold value and the RGB value already calculated for the pixel in the luminance region smaller than the second threshold value are set as the luminance of the pixel and the first and second values. Weighted average according to the distance to the threshold,
6. The photographing apparatus according to claim 1, wherein the RGB value of the integrated image is calculated as described above. The integrated image generation means includes
When generating an integrated image obtained by integrating photographed images obtained by photographing the photographing objects with the plurality of color imaging devices and monochrome imaging devices,
In the second color image obtained by photographing the photographing object, for pixels in a luminance region that is larger than a third threshold value that is predetermined as the upper limit of the luminance of the first color image,
Obtain the RGB value of the RGB plane generated by demosaicing from the second color image,
For pixels in the luminance region smaller than the fourth threshold that is smaller than the third threshold,
The second color image is demosaiced again, and in this demosaic, the RGB value of the RGB plane generated after interpolation using the gradient of the luminance value of the integrated image is calculated,
For pixels in the luminance region that are greater than or equal to the fourth threshold and less than or equal to the third threshold,
The RGB value already calculated for the pixel in the luminance area larger than the third threshold value and the RGB value already calculated for the pixel in the luminance area smaller than the fourth threshold value are set as the luminance of the pixel and the third and fourth values. Weighted average according to the distance to the threshold,
The imaging apparatus according to claim 6, wherein the RGB value of the integrated image is calculated. Position where one or more single-plate color imaging devices having different spectral characteristics with RGB color filters and a monochrome imaging device without color filters capture the same imaging area with the same viewpoint and the same angle of view via the spectral means In order to perform a process of integrating the captured images captured using the imaging units arranged in a relationship into one,
The luminance value of the pixel in the color image is obtained by converting the RGB value of the RGB plane generated by the demosaic that interpolates the RGB pixels of the color image obtained by photographing the color test photographing target with the color imaging device by the color luminance conversion matrix. And the difference between the luminance value of the pixel in the monochrome photographic image obtained by photographing the color test photographing target with the monochrome imaging device simultaneously with the color imaging device and the luminance value of the pixel in the color image is minimized. Luminance calibration means for optimizing and obtaining the color luminance conversion matrix,
The matrix including the optimized color luminance conversion matrix for the RGB values of the RGB plane generated by demosaicing from the color images of the captured images captured by the color imaging device and the monochrome imaging device. Among the luminance values converted by multiplying by the color space conversion matrix that maps to the color space, the luminance value of the region that is not saturated in the monochrome captured image, and the region that is saturated in the monochrome captured image An integrated luminance and color information in the color space are obtained by selecting a luminance value in a region that is not saturated in the converted luminance value, and this is obtained as an integrated image obtained by integrating the captured images into one. Image for functioning as integrated image generation means for generating an integrated image with color information and using the integrated luminance as luminance information If the program.