JP2012199804A - Image processor and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem that when color conversion to an observational light source which is different from a shooting light source is performed to a shot image of a scene in which a reflecting object and a light emitting object are mixed, color is reproduced which is different from the original visibility, especially for the light emitting object.SOLUTION: An image shot under a shooting light source is input (S302), spectral radiance of the shooting light source is obtained (S306), and spectral radiance of the observational light source which is different from the shooting light source is obtained (S308). In the shot image, a light emitting object area corresponding to the light emitting object emitting light by itself in the scene is detected (S310), and the shot image is converted so as to reproduce the visibility under the observational light source by using the spectral radiance of the shooting light source and the observational light source (S313). At the time of conversion, it is possible to perform control so that the light emitting object area is not reflected by the observational light source by differentiating a degree of reflecting the spectral radiance of the observational light source between the light emitting object area and the remaining areas.

Description

  The present invention relates to an image processing apparatus and an image processing method for converting an image photographed by an imaging device into a color that is reproduced under an illumination light source different from that at the time of photographing.

  Conventionally, there is a color conversion processing technique using a multiband camera as a method of converting an image acquired by an imaging device such as a camera into a color that is reproduced under an illumination light source different from that at the time of shooting (for example, Patent Literature). 1, see Patent Document 2). In a color conversion processing technique using a multiband camera, first, a spectral reflectance obtained by removing information on a photographing light source from an image acquired by a multiband camera is estimated. Then, by multiplying the estimated spectral reflectance by the spectral radiance of an arbitrary illumination light source, color reproduction under an arbitrary illumination light source is possible.

JP 2010-246036 A JP-A-9-327024

  However, in the above conventional color conversion processing technique, when a scene to be photographed includes a light emitting object that emits light, the problem is that the light emitting object is also converted to a color different from the original color. is there. For example, consider a case where a scene in which a light-emitting object such as a display is reflected is converted to a color under an illumination light source different from that at the time of shooting. In this case, for the reflective object in the scene, the spectral radiance of the photographic illumination is removed from the captured image and the spectral reflectance is estimated. Can be reproduced. However, since a light-emitting object (display or the like) in a scene is not significantly affected by a photographing light source or an illumination light source, if a process similar to that of a reflective object is performed, it is converted into a color different from the original appearance.

  The present invention has been made in view of the above-described problems, and color conversion to an observation light source different from a photographing light source is performed on a photographed image of a scene including a light emitting object in consideration of color reproduction of the light emitting object. The purpose is to be accurate.

  As a means for achieving the above object, an image processing apparatus of the present invention comprises the following arrangement.

  That is, under an imaging light source, an image input unit that inputs a captured image captured by a multiband camera having four or more color filters, an imaging light source information acquisition unit that acquires a spectral radiance of the imaging light source, Observation light source information acquisition means for acquiring a spectral radiance of an observation light source different from the imaging light source, and a setting for setting a light emitting object region corresponding to a light emitting object emitting light in the scene in the captured image A first reflection result in which the spectral radiance of the photographing light source is reflected in the photographed image, and a second reflection result in which the spectral radiance of the observation light source is reflected in the photographed image. Conversion means for converting the photographed image so as to reproduce the appearance under the observation light source, and the conversion means includes the light emitting object region and the other region, The ratio of combining the first and second reflection results is different.

  According to the present invention, it is possible to perform color conversion of a captured image of a scene including a light emitting object to an observation light source different from the photographing light source with high accuracy in consideration of color reproduction of the light emitting object.

FIG. 2 is a block diagram showing the configuration of the image processing apparatus in the first embodiment. An example of a user interface in the first embodiment; The flowchart which shows the image processing in 1st Embodiment, The figure for explaining the luminous object judgment coefficient, The block diagram which shows the structure of the image processing apparatus in 2nd Embodiment, The flowchart which shows the image processing in 2nd Embodiment, It is a flowchart which shows the image processing in 3rd Embodiment.

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings. The following embodiments do not limit the present invention according to the claims, and all the combinations of features described in the present embodiments are not necessarily essential to the solution means of the present invention. Absent.

<First Embodiment>
In the present embodiment, a conversion process for reproducing an appearance under an observation light source different from an imaging light source is performed on a captured image obtained by capturing a scene in which a reflective object and a light emitting object are mixed with a multiband camera. At this time, by making the contribution of the light source different between the light emitting object region and the reflecting object region in the photographed image, it is possible to reproduce an accurate color appearance under the observation light source. Note that the image conversion process in the present embodiment is executed while a user inputs an instruction via a user interface (UI).

Apparatus Configuration FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to this embodiment. As shown in the figure, the image processing apparatus 100 of the present embodiment is roughly divided into an image processing unit 101 and a control unit 130, which are connected by a system bus 116.
In the image processing unit 101, 102 is an image input unit for inputting image data, and 103 is an input image storage unit for storing the input image data. Reference numeral 104 denotes a flash image storage unit that stores image data captured using a flash among image data input from the image input unit 102. Reference numeral 105 denotes an imaging light source input unit that inputs spectral radiance data (imaging light source information) of the imaging light source, and reference numeral 106 denotes an imaging light source storage unit that stores the imaging light source information. 107 is an observation light source storage unit that stores light source information to be reproduced in advance, and 122 is an observation light source selection unit that selects observation light source information. Reference numeral 108 denotes a conversion parameter input unit that inputs parameters necessary for the conversion process, and reference numeral 109 denotes a conversion parameter storage unit that stores the input conversion parameters. Reference numeral 110 denotes a light emission / reflection determination unit that determines a light emitting object and a reflection object with respect to image data stored in the input image storage unit 103. This determination is performed using the spectral radiance data of the photographic light source stored in the photographic light source storage unit 106 and the conversion parameters stored in the conversion parameter storage unit 109. Details thereof will be described later. Reference numeral 111 denotes a light emission / reflection determination result storage unit that stores a determination result by the light emission / reflection determination unit 110.

  An image conversion unit 112 performs image conversion processing on the image data stored in the input image storage unit 103, and a conversion image storage unit 113 stores the converted image data. Reference numeral 114 denotes an image output unit that outputs the determination result stored in the light emission / reflection determination result storage unit 111 and the converted image data stored in the converted image storage unit 113 to an image display device such as a display. Reference numeral 115 denotes a user interface (UI) unit for the user to instruct the operation of the image processing unit 101.

  In the control unit 130, the CPU 117 is involved in all the processing of each component, and sequentially reads and executes instructions stored in the RAM 118 and the ROM 119. The RAM 118 and ROM 119 provide the CPU 117 with programs, data, work areas, and the like necessary for the processing. The operation unit 120 is a keyboard or a mouse pointer, and inputs an instruction from the user to the image processing unit 101. The external storage device 121 is a storage device such as a hard disk.

  FIG. 2 is a diagram illustrating an example of a user interface in the UI unit 115. In the UI screen 201, 202 is an image reading button for instructing reading of a captured image to be processed, and 203 is a conversion parameter reading button for instructing reading of a conversion parameter. 204 is an imaging light source reading button for instructing reading of imaging light source information, 205 is an observation light source selection button for selecting an observation light source by displaying various light sources in a list, and 208 is a conversion processing execution button for instructing execution of image conversion processing. is there. 206 is a light emission / reflection determination button for classifying a photographed image into a light emitting object area and a reflection object area, 207 is a light emission / reflection determination result display section for displaying a light emission / reflection determination result, and 209 is an input image and a conversion result. It is an image display part which displays an image. Reference numeral 210 denotes a converted image storage button for instructing storage of an image of the conversion result.

Image Conversion Processing Hereinafter, image conversion processing in the present embodiment will be described using the flowchart of FIG.

  First, it waits until the image reading button 202 is pressed in S301, and if it is pressed, it proceeds to S302 and performs image input processing. In S302, an input image selection screen (not shown) is displayed on the UI unit 115, and for example, a list of image data held in the external storage device 121 is presented on the screen in a form that can be selected by the user. From the screen, the user selects both an image that is normally captured without using a flash under a photographing light source and an image that is captured using a flash under a photographing light source as input targets for the same scene. Then, the image input unit 102 stores the selected normal captured image and flash captured image in the input image storage unit 103 and the flash image storage unit 104, respectively. The input image in the present embodiment is a multiband image taken by a multiband camera, details of which will be described later.

  Next, it waits until the conversion parameter reading button 203 is pressed in S303, and if it is pressed, the process proceeds to S304. In S304, a conversion parameter selection screen (not shown) is displayed on the UI unit 115, and on this screen, for example, a model list of multiband cameras held in the external storage device 121 is presented to the user in a selectable form. From the screen, the user selects the model of the multiband camera that captured the normal captured image and the flash captured image. Then, a conversion parameter stored in advance in the external storage device 121 corresponding to the selected model is input by the conversion parameter input unit 108 and stored in the conversion parameter storage unit 109. Details of the conversion parameter will be described later.

  In step S305, the process waits until the photographing light source reading button 204 is pressed. In S306, a photographing light source information selection screen (not shown) is displayed on the UI unit 115, and for example, a list of photographing light source types held in the external storage device 121 is presented in a form that can be selected by the user. From the screen, the user selects the type of light source at the time of shooting a normal shot image. Then, photographing light source information stored in advance in the external storage device 121 corresponding to the selected light source type is input by the photographing light source input unit 105 and stored in the photographing light source storage unit 106.

  In step S307, the process waits until the type of light source to be converted is selected by the observation light source selection button 205. If selected, the process proceeds to step S308. In S308, observation light source information based on the selected light source type (eg, “fluorescent lamp”) is selected from the observation light source storage unit 107. Here, the imaging light source information read in S306 and the observation light source information selected in S308 are spectral radiance data sampled every 10 nm with respect to the visible light region 380-730 nm.

  In step S309, the process waits until the light emission / reflection determination button 206 is pressed. In S310, the light emission / reflection determination unit 110 determines, for each pixel of the normal captured image, whether or not the light emitting object emitting light in the scene belongs to the captured light emitting object region. Here, pixels that do not belong to the light emitting object region are regarded as belonging to the reflecting object region where the reflecting object is photographed. This determination is performed using a normal photographed image and a flash photographed image obtained by photographing the same scene stored in the input image storage unit 103 and the flash image storage unit 104, respectively. Details of the light emission / reflection determination process will be described later. The determination result is stored in the light emission / reflection determination result storage unit 111 and displayed on the light emission / reflection determination result display unit 207 in S311. As the display method, it is only necessary that the user can identify the light emitting object region and the reflecting object region in the photographed scene. Therefore, for example, it is conceivable to display only the detected light emitting object region brightly while displaying the entire normal captured image darkly.

  Next, in S312, the process waits until the conversion process execution button 208 is pressed, and if it is pressed, the process proceeds to S313. In S313, the image conversion unit 112 performs a conversion process on the normal captured image stored in the input image storage unit 103 so that the image is viewed under the observation light source selected in S307, and the conversion result is converted into a converted image. Store in the storage unit 113. Details of the image conversion process will be described later. In step S314, the converted image stored in the converted image storage unit 113 is displayed on the image display unit 209.

  In step S315, the process waits until the converted image save button 210 is pressed. If the button is pressed, the process advances to step S316, and the image output unit 114 stores the image data stored in the converted image storage unit 113 in the external storage device 121. .

Input Image Data In this embodiment, the image data input in S302 is assumed to be a multiband image taken by a multiband camera. Here, the multiband camera is a camera having four or more color filters having different spectral characteristics, and image data captured by the camera has four or more color information for each pixel. Hereinafter, it is assumed that the input image data in the present embodiment is an image (hereinafter referred to as a multiband image) taken by a multiband camera having six types of color filters.

Conversion Parameter The conversion parameter input in S304 in this embodiment is a conversion matrix for converting a multiband image acquired by a multiband camera into a spectral image having a spectral reflectance for each pixel, and a color filter. Spectral transmittance data. The spectral reflectance handled in this embodiment is 36-dimensional data sampled at 10 nm with respect to 380-730 nm in the visible light region.

Here, in order to convert a multiband image acquired by a multiband camera having six types of color filters into a spectral image, it is necessary to convert 6-dimensional data into 36-dimensional data for each pixel. Therefore, by performing conversion as in the following expression (1), conversion for each pixel from the multiband image to the spectral image becomes possible. In Expression (1), o represents a spectral image to be converted, and is composed of 36-dimensional spectral reflectance data o _{380 to} o ₇₃₀ . v is a multi-band image to be converted, consist multiband pixel value v ₁ to v ₆ six dimensions. G is a 36 × 6 transformation matrix. W is a white balance coefficient and has an effective coefficient of w _{1 to} w ₆ .

The white balance coefficient W is calculated by using the spectral data light (λ) of the photographic light source and the spectral transmittance filter _n (λ) (n is the filter number) of the color filter as shown in the following equation (2). The

That is, in S304, the conversion matrix G shown in Expression (1) and the spectral transmittance data filter _n (λ) (n = 1 to 6) of the color filter shown in Expression (2) are input as conversion parameters.

Light emission / reflection determination process The light emission / reflection determination process in S310 will be described in detail below. For the determination of the light emitting object region, a difference in pixel value between the normal captured image stored in the input image storage unit 103 and the flash captured image stored in the flash image storage unit 104 is used. Specifically, the light emitting object coefficient α indicating the degree of the light emitting object region is calculated for each pixel (x, y) by the following equation (3), and this is used as the determination result.

In Expression (3), p _input (x, y) is the luminance value of the pixel (x, y) in the normal captured image, and p _flash (x, y) is the luminance value of the pixel (x, y) in the flash captured image. is there. These luminance values are derived by performing a well-known integration process on each multiband image v. Further, p _max is a maximum luminance value in the normal captured image p _input (x, y).

  The luminescent object coefficient α calculated by Expression (3) takes a value of 0 ≦ α ≦ 1, and the closer α is to 1, the higher the degree that the pixel is a luminescent object region. Therefore, in S311, a pixel having a light emitting object coefficient α that is equal to or larger than a predetermined threshold in a normal captured image may be detected, set, and displayed as a light emitting object region.

Further, k in the expression (3) is a light emitting object determination coefficient for controlling the determination accuracy of the light emitting object. In this embodiment, this value is 0 ≦ k ≦ 1 according to the luminance value p _input of the normal photographed image. Set within the range. Here, FIG. 4 shows the value of the luminous object determination coefficient k with respect to the luminance value p _input of the normal captured image. In general, a light emitting object region in a normal captured image has a higher luminance than a reflective object region, and a light emitting object region can be regarded as a light emitting object region if it is a high luminance region of a certain level or more. Therefore, in this embodiment, as shown in FIG. 4, if the luminance value p _input of the normal captured image is a high luminance part having a predetermined value or more, the light emitting object determination coefficient k is set to zero. As a result, the light emitting object coefficient α calculated by the above equation (3) is unconditionally set to 1 in the high luminance portion. As shown in FIG. 4, the light-emitting object determination coefficient k is set to 1 if the luminance value p _input of the normal photographed image is a low luminance part that is equal to or less than a predetermined value. As a result, the light emitting object coefficient α calculated by the above equation (3) is set with high accuracy in the low luminance portion. That is, the detection accuracy of the light emitting object region in the low luminance part is improved. In addition, what is necessary is just to make it continuous between the low-intensity part and high-intensity part in a normal picked-up image so that the light emission object coefficient k may become small, so that a luminance value is large, as shown in FIG.

Image Conversion Processing Hereinafter, the image conversion processing in S313 will be described in detail. Here, first, the multiband image v input as a normal photographed image is converted into the above equation (1) using the conversion parameters (conversion matrix G, color filter spectral transmittance filter _n (λ)) input in S304. ) To convert to a spectral image o. Then, the spectral image o is converted into a linear sRGB RGB value under the observation light source by using the luminescent object coefficient α obtained by Equation (3) in S310. This conversion follows the following formula (4).

  According to the equation (4), the first reflection result reflecting the spectral radiance L0 of the photographing light source and the second reflecting the spectral radiance L1 of the observation light source to the spectral image o of the normal photographed image. The reflection results are synthesized at a rate corresponding to α. That is, for pixels that can be regarded as being within the light emitting object region because of the high light emitting object coefficient α, the contribution of the observation light source is reduced, and color reproduction that does not reflect the observation light source is performed. On the other hand, for the pixels that can be regarded as being outside the light emitting object region, that is, inside the reflecting object region because the light emitting object coefficient α is low, the contribution of the observation light source becomes large, and color reproduction reflecting the observation light source is performed.

  The input multiband image is converted into sRGB data by the above processing. When the sRGB data is further displayed on a display or the like, gamma correction processing or color matching processing according to the color characteristics of the display is performed. Just do it.

  As described above, according to this embodiment, when a light-emitting object is included in a scene photographed using a multiband camera, control is performed so that the observation light source is not reflected in the light-emitting object region. be able to. Therefore, it is possible to accurately reproduce the appearance of an image of a scene in which a light emitting object and a reflective object are mixed, under an observation light source that is different from that of the image light source.

  In this embodiment, an example is shown in which the user performs the conversion process while instructing via the UI unit 115, but the conversion process is performed while automatically selecting each file without inputting the user instruction. Also good.

Second Embodiment
Hereinafter, a second embodiment according to the present invention will be described. In the first embodiment described above, an example in which an image conversion process is performed while a user performs an instruction via the UI unit 115 has been described. In general, in a dark scene, a flash may be used to secure a light amount. However, when the spectral radiances of the flash and the photographing light source are different, a light source conversion process is required for the flash image in order to reproduce the appearance under the photographing light source. At this time, if a luminescent object is handled in the same manner as a reflective object in the scene, it will be converted into an appearance different from the original as described above. In the second embodiment, an example in which automatic conversion processing is performed in an image input device such as a digital camera without using a user instruction will be described. Further, as this automatic conversion processing, an example is shown in which a flash photographed image is converted into a normal photographed image in consideration of the luminescent object region.

  FIG. 5 is a block diagram illustrating a configuration of a digital camera (hereinafter referred to as a camera) which is an image input apparatus according to the second embodiment. In the first embodiment described above, the same components (image processing unit 101 and the like) as those shown in FIG. In the camera 501 shown in FIG. 5, reference numeral 502 denotes an image pickup unit including an optical lens, 503 denotes a flash, 504 denotes a photographing button, and 505 denotes a sensor in which four or more types of color filters are arranged. A control unit 506 controls the entire camera 501, and a signal processing unit 507 performs signal processing on data acquired by a shooting operation. Reference numeral 508 denotes an external interface that controls an external device such as the external storage device 121, a spectral radiance meter 509 that acquires the spectral radiance of the imaging light source, and the like.

  Hereinafter, image conversion processing in the camera 501 of the second embodiment will be described with reference to the flowchart of FIG. In step S601, the camera 501 waits until the shooting button 504 is pressed. In S602, the control unit 506 operates the imaging unit 502 and the sensor unit 505 to perform shooting. In S603, the image captured in S602 is developed by performing signal processing such as demosaicing processing and noise reduction processing in the signal processing unit 507, and the input image storage unit 103 in the image processing unit 101 is processed. To store. In step S604, the spectral radiance of the photographing light source is input from the spectral radiance meter 509 as photographing light source information.

  In step S605, the control unit 506 operates the flash 503, the imaging unit 502, and the sensor unit 505 to perform shooting. In S606, the image captured in S605 is developed by performing signal processing in the signal processing unit 507 and stored in the flash image storage unit 104. In step S607, the spectral radiance of the flash 503, which is stored in advance in the apparatus, is input as flash information.

  In S608, the conversion parameter is read and stored in the conversion parameter storage unit 109, as in S304 in the first embodiment described above. However, here, conversion parameters preset in the camera 501 are read.

  Next, in S609, the flash photographed image photographed with the flash is used as the input image (normal photographed image in the first embodiment) to reproduce the appearance under the photographing light source by the same method as in the first embodiment described above. Convert to image. That is, in the second embodiment, the spectral radiance of the imaging light source acquired in S604 is treated as the spectral radiance of the observation light source in the first embodiment, and the spectral radiance of the flash 503 is similarly used for the imaging light source in the first embodiment. Treat as spectral radiance.

  Specifically, in S609, based on the difference between the pixel values of the normal captured image and the flash captured image, a light emitting object region corresponding to the light emitting object that emits light in the scene is detected from the flash captured image. Then, when converting the flash photographed image so as to reproduce the appearance under the photographing light source, control is performed so that the degree of reflection of the spectral radiance of the photographing light source differs between the light emitting object region and the other reflecting object region. Thus, conversion is performed so that the photographic light source is not reflected on the light emitting object region.

  Finally, in S610, the image converted in S609 is stored in the external storage device 121 via the external interface 508.

  As described above, according to the second embodiment, an image photographed using a flash is converted into an image photographed under a photographing light source without using a flash, taking into consideration a light emitting object region in the scene. It becomes possible. That is, in the converted image, the influence of the photographing light source is reduced in the light emitting object region.

<Third Embodiment>
The third embodiment according to the present invention will be described below. In the first embodiment described above, an example in which a light emitting object area is automatically determined is shown, but in the third embodiment, an example in which the user specifies the area is shown. Since the apparatus and user interface of the image processing apparatus in the third embodiment are substantially the same as those in the first embodiment described above, the description thereof is omitted.

  Hereinafter, the image conversion processing in the third embodiment will be described in detail with reference to the flowchart of FIG. First, the process waits until the image reading button 202 is pressed in S701, and if it is pressed, the process proceeds to S702. In S <b> 702, the image input unit 102 selects a normal photographed image photographed under a photographing light source as input image data and stores it in the input image storage unit 103.

  Next, in S703, the process waits until the conversion parameter reading button 203 is pressed, and if it is pressed, the process proceeds to S704. In S704, as in S304 of the first embodiment, the conversion parameter selected by the user is input by the conversion parameter input unit 108 and stored in the conversion parameter storage unit 109. In S705, the process waits until the photographing light source reading button 204 is pressed, and if it is pressed, the process proceeds to S706. In S706, as in S306 of the first embodiment, the photographic light source input unit 105 inputs the photographic light source information selected by the user and stores it in the photographic light source storage unit 106. Next, in S707, it waits for a light source to be selected by the observation light source selection button 205, and if selected, proceeds to S708. In S708, the observation light source selection unit 122 selects observation light source information based on the selected light source from the observation light source storage unit 107.

  Next, in S709, first, regarding the normal captured image stored in the input image storage unit 103, the entire image area is regarded as a reflecting object, and image conversion processing using an observation light source is performed. This image conversion process is the same as the image conversion process for reproducing the appearance under the observation light source in S313 of the first embodiment. That is, the image conversion process is performed by setting the luminescent object coefficient α to 0 for all pixels. . In step S710, the image converted in step S709 is displayed on the image display unit 209. In step S711, the process waits until the light emitting object area is specified by the user. If the light emitting object area is specified, the process proceeds to step S712. Although this designation method is not particularly defined, for example, it is conceivable to designate an area on the displayed image using a pointing device or the like.

  In S712, the light emitting object coefficient α is set to 1 for the pixels in the light emitting object area designated in S711, and the light emitting object coefficient α is set to 0 for the pixels outside the light emitting object area as a reflecting object area.

  Next, in S713, the process waits until the conversion process execution button 208 is pressed, and if it is pressed, the process proceeds to S714. In S714, the image conversion unit 112 performs a conversion process on the normal captured image stored in the input image storage unit 103 so that the image is viewed under the observation light source selected in S707, and converts the conversion result. Stored in the image storage unit 113. In S715, the converted image stored in the converted image storage unit 113 is displayed on the image display unit 209. In step S716, the process waits until the converted image save button 210 is pressed. If the button is pressed, the process advances to step S717, and the image output unit 114 stores the image data stored in the converted image storage unit 113 in the external storage device 121. To do.

  As described above, according to the third embodiment, when the user designates the light emitting object region, the light emitting region and the reflective region in the normal photographed image can be distinguished and processed without acquiring the flash photographed image. .

<Other embodiments>
In the first and second embodiments described above, the example in which the light emitting object area and the reflective object area in the shooting scene are determined based on the absolute value of the difference between the flash photographed image and the normal photographed image is shown. The method is not limited to this example. For example, when a white balance coefficient different from that of the photographing light source is detected in a high luminance region in the photographing scene, a method of discriminating the region as a light emitting object region is also conceivable.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

An image input means for inputting a photographed image photographed by a multiband camera having four or more color filters under a photographing light source;
Means for acquiring photographing light source information for obtaining spectral radiance of the photographing light source;
Observation light source information acquisition means for acquiring a spectral radiance of an observation light source different from the imaging light source;
In the captured image, setting means for setting a light emitting object region corresponding to a light emitting object that is emitting light in the scene;
By combining the first reflection result that reflects the spectral radiance of the imaging light source in the captured image and the second reflection result that reflects the spectral radiance of the observation light source in the captured image, the imaging Conversion means for converting the image so as to reproduce the appearance under the observation light source,
The image processing apparatus according to claim 1, wherein the conversion unit has different ratios for combining the first and second reflection results between the light emitting object region and the other region.   The conversion means is configured such that the ratio of the second reflection result combined with the light emitting object area is smaller than the ratio of the second reflection result combined with pixels outside the light emitting object area. The image processing apparatus according to claim 1, wherein the conversion process is performed as described above. The image input means inputs a flash photographed image taken with the flash on the scene under the photographing light source and a normal photographed image taken without the flash,
The said setting means detects and sets the said light emission object area | region in this normal captured image based on the difference of the pixel value in the said normal captured image and the said flash captured image, The setting of Claim 1 or 2 characterized by the above-mentioned. Image processing device. Furthermore, it has a parameter acquisition means for acquiring a conversion parameter for converting the captured image input by the image input means into spectral reflectance data,
The conversion unit converts the captured image into spectral reflectance data using the conversion parameter, and performs a conversion process so that the spectral radiance of the imaging light source and the observation light source is reflected in the spectral reflectance data. The image processing apparatus according to claim 1, wherein the image processing apparatus is performed.   The image processing apparatus according to claim 1, wherein the setting unit sets the light-emitting object region in accordance with a user instruction for the captured image. Image input for entering the same scene under a shooting light source with a multi-band camera that has four or more color filters and shooting with a flash and a normal shot with a flash. Means,
Means for acquiring photographing light source information for obtaining spectral radiance of the photographing light source;
Flash information acquisition means for acquiring the spectral radiance of the flash;
Detecting means for detecting a light emitting object region corresponding to a light emitting object emitting light in the scene in the flash photographed image based on a difference between pixel values in the normal photographed image and the flash photographed image;
By combining the first reflection result reflecting the flash spectral radiance in the flash photographed image and the second reflection result reflecting the spectral radiance of the photographing light source in the flash photographed image, Conversion means for converting a flash image so as to reproduce the appearance under the photographing light source,
The image processing apparatus according to claim 1, wherein the conversion unit has different ratios for combining the first and second reflection results between the light emitting object region and the other region. An image processing method in an image processing apparatus having an image input means, an imaging light source information acquisition means, an observation light source information acquisition means, a setting means, and a conversion means,
The image input means inputs an image captured by a multiband camera having four or more color filters under an imaging light source; and
The acquisition means of the imaging light source information, the acquisition step of imaging light source information for acquiring the spectral radiance of the imaging light source,
The observation light source information acquisition means acquires observation light source information for acquiring spectral radiance of an observation light source different from the imaging light source, and
A setting step in which the setting means sets a light emitting object region corresponding to a light emitting object that emits light in the scene in the captured image;
The conversion unit synthesizes a first reflection result that reflects the spectral radiance of the imaging light source in the captured image and a second reflection result that reflects the spectral radiance of the observation light source in the captured image. A conversion step for converting the photographed image so as to reproduce the appearance under the observation light source,
In the conversion step, the ratio of combining the first reflection result and the second reflection result is different between the light emitting object region and the other region.   A program for causing a computer device to function as each unit of the image processing device according to any one of claims 1 to 6 when executed on the computer device.

Images