JP2016139927A - Image processing apparatus, image processing method, image processing program and imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To ensure an effect for enhancing the reproducibility of color, even in a scene where correct exposure cannot be obtained.SOLUTION: An image processing apparatus includes an image acquisition unit for acquiring a plurality of images of different brightness for the same subject, an image storage unit for storing the plurality of images, a first determination unit performing color component determination of a predetermined object region in an image, by using an image of first brightness out of the plurality of images stored in the image storage unit, a second determination unit performing region determination of the object region, by using an image of second brightness out of the plurality of images stored in the image storage unit, and an image correction unit for generating a correction image, where the colors of color component determination results are applied to the object region in one image, out of the plurality of images, on the basis of the determination results from the first and second determination units.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing program, and an imaging apparatus that improve color visibility or reproducibility.

  In recent years, portable devices (imaging devices) with a photographing function such as digital cameras have become widespread. Some image pickup apparatuses of this type include a display unit and have a function of displaying a captured image. Further, there is a display screen that displays a menu screen to facilitate the operation of the imaging apparatus.

  In some imaging apparatuses, various settings at the time of shooting are automated. For example, some digital cameras and the like are equipped with an AF function that automates focusing and an automatic exposure (AE) function that automates exposure. Some imaging apparatuses have an auto white balance function that automates white balance correction. For example, Patent Document 1 discloses an apparatus that improves the color reproducibility of a subject by determining the type of light source and performing white balance control.

JP 2010-50740 A

  However, conventionally, an imaging apparatus may have a low color reproducibility depending on the subject and the exposure state. The imaging device controls exposure by measuring an average reflected light amount in a photographing range, a reflected light amount from a subject of interest, and the like. Therefore, depending on the distribution of brightness within the shooting range, appropriate exposure may not be obtained throughout the shooting range. For example, when shooting night scenes with dark backgrounds and buildings mixed with bright neon lights and windows, the brightness difference between the light source parts such as neon and light windows and other parts is large, so the light source part is saturated white. May be taken. In addition, the light source part may be blurred or blurred, the shape is not clear, the size is large, and the color may not be reproduced at all. The same thing happens in places where light hits the spot or where the light does not reach.

  In addition, not only night scenes but also scenes where proper exposure cannot be obtained over the entire shooting range, there is a problem that colors and shapes may not be reproduced correctly depending on the state of exposure. there were.

  An object of the present invention is to provide an image processing device, an image processing method, an image processing program, and an imaging device that can improve the visibility or reproducibility of colors even in a scene where proper exposure cannot be obtained. .

  An image processing apparatus according to the present invention includes an image acquisition unit that acquires a plurality of images with different brightness for the same subject, an image storage unit that stores the plurality of images, and a plurality of images stored in the image storage unit. A first determination unit that performs color component determination of a predetermined target area in the image using an image having the first brightness, and a second brightness among the plurality of images stored in the image storage unit. A second determination unit that performs region determination of the target region using the image of the target, and the target region in one of the plurality of images based on the determination results of the first and second determination units And an image correction unit for generating a corrected image to which the color resulting from the color component determination is applied.

  In addition, an imaging apparatus according to the present invention includes an imaging unit that can capture a plurality of images with different brightness for the same subject, an image storage unit that stores the plurality of images from the imaging unit, and the image storage unit. A first determination unit that performs color component determination of a predetermined target region in the image using an image having a first brightness among the plurality of stored images, and a plurality of images stored in the image storage unit. Of these, based on the determination results of the second determination unit that performs region determination of the target region using the second brightness image, and the determination results of the first and second determination units, one of the plurality of images And an image correction unit that generates a corrected image in which the color as a result of the color component determination is applied to the target region in the image.

  The image processing method according to the present invention includes an image acquisition step for acquiring a plurality of images with different brightness for the same subject, an image storage step for storing the plurality of images, and a plurality of images stored in the image storage step. A first determination step for determining a color component of a predetermined target region in the image using an image having the first brightness among the images of the second image, and a second of the plurality of images stored in the image storage step Based on the determination result of the second determination step for performing the region determination of the target region using an image of brightness and the determination results of the first and second determination steps, the above-mentioned one of the plurality of images And an image correction step of generating a corrected image in which the color resulting from the color component determination is applied to the target area.

  In addition, an image processing program according to the present invention stores in a computer an image acquisition step of acquiring a plurality of images with different brightness for the same subject, an image storage step of storing the plurality of images, and the image storage step. A first determination step for determining a color component of a predetermined target area in the image using an image having a first brightness among the plurality of images, and among the plurality of images stored in the image storage step One image of the plurality of images based on the second determination step for determining the region of the target region using the second brightness image and the determination results of the first and second determination steps. And an image correction step of generating a corrected image in which the color resulting from the color component determination is applied to the target region.

  According to the present invention, there is an effect that the visibility or reproducibility of colors can be improved even in a scene where the luminance difference of each part of the image is large.

1 is a block diagram showing a circuit configuration of an imaging apparatus incorporating an image processing apparatus according to a first embodiment of the present invention. Explanatory drawing for demonstrating the outline | summary of the determination method of the color in this Embodiment, and the determination method of a shape. Explanatory drawing for demonstrating the outline | summary of the determination method of the color in this Embodiment, and the determination method of a shape. 4 is a flowchart for explaining processing of the image processing apparatus 20; 4 is a flowchart for explaining processing of the image processing apparatus 20; 6 is a flowchart for explaining color determination and area determination processing in the imaging apparatus 1 for a light spot. The flowchart which shows the operation | movement flow employ | adopted in the 2nd Embodiment of this invention. The flowchart which shows the operation | movement flow employ | adopted in the 3rd Embodiment of this invention. It is explanatory drawing for demonstrating this Embodiment. It is explanatory drawing for demonstrating this Embodiment. It is explanatory drawing for demonstrating this Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a circuit configuration of an imaging apparatus incorporating an image processing apparatus according to the first embodiment of the present invention. The imaging apparatus 1 in the present embodiment acquires a plurality of images obtained by imaging while changing the exposure of the same subject (hereinafter referred to as images having different exposures) or a plurality of images having different brightness with respect to the same subject, Select images that can be used to determine the color or shape of a part of the image from images with different exposure or brightness, determine the color or shape, and perform image composition based on the determination result to cover the entire image. Thus, an image with improved color reproducibility or visibility and shape reproducibility is obtained.

  The color is determined by the RGB signal ratio and signal amount. In this specification, the ratio of the RGB signal that cannot be reproduced if lost especially at the time of acquisition is expressed as “color” or “color component”. The signal amount is explained by assuming that the degree of freedom of reproduction is greater by correction and replacement such as amplification. In addition, the color can be determined by determining the ratio of the spectral components, and “color” or “color component” in the specification is obtained by various signals that obtain the ratio of other spectral components as well as the RGB signals. Means. Depending on screen characteristics and image conditions, the signal amount at the time of color reproduction may be corrected to a standard signal amount or a good signal amount for each color. In an easy-to-understand example, the signal amount may be adopted for warnings, instructions, icons indicating switches, subtitles, etc. on the display screen. When icons or subtitles are displayed in superimpose, it is known that the appearance may deteriorate depending on the background. In this case, the icons may be adjusted in brightness.

First, referring to the explanatory diagrams of FIGS. 2 and 3, an outline of the color determination method (in particular, the RGB ratio as described above, but the signal amount may be determined) and the shape determination method in the present embodiment will be described. To do.
FIG. 2 shows a captured image obtained by photographing a night scene with a predetermined exposure amount. The image 41 includes a sky image portion 42, a star image portion 43 indicated by an asterisk, a building image portion 44, and a light spot image portion 45 caused by light or neon leaking from a building window or the like. In a relatively dark night scene, the difference between the appropriate exposure value suitable for expressing the entire image (for example, the appropriate exposure value for the night scene) and the appropriate exposure value for expressing the color of each position in the image is compared. Big. For example, in the night view mode of a digital camera, shooting is performed with a relatively large exposure amount so that the shape of a building or the like can be visually recognized. As a result, the image portion 45 of the bright light spot is saturated with the image signal, and becomes white regardless of the actual color of light, blurring occurs, the size becomes larger than the actual size, and the shape cannot be discriminated.

  Note that the light spot is, for example, a point (part) where the light from the lamp that illuminates the room leaks from the window, but the illumination light from the lamp may be directly imaged. In this case, the light spot is The light source itself.

  FIG. 3 shows changes in the charge accumulation amount of the image sensor of the digital camera that obtains the image 41 of FIG. 2 with the horizontal axis representing the time for each predetermined time interval and the vertical axis representing the charge accumulation amount. 3A corresponds to the central portion of the image portion of the predetermined light spot in the image 41, and FIG. 3B corresponds to the peripheral portion of the image portion of the light spot in FIG. 3A. FIG. 3 shows the charge accumulation amount for each of the color components RGB in the four times indicated by the exposure amounts A to D.

  For each color component RGB, the amount of charge accumulation increases with time and the brightness changes, but the color can be obtained from the ratio between the color components RGB. If the ratio of the color components RGB cannot be obtained correctly, the colors cannot be reproduced correctly. Actually, it cannot be said that the color is correct only by the RGB ratio, and it is necessary to reproduce the signal amount such as brightness, but it is simplified to assume a color with a standard signal amount (RGB signal ratio). doing. In the example of FIG. 3A, it is shown that the charge accumulation amount of the R component at the light spot exceeds the saturation amount allowed for the image sensor in the time of the exposure amount D. For this reason, in the time of the exposure amount D, the ratio between the color components RGB obtained by the image sensor collapses, and the original color cannot be reproduced. In other words, if none of the accumulated charge amounts of the color components RGB reaches the saturation amount, the color can be determined even at the light spot. Therefore, in the present embodiment, for color reproduction, for example, an image with a relatively low exposure amount is selected and color determination is performed. For example, by obtaining the component ratio of the color component RGB at the time of the exposure amount C, it is possible to determine the color specific to the light spot. In consideration of the influence of noise, it is better to perform color determination with an exposure amount equal to or greater than a predetermined exposure amount.

  The RGB ratio changes depending on the saturation, but if the exposure is changed even if the saturation is constant, the impression will be different even if the same color is bright or dark, so the exposure or signal amount (standard signal amount) appropriate for the hue. ) May also be determined. If the saturation changes according to the exposure, the color may be determined based on, for example, the ratio of the color components RGB obtained at the highest saturation, that is, the exposure amount when the ratio of the color components RGB is the largest. The ratio of RGB at the exposure amount having the largest difference from the noise level may be used. Of course, color determination (color component determination) may be performed with reference to the saturation and exposure of other parts, and such an idea makes it possible to express an image with less unevenness.

  In the present embodiment, the color reproducibility is improved by applying the determined color to the original image portion. For example, the determined color may be filled (with a standard signal amount, or at a level that matches the surrounding signal amount and exposure amount), and gradation may be applied toward the periphery of the original image portion. .

  In addition, in order to apply the determined color to the image portion of the light spot, the region to which the determined color is to be applied, that is, the region in the image of the image portion of the light spot (hereinafter referred to as the light spot region) is determined. There is a need to. In order to detect the light spot region, an image having an exposure or brightness suitable for recognizing the shape is required.

  Depending on the scene, the brightness is not uniform and has a predetermined distribution in the light spot region. In a relatively dark region in the light spot region, it is conceivable that sufficient brightness cannot be obtained with the exposure amount used for color determination (color component determination), and the shape cannot be determined. FIG. 3B shows the charge accumulation amount in the peripheral region in such a light spot region. With the exposure amount (signal amount) C, a sufficient charge accumulation amount is obtained so that the shape can be accurately determined. It shows no.

  While the center of the light spot is relatively bright, it gradually becomes darker as it deviates from the center, and the surrounding area may not necessarily be bright. For this reason, if the exposure amount is too low, it is conceivable that only the vicinity of the light spot center is detected and the entire light spot is not detected. Thus, it is conceivable that the shape does not appear correctly in an image with a relatively low exposure amount. In order to detect the shape of the light spot (light spot region), the charge accumulation amount at the center of the light spot may be saturated, so an image picked up with a sufficiently high exposure amount is selected. For example, in the example of FIG. 3, an image at the exposure amount D at which a sufficient charge accumulation amount is obtained is used for determining the light spot region.

  If the amount of exposure is too high, the image area of the light spot may be detected wider than it actually is, or it may be detected in an area overlapping with other light spot parts. In this case, it is necessary to set an appropriate exposure amount.

  As described above, in the example of FIG. 3, an image with the exposure amount C is used for color determination (color component determination), and an image with the exposure amount D is used for light spot region determination. For example, the light spot region is determined from the image with the exposure amount D, and the color determined using the image with the exposure amount C is applied to the light spot region of the image with the appropriate exposure amount as the entire image. As a result, it is possible to obtain an image that is relatively clear as a whole and clear in color and shape at the light spot.

  The above description can be applied not only when an image is captured by a digital camera, but also when an image captured from the outside is edited. That is, instead of capturing and acquiring a plurality of images with different exposure amounts for the same subject, it is only necessary to acquire images with different brightness for the same subject. For example, the color can be determined based on the ratio of the RGB components of the light spot portion from a relatively dark image, and the region of the light spot portion can be determined from a relatively bright image. Thus, the color and shape of the light spot region can be obtained by embedding the determined color in the image portion of the light spot region with a good-looking brightness (either a standard signal amount or a signal amount determined by a balance with other parts of the image). Can be obtained.

  In FIG. 1, the imaging device 1 includes an image processing device 20 and an imaging unit 10. The image processing apparatus 20 is provided with a captured image from the imaging unit 10. The image acquisition unit 21 captures an image from the imaging unit 10. The image acquisition unit 21 is provided with an imaging control unit 22, and the imaging control unit 22 controls the imaging unit 10.

  The imaging unit 10 includes an imaging device (not shown) such as a CCD or CMOS sensor, and an optical system (not shown) that guides an optical image of a subject to the imaging surface of the imaging device. The optical system includes a lens and a diaphragm for zooming and focusing.

  The imaging control unit 22 drives and controls the imaging element and the optical system of the imaging unit 10. For example, the imaging control unit 22 can change a zoom and focus position and an aperture by driving a lens and an aperture for zooming and focusing. In addition, the imaging control unit 22 can drive and control the imaging device of the imaging unit 10 and acquire a captured image from the imaging device. The imaging control unit 22 can also add and thin out the pixel signals as necessary when outputting the pixel signals from the imaging device.

  Furthermore, in the present embodiment, the imaging control unit 22 controls the shutter speed, exposure time, and the like of the imaging element, and captures a plurality of captured images that are captured with different exposure amounts for the same subject. 10 can be imaged. In this way, the image acquisition unit 21 acquires a plurality of captured images with different exposure amounts. Even when the shutter speed and the exposure time are fixed, the brightness of the obtained image varies depending on the presence or absence of pixel addition and the like, and this embodiment includes such a case. Some image sensors have a sensitivity switching function, but such a method may be used.

  Note that, as described above, the image processing device 20 only needs to be able to acquire a plurality of images with different brightness for the same subject. For example, a plurality of images captured with different exposure amounts can be obtained from a communication unit or a memory (not shown). You may come to get. In this case, the imaging unit 10 in FIG. 1 can be omitted, and the present embodiment can be configured only by an image processing apparatus in which the imaging control unit 22 that controls the imaging unit 10 is omitted.

  The plurality of images acquired by the image acquisition unit 21 are given to the image storage unit 23. The image storage unit 23 stores a plurality of input images. That is, the image storage unit 23 can store a plurality of images with different exposure amounts or brightnesses for the same subject. The image storage unit 23 outputs the stored image to the first determination unit 24 and the second determination unit 25 and also outputs to the image correction unit 27. In addition, since images obtained by imaging with different exposure amounts have different brightness as a result, hereinafter, images with different exposure amounts are described as brightness images including different exposure amounts or brightness. Description of images with different lengths may be omitted.

  The image processing apparatus 20 is provided with a control unit 26. The control unit 26 can be configured by a processor such as a CPU (not shown) that performs control according to a program stored in a memory (not shown), for example, and controls each unit of the image processing apparatus 20. Further, each other circuit unit of the image processing apparatus 20 may be configured by one or a plurality of processors (not shown).

  The image correction unit 27 can also perform predetermined signal processing, such as color adjustment processing, matrix conversion processing, noise removal processing, and other various signal processing, on the input image. The image correction unit 27 can perform an image composition (image correction) process, which will be described later, on the input captured image and output the processed image to the display control unit 29. Note that the image correction unit 27 can also directly apply the input image to the display control unit 29 without performing image correction on the input image.

  The display control unit 29 executes various processes related to display. The display control unit 29 can give the given image to the display unit 31 for display. The display unit 31 has a display screen such as an LCD, and displays an image given from the display control unit 29. In addition, the display control unit 29 can display various menu displays on the display screen of the display unit 31.

  Note that the display screen of the display unit 31 is provided along the rear surface of the housing of the imaging device 1, and the photographer may check a through image displayed on the display screen of the display unit 31 at the time of shooting. It is also possible to perform a shooting operation while confirming a through image.

  The operation determination unit 28 receives user operations on the operation unit 32 including various buttons such as a shutter button, a function button, and a shooting mode setting (not shown). A touch panel 31 a is provided on the display screen of the display unit 31. The touch panel 31a can generate an operation signal corresponding to a position on the display screen pointed by the user with a finger or a slide operation. This operation signal is also supplied to the operation determination unit 28. The operation determination unit 28 determines an operation on the operation unit 32 and the touch panel 31 a and gives a determination result to the control unit 26. In this way, the control unit 26 can control each unit including the image correction process in accordance with a user instruction.

  In the present embodiment, the first determination unit 24 performs color determination (color component determination) on each part of the input image and outputs the determination result to the image correction unit 27. The second determination unit 25 performs shape determination on each part of the input image, and outputs a determination result indicating the region of the shape part in the image (hereinafter referred to as a target region) to the image correction unit 27. It has become. For example, the first determination unit 24 performs light spot color determination and outputs a determination result, and the second determination unit 25 performs light spot region determination and outputs the determination result.

  For example, the first and second determination units 24 and 25 are suitable for determining the color and shape of an image portion where the exposure from the image storage unit 23 is not appropriate or the image portion where the brightness is not appropriate. Judgment is performed using an image of the exposure amount or brightness. Note that the first and second determination units 24 and 25 may select an image with an exposure amount or brightness suitable for determination based on, for example, the level of the RGB components, and are designated by the control unit 26. Also good. For example, the control unit 26 may control the first and second determination units 24 and 25 so as to use an image having a preset exposure amount for the determination.

  The first determination unit 24 may determine the brightness distribution in the image at the same time as the color determination. Similarly, the second determination unit 25 may determine the brightness distribution at the same time when determining the target region.

  In the present embodiment, the image correction unit 27 is given the determination results of the first determination unit 24 and the second determination unit 25, and performs the second determination unit 25 on the image input from the image storage unit 23. The corrected image is generated by correcting the image by applying the color determined by the first determination unit 24 (or the appropriate brightness based thereon) for the target region determined by the above. For example, the image correction unit 27 acquires an image captured with appropriate exposure as the entire image from the image storage unit 23 or an image with appropriate overall brightness as an original image, and the exposure in the original image is A correction image is generated by filling the target area with a color pattern obtained by the first determination unit 24 for the target area, which is an inappropriate image part or an image part whose brightness is not appropriate. In addition, the image correction unit 27 may perform gradation in accordance with the brightness distribution of the target area (but with good color visibility) when filling the target area. Note that the color visibility is not determined only by the ratio of the color components such as RGB, but the brightness that looks good depends on the subject, but the brightness of the icon display that is displayed on the display section is also helpful. Alternatively, it may be determined for each color, or may be adjusted according to the brightness of the surroundings or the brightness of the entire image. The image correcting unit 27 gives the generated corrected image to the recording unit 33 for recording. Note that the image correction unit 27 may also give the image before correction given from the image storage unit 23 to the recording unit 33 as it is for recording.

  As the recording unit 33, various recording media can be employed. For example, a card interface is employed as the recording unit 33, and an image can be recorded on a recording medium such as a memory card.

  Next, the operation of the embodiment configured as described above will be described with reference to FIGS. 4 and 5 are flowcharts for explaining the processing of the image processing apparatus 20. FIG. 6 is a flowchart for explaining color determination and area determination processing in the imaging apparatus 1 for a light spot.

  The image acquisition unit 21 sequentially acquires a plurality of images with different exposure or brightness for the same subject, gives them to the image storage unit 23, and stores them. In step S1 of FIG. 4, the first determination unit 24 obtains an image having an exposure or brightness suitable for determining the color (RGB component, color component) of the light spot from the image storage unit 23, and in step S2. Color determination (color component determination) is performed using the acquired image. The first determination unit 24 gives the determination result of the color determination to the image correction unit 27.

  On the other hand, in step S3, the second determination unit 25 acquires an image having an exposure or brightness suitable for determining the light spot region from the image storage unit 23, and uses the image acquired in step S4. The area is determined. The second determination unit 25 gives the light spot region determination result to the image correction unit 27.

  The image correction unit 27 acquires an image having an appropriate exposure or brightness as the original image from the image storage unit 23 as a whole. The image correction unit 27 sets the light spot area in the original image based on the determination result of the second determination unit 25 to the color (determined color component and appropriate brightness) based on the determination result of the first determination unit 24. A corrected image filled with (having) is generated (step S5).

  This corrected image is an image captured with appropriate exposure or an appropriate brightness as the entire image, and is an image that is relatively close to the actual appearance as the entire image. People have a wide visual dynamic range, and color recognition with a sense of changing every moment, or correction with information in the brain, so it tends to capture images with impressive colors, due to saturation and blackout It is easy to feel a sense of incongruity with the color reproduction and visibility of the image, and it is easy to feel a vivid color. Devise measures to meet these needs. That is, here, the light spot region has a color similar to the color actually recognized by a person because the color is detected from an image with appropriate exposure or brightness. In addition, since the position, shape, and size of the light spot region are detected from an image with appropriate exposure or brightness, the light spot region has an appropriate position, shape, and size. For example, the light spot region may be too large or blurring. The shape and size is similar to the shape and size that a person actually feels. People tend to perform recognition correction similar to the color description described above with respect to shape and size.

  Note that the result of color component determination changes slightly by changing the exposure amount or brightness of the image used to determine the color component, so the exposure amount or brightness of the image used for color determination is controlled according to the user's sense. You may be able to do that. Similarly, by changing the exposure amount or brightness of the image used for determination of the target region, the size and shape of the target region also change somewhat. Therefore, emphasis and correction may be performed according to the user's feeling. In consideration of this, the exposure amount or brightness of an image used for region determination may be controlled.

  The example of FIG. 4 shows an example in which the target area is simply painted with the determined color (color component (component ratio) and brightness (signal amount)). FIG. 5 shows an example in which the determined area is applied with gradation according to the brightness distribution by applying the determined color to the target area while applying gradation. In FIG. 5, the same steps as those in FIG.

  In step S11 of FIG. 5, the first determination unit 24 determines the light amount distribution for each part of the image. The first determination unit 24 gives the determination result of the color and the determination result of the light amount distribution to the image correction unit 27. The image correction unit 27 fills the light spot area in the original image based on the determination result of the second determination unit 25 with a color based on the determination result of the first determination unit 24 with a gradation according to the light amount distribution. A corrected image is generated (step S12).

  Thus, in the example of FIG. 5, the light spot region is filled with the determined color and gradation according to the determined light amount distribution, so that an image closer to the user's sense is obtained. be able to. The determination of the light amount distribution may be performed by the first determination unit 24 or the second determination unit 25.

  4 and 5 have been described on the assumption that the image acquisition unit 12 acquires a plurality of images with different exposure or brightness for the same subject and stores them in the image storage unit 23. FIG. 6 is a diagram for explaining an operation in a case where such a series of processing of image acquisition, color and target region determination, and correction image generation is performed by the imaging apparatus of FIG. 1 configured by a digital camera or the like. It is.

  FIG. 6 shows an example in which an image sensor capable of nondestructive readout is basically employed. By using the output of the image sensor capable of nondestructive reading, an image for each predetermined exposure amount can be acquired in the middle of obtaining an image with an appropriate and sufficient exposure amount for the entire image. However, even when an image sensor that cannot perform non-destructive readout is used, a function for obtaining an image for each predetermined exposure amount (hereinafter referred to as a light valve) by repeating exposure and readout a plurality of times and integrating image signals for each readout. By adopting the function, reading similar to nondestructive reading is possible, and the description of FIG. 6 also includes this case.

  When the user presses the shutter button to instruct imaging, the imaging control unit 22 controls the imaging unit 10 to start exposure. The imaging control unit 22 performs exposure by setting a predetermined exposure time (step S21). Thereby, imaging with a predetermined exposure amount is started. The imaging control unit 22 reads a pixel signal from the imaging device when the exposure time for obtaining a predetermined exposure amount is reached. Thus, the image acquisition unit 12 obtains an image at a predetermined exposure amount. This image is supplied to the image storage unit 23 and is supplied to the first determination unit 24 and the second determination unit 25.

  In step S22, the imaging control unit 22 determines whether or not the appropriate exposure amount of the entire image has been reached. If it is determined that the appropriate exposure amount has been reached, the process is terminated.

  The first determination unit 24 determines whether or not color determination is possible in step S23. Further, the second determination unit 25 determines whether or not the light spot region can be determined in step S25. For example, the first determination unit 24 and the second determination unit 25 determine that determination is impossible when the exposure amount is such that a sufficient pixel signal cannot be obtained compared to the noise level. Further, the second determination unit 25 determines that the pixel signal cannot be determined with an exposure amount that has not reached a level where the light spot region can be detected.

  If it is determined that the color can be determined, the first determination unit 24 determines the color in step S24. If the second determination unit 25 determines that the light spot region can be determined, the second determination unit 25 determines the light spot region in step S26.

  Note that the color determination may be made on condition that each RGB level is not saturated and each RGB level is equal to or higher than the noise level. In addition, since none of the RGB components is saturated in the exposure amount for performing color determination, the brightness distribution in the light spot region can be accurately obtained.

  Further, the determination of the target area may be performed with a sufficient exposure amount so that not only a light spot having a relatively large size but also a light spot having a relatively small size can be detected.

  In step S27, the imaging control unit 22 performs setting to add a predetermined exposure time, and then returns the process to step S22. Thus, after that, the image is read at every predetermined unit exposure time, and the color determination and the light spot region determination are performed until the appropriate exposure amount of the entire image is reached.

  The color determination by the first determination unit 24 may be performed for each pixel, whereas the light spot region determination by the second determination unit 25 needs to be performed on the entire image. That is, the first determination unit 24 determines the color by determining saturation for each light spot while changing the exposure amount. The second determination unit 25 obtains all target regions of the image with a predetermined exposure amount because the region size of the target region to be determined increases each time the exposure amount increases.

  For example, the first determination unit 24 determines the color of each image portion using the RGB component that provides the highest saturation among the images of each exposure amount that can be color determined, and outputs the determination result to the image correction unit 27. To do. Note that which exposure amount the first determination unit 24 uses is controlled independently for each image portion.

  In addition, the second determination unit 25 performs determination using, for example, an exposure amount image in which a light spot region having a predetermined size or more can be detected among the exposure amount images in which the target region determination is possible. As described above, in this determination, an image having an exposure amount common to the entire image is used so that the difference in size for each light spot in the image can be understood.

  Further, when the light spot is obtained by neon forming a character, for example, the second determination unit 25 may perform the region determination based not only on the brightness of the image but also on the contrast. In this case, the second determination unit 25 performs determination with a relatively small exposure amount. In addition, when performing a slightly blurred expression suitable for the atmosphere of a light spot in a night view, it is sufficient that only the magnitude of each light spot can be determined, and there may be cases where each light spot portion is spread and determined. . Assuming such a case, for example, the second determination unit 25 performs region determination using an image having an exposure amount that is a relatively high exposure amount and can also determine a sufficiently small light spot.

  In addition, in order to understand the difference in the size of each light spot in the image, it has been described that the area determination is performed using an image having a common exposure amount in the entire image. However, each part of the image is used with an image having a different exposure amount. Area determination may be performed. For example, when a part of the light spot constitutes a character, the region may be determined using an image with an exposure amount that is different from the other part only for the character part.

  Note that the processing of the first and second determination units 24 and 25 described above is performed for each unit exposure time during one shooting period in which shooting is performed with an appropriate exposure amount for the entire image. Depending on the camera, the RGB component may not be directly detected during the exposure period of one image. For example, in a camera in which exposure control is performed based on a brightness signal (Y signal) obtained from RGB components, whether or not each RGB component is saturated during the exposure period or whether it is above the noise level It may be impossible to determine directly whether or not. Therefore, in this case, such a determination may be made based on the level of the Y signal instead of the RGB components. For example, a predetermined margin may be set for the Y signal, and if it is within a certain Y signal level range, it may be determined that the RGB component is not saturated and is above the noise level. Alternatively, assuming that the ratio of the RGB components is a certain value, and estimating the maximum and minimum levels of the RGB components based on the assumed ratio and the level of the Y signal, all the RGB components are larger than the noise level and not saturated. You may determine that. Of course, it does not have to be RGB, and the same can be said for other color components, and determination may be made based on a value obtained by calculating the component or only a part of the components.

  In step S22, when it is determined that the proper exposure of the entire image has been reached, the image of the proper exposure is supplied from the image storage unit 23 to the image correction unit 27 as an original image.

  For example, the image correction unit 27 paints the light spot area of the original image with the determined color, thereby generating a correction image with appropriate exposure as a whole and capable of recognizing the color, shape, and size of the light spot portion. The corrected image is given to the recording unit 33 and recorded.

  As described above, in the present embodiment, images having different exposure amounts or brightness are obtained, and even if the entire image is proper exposure, the color exposure is partially inappropriate and the color reproducibility is poor. By determining the color using an image with an exposure amount or brightness that can be determined, and applying the determined color to the image part such as filling the image part with this color, the entire image and each part Thus, an image with improved color reproducibility can be obtained. Further, even if the exposure is appropriate for the entire image, the exposure amount or brightness that allows the target region to be determined for an image portion where the exposure is not appropriate partially and the reproducibility of the region (target region) of the image portion is poor. By determining the target area using the image and applying the determined color to the target area, such as the process of filling the target area with the determined color, the reproducibility of the shape in the entire image and each part is improved. Images can be obtained. The target area of such a corrected image is a color similar to the color that a person actually recognizes, has an appropriate position, shape, and size, for example, without being too large or blurring. The shape and size are the same as those actually felt by humans, and the corrected image has excellent color and shape reproducibility.

(Second Embodiment)
FIG. 7 is a flowchart showing an operation flow employed in the second embodiment of the present invention. The hardware configuration in this embodiment is the same as that in the first embodiment. Also in the present embodiment, as described above, the color is determined by obtaining the ratio of the spectral components (color components) by, for example, the ratio of the RGB signals, and the signal amount is determined in consideration of appearance. It is something that can be done. The determination method for color determination and region determination is the same as that in the first embodiment, and a description thereof will be omitted.

  In the first embodiment, description has been given mainly assuming that the light spot region is the target region. Not only the light spot but also the part where the exposure is not appropriate is set as the target area, the target area is determined and the color distribution in the target area is determined, and by applying the determined color to the target area, the entire image and the image A corrected image with appropriate exposure can be obtained for each part. This embodiment shows an example in which the target area is other than the light spot area.

  For example, when a mandarin orange field is photographed, if the overall exposure amount of the image is set appropriately, the orange color of the mandarin orange may not be reproduced even if the mandarin orange leaf is reproduced green. Even in this case, by using the same method as in the first embodiment, by selecting an image in which all of the RGB components are not saturated and the exposure amount or brightness is sufficiently larger than the noise level, The color of the target area can be reproduced.

  However, in the daytime shooting, unlike night scene shooting, it is conceivable that the exposure time for obtaining an appropriate exposure amount for the entire image is extremely short. In this case, color determination cannot always be performed during one shooting. FIG. 7 corresponds to this case, and the color and the target area are determined by a plurality of times of photographing.

  When the user presses the shutter button to instruct imaging, the imaging control unit 22 sets a predetermined exposure time (step S31) and instructs the imaging unit 10 to perform imaging. Thereby, photographing with a predetermined exposure amount is performed (step S32). An image from the image sensor is captured by the image acquisition unit 21, supplied to the image storage unit 23, and stored. The first determination unit 24 and the second determination unit 25 read the image stored in the image storage unit 23.

  The second determination unit 25 determines whether or not it is possible to determine an area (hereinafter referred to as a color point area) occupied by a portion (hereinafter referred to as a color point) for determining a color in the image in step S33. To do. The first determination unit 24 determines whether or not color determination is possible for the color point in step S35. The conditions for determination in steps S33 and S35 by the second determination unit 25 and the first determination unit 24 are the same as those in the first embodiment.

  If it is determined that the color point region can be determined, the second determination unit 25 determines the color point region in step S34. When determining that the color can be determined, the first determination unit 24 determines the color in step S36.

  The imaging control unit 22 determines whether or not the color determination and the color point area determination are completed based on the determination results of the first and second determination units 24 and 25 (step S37). If both the color determination and the color point area determination are completed, the process is terminated, and if not completed, the process proceeds to the next step S38.

  In step S38, the imaging control unit 22 sets an exposure time that is increased compared to the previous shooting (step S38), and repeats shooting (step S32). Thereafter, the same operation is repeated, and the color determination of the color point and the determination of the color point region are performed at each exposure amount. Note that the determination in step S37 may be omitted, and the color determination and the area determination may be repeated until a predetermined exposure amount is reached.

  The determination results of the first and second determination units 24 and 25 are given to the image correction unit 27. The image correction unit 27 generates, for example, a correction image in which the color, shape, and size of the color point portion can be recognized with appropriate exposure as a whole image by painting the color point area of the original image with the determined color, for example. The corrected image is given to the recording unit 33 and recorded.

  As described above, also in this embodiment, the same effect as that of the first embodiment can be obtained. Further, in the present embodiment, even when color determination and color point area determination cannot be performed for a color point by a single shooting, an image obtained by a plurality of shootings while changing the exposure amount is used. It is possible to reliably perform the color determination and the color point region determination for the color point.

(Third embodiment)
FIG. 8 is a flowchart showing an operation flow employed in the third embodiment of the present invention. In FIG. 8, the same steps as those in FIG. 9 to 11 are explanatory diagrams for explaining the present embodiment. The hardware configuration in this embodiment is the same as that in the first embodiment. Also in the present embodiment, as described above, the color is determined by obtaining the ratio of the spectral components (color components) by, for example, the ratio of the RGB signals, and the signal amount is determined in consideration of appearance. It is something that can be done. The determination method for color determination and region determination is the same as that in the first embodiment, and a description thereof will be omitted.

  In each of the above embodiments, an example has been described in which color determination and region determination in the target region are performed for the entire image. This embodiment shows an example in which color determination and region determination are performed on a target region in a partial region of an image. For example, as a partial region of the image, there is a character portion included in the image.

  FIG. 9 shows a captured image obtained by photographing a night view with a predetermined exposure amount. The image 51 includes a sky image portion 52, a star image portion 53 indicated by an asterisk, a building image portion 54, a person image portion 57, and a light spot image portion 58a caused by light or neon leaking from a building window. 58b. In the example of FIG. 9, as a result of shooting with a relatively large exposure amount as in the night view mode of a digital camera, the image portions 58a and 58b with bright light spots are saturated with image signals, and the actual light color. Regardless of the color, the color becomes white, blurring occurs, the size becomes larger than the actual size, and the shape cannot be identified.

  In the present embodiment, color determination and area determination are performed only at a position designated by the user. FIG. 10 shows this state, and a display screen 31b of the display unit 31 is disposed on the back surface of the housing 1a in which the imaging device 1 is accommodated. The image 51 of FIG. 9 is displayed on the display screen 31b. A shutter button 32a corresponding to the operation unit 32 in FIG. 1 is disposed on the top surface of the housing 1a. The touch panel 31a of FIG. 1 is provided on the display screen 31b, and the touch panel 31a receives an operation by the user. That is, the touch panel 31 a outputs information on the position on the display screen 31 b pointed by the user with the finger 60 to the operation determination unit 28.

  The image processing apparatus 20 performs color determination and area determination at a position specified by the user by, for example, a touch operation of the finger 60. In this case, when the touch position is a character part, the second determination unit 25 may employ contrast determination for shape determination. Furthermore, the second determination unit 25 may perform character recognition by referring to a database (not shown) to improve the accuracy of region determination.

  Next, an operation when a corrected image is generated by the imaging apparatus of FIG. 1 configured by a digital camera or the like will be described with reference to FIG.

  In step S41 of FIG. 8, the control unit 26 determines whether or not the shooting mode is instructed. If the shooting mode is not instructed, the control unit 26 determines whether or not the playback mode is designated (step S51). When the reproduction mode is instructed, the control unit 26 reproduces the designated image in step S52. The control unit 26 determines whether or not a change of the reproduction image is instructed in step S53, and when the change is instructed, the control unit 26 changes to the designated image and performs reproduction (step S54). If the playback mode is not instructed, the control unit 26 shifts to the image communication mode in step S55.

  On the other hand, when the shooting mode is instructed, the control unit 26 controls the imaging control unit 22 to start imaging of the imaging unit 10. The control unit 26 controls the image acquisition unit 21 to acquire a captured image from the imaging unit 10, provides the acquired image to the display control unit 29 via the image storage unit 23 and the image correction unit 27, and displays the display unit 31. The through image is displayed on the display screen 31b (step S43). Thus, the captured image is displayed on the display screen 31b of the display unit 31, as shown in FIG.

  The control unit 26 determines the touch operation by the user after step S43 or in a shooting mode other than when imaging is started (step S44). For example, the control unit 26 detects whether or not a touch operation on the touch panel 31a has occurred during imaging of a dark scene. If there is no touch operation, the control unit 26 determines whether or not there is a shooting operation in step S45. If there is no shooting operation, the process returns to step S41 and the same process is repeated.

  When photographing, the user can perform a photographing operation while looking at a through image on the display screen 31b arranged in substantially the entire rear surface of the housing 1a.

  Now, as shown in FIG. 10, it is assumed that the user performs a touch operation on the display screen 31 b with the finger 60. For example, the user touches the position of the image portion 58 b of the light spot in the image 51. The control unit 26 detects this touch operation, and the process proceeds from step S44 to step S21. That is, the control unit 26 controls the imaging control unit 22 so that the imaging unit 10 performs exposure by setting a predetermined exposure time. The imaging control unit 22 reads a pixel signal from the imaging device when the exposure time for obtaining a predetermined exposure amount is reached. Thus, the image acquisition unit 12 obtains an image at a predetermined exposure amount. This image is supplied to the image storage unit 23 and is supplied to the first determination unit 24 and the second determination unit 25.

  In the present embodiment, the second determination unit 25 is controlled by the control unit 26, and in step S61, whether or not the light spot region can be determined for the position specified by the touch operation by the user in the image. Determine. Further, the first determination unit 24 is controlled by the control unit 26 to determine whether or not color determination is possible at the position designated by the user in the image by the touch operation in step S63.

  If it is determined that the light spot area can be determined, the second determination unit 25 determines the light spot area in step S62. Note that, as described above, the second determination unit 25 may determine the light spot region by image contrast determination or character recognition processing. According to such region determination, when the light spot is a character part such as neon, the determined region is the same shape as the character formed by neon or the like, and the character is clearly displayed in the image. Discrimination becomes possible. For example, for the character portion, the second determination unit 25 can accurately reproduce the character shape by performing region determination using an image with a relatively small exposure amount. If the first determination unit 24 determines that the color determination is possible, the first determination unit 24 determines the color in step S64.

  Furthermore, in the present embodiment, the same color as the color determination result of the light spot at the position specified by the touch operation by the user is detected from the image, and the light spot portion of the color is subjected to the second determination. The area determination may be performed by the unit 25. For example, when the second determination unit 25 performs the region determination on the character part described above, the shape of all characters of the same color in the image can be clearly determined.

  In step S27, the imaging control unit 22 performs setting to add a predetermined exposure time, and then returns the process to step S22. Thus, after that, the image is read at every predetermined unit exposure time, and the color determination and the light spot region determination are performed until the appropriate exposure amount of the entire image is reached.

  In step S22, when it is determined that the proper exposure of the entire image has been reached, the image of the proper exposure is supplied from the image storage unit 23 to the image correction unit 27 as an original image. In step S47, the image correction unit 27 fills the light spot area of the area specified by the user in the original image with the determined color, for example, so that the image as a whole is appropriately exposed and the color and shape of the light spot part. And a corrected image whose size is recognizable.

  In the present embodiment, the image correction unit 27 gives the corrected image to the display control unit 29 and displays it on the display screen 31b (step S48). FIG. 11 shows an image 51 displayed on the display screen 31b in this case. In the image 51, the light spot image portion 58b of FIG. 9 is replaced with a character image 58c. As for the portion other than the position touched by the user, the shape of the portion determined to be the same color as the image at the touch position is accurately determined by character recognition or the like. As shown in FIG. Is displayed in a clear shape.

  When the display of the corrected image is completed, the control unit 26 shifts the process from step S49 to step S45, and enters a standby state for a shooting operation. Here, it is assumed that the user operates the shutter button 32a to instruct shooting. In the present embodiment, in step S46, the same processing as before photographing, that is, color determination, region determination, and generation of a corrected image for a through image are performed, and the generated corrected image is given to the recording unit 33. To be recorded.

  Since the main subject such as a person is concerned about the color cast of night illumination, white balance adjustment may be performed so that the face color does not become unnatural. Further, it is not necessary to adjust the white balance because it is considered that the light spot of the background night scene is small when the color is covered there. In some cases, strong illumination may overlap, in which case white balance adjustment is performed.

  As described above, in the present embodiment, the same effects as in the first embodiment can be obtained, and color determination and area determination can be performed at a position designated by the user. In addition, by employing contrast determination, character recognition, and the like for shape determination, characters in the image can be accurately determined and converted into a clear character image. The user can convert to a clear image by touching an image part that cannot be clearly discriminated by light, and can take a picture while confirming characters such as neon on a through image. A captured image including a character image can be recorded.

  In the example of FIG. 8, an example in which a corrected image is generated by performing color determination and region determination on a through image at the time of shooting has been described, but a plurality of images with different exposure amounts are also recorded in an already recorded image. It is clear that recording a correction image can be performed by performing color determination and region determination on a part of the image by recording the image.

  Furthermore, in each embodiment of the present invention, a digital camera has been described as an apparatus for photographing. However, the camera may be a lens type camera, a digital single-lens reflex camera, or a compact digital camera, a video camera, A camera for moving images such as a movie camera may be used, and a camera built in a personal digital assistant (PDA) such as a mobile phone or a smartphone may of course be used. In addition, it may be an industrial or medical optical device such as an endoscope or a microscope, such as a surveillance camera, an on-vehicle camera, a stationary camera, for example, a camera attached to a television receiver or a personal computer. There may be. For example, by applying to an endoscope or a microscope, it is possible to make it easier to observe a specific color in the subject.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  It should be noted that even if the operation flow in the claims, the description, and the drawings is described using “first,” “next,” etc. for convenience, it is essential to carry out in this order. It doesn't mean. In addition, it goes without saying that the steps constituting these operation flows can be omitted as appropriate for portions that do not affect the essence of the invention.

  Of the technologies described here, many of the controls and functions mainly described in the flowcharts can be set by a program, and the above-described controls and functions can be realized by a computer reading and executing the program. it can. As a computer program product, the program may be recorded or stored in whole or in part on a portable medium such as a non-volatile memory such as a flexible disk or a CD-ROM, or a storage medium such as a hard disk or a volatile memory. It can be distributed or provided at the time of product shipment or via a portable medium or communication line. The user can easily realize the image processing apparatus according to the present embodiment by downloading the program via a communication network and installing the program on a computer, or installing the program from a recording medium on the computer.

    DESCRIPTION OF SYMBOLS 1 ... Imaging device, 10 ... Imaging part, 20 ... Image processing apparatus, 21 ... Image acquisition part, 22 ... Imaging control part, 23 ... Image storage part, 24 ... 1st determination part, 25 ... 2nd determination part, 26 ... Control part 27 ... Image correction part 28 ... Operation determination part 29 ... Display control part 31 ... Display part 31a ... Touch panel 32 ... Operation part 33 ... Recording part.

Claims (13)

An image acquisition unit that acquires a plurality of images with different brightness for the same subject;
An image storage unit for storing the plurality of images;
A first determination unit that performs color component determination of a predetermined target region in the image using an image having a first brightness among the plurality of images stored in the image storage unit;
A second determination unit that performs region determination of the target region using a second brightness image among the plurality of images stored in the image storage unit;
An image correction unit that generates a corrected image in which the color of the color component determination result is applied to the target region in one of the plurality of images based on the determination results of the first and second determination units. An image processing apparatus comprising: The said image acquisition part controls the imaging of the imaging part which can output the captured image of different exposure amount about the same subject, and acquires the several image from which the brightness differs about the same subject. The image processing apparatus according to 1. The image processing apparatus according to claim 2, wherein the image acquisition unit acquires a plurality of images with different brightness for the same subject during one shooting by the imaging unit. The image processing apparatus according to claim 2, wherein the image acquisition unit acquires a plurality of images having different brightness for the same subject by performing a plurality of shootings by the imaging unit. 5. The apparatus according to claim 1, wherein the first determination unit selects an image in which each color component in the target region is not saturated as the image having the first brightness. The image processing apparatus described. 6. The first determination unit according to claim 1, wherein the first determination unit selects, as the first brightness image, an image in which each color component in the target region is higher than a predetermined noise level. The image processing apparatus according to any one of the above. 7. The second determination unit according to claim 1, wherein the second determination unit selects, as the second brightness image, an image in which each color component in the target region is higher than a predetermined noise level. The image processing apparatus according to any one of the above. The said 2nd determination part selects the image of the brightness which can detect the said object area | region more than the predetermined size in an image as said 2nd brightness image, Any one of Claim 1 thru | or 7 characterized by the above-mentioned. The image processing apparatus according to claim 1. The image processing apparatus according to claim 1, wherein the first and second determination units specify the target area based on a user operation. The image correction unit is configured to apply the color of the color component determination result to the target region in the third brightness image among the plurality of images based on the determination results of the first and second determination units. The image processing apparatus according to claim 11, wherein an image is generated. An imaging unit capable of capturing a plurality of images with different brightness for the same subject;
An image storage unit for storing the plurality of images from the imaging unit;
A first determination unit that performs color component determination of a predetermined target region in the image using an image having a first brightness among the plurality of images stored in the image storage unit;
A second determination unit that performs region determination of the target region using a second brightness image among the plurality of images stored in the image storage unit;
An image correction unit that generates a corrected image in which the color of the color component determination result is applied to the target region in one of the plurality of images based on the determination results of the first and second determination units. An image pickup apparatus comprising: An image acquisition step of acquiring a plurality of images with different brightness for the same subject;
An image storage step for storing the plurality of images;
A first determination step of determining a color component of a predetermined target region in the image using an image having a first brightness among the plurality of images stored in the image storage step;
A second determination step of performing region determination of the target region using a second brightness image among the plurality of images stored in the image storage step;
An image correction step for generating a corrected image in which the color of the result of the color component determination is applied to the target region in one of the plurality of images based on the determination results of the first and second determination steps. And an image processing method. On the computer,
An image acquisition step of acquiring a plurality of images with different brightness for the same subject;
An image storage step for storing the plurality of images;
A first determination step of determining a color component of a predetermined target region in the image using an image having a first brightness among the plurality of images stored in the image storage step;
A second determination step of performing region determination of the target region using a second brightness image among the plurality of images stored in the image storage step;
An image correction step for generating a corrected image in which the color of the result of the color component determination is applied to the target region in one of the plurality of images based on the determination results of the first and second determination steps. An image processing program for executing