JP2016173777A - Image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To stably provide a video having corrected gradation crush of a predetermined object and improved visibility, when gradation of a region containing the predetermined object in the video is crushed.SOLUTION: An image processing apparatus is provided with: an image input unit 0101; an image signal correction unit 0102 which subjects an image signal input from the image input unit to signal level correction processing, and outputs the image signal of which the level has been corrected; a gradation crush detecting unit 0103_1 which detects presence or absence of the region of which the gradation has been crushed from the image signal input from the image input unit; a predetermined object presence or absence estimating unit 0103_2 which estimates the presence or absence of the predetermined object in the region of which the gradation has been crushed from the image signal input from the image input unit; and an image signal correction control unit 0103_3 which performs control of input/output characteristics of signal level correction processing of the image signal correction unit based on the detected result of the gradation crush detecting unit and the estimated result of the predetermined object presence or absence estimating unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus.

  As a background field of the present technology, for example, there is JP 2010-204404 A (Patent Document 1). In this publication, “Providing a Surveillance Camera that Controls Appropriate Exposure to a Moving Object Even in an Imaging Environment such as Backlight or Shade, and Accurately Captures the Face and Clothing of an Intruder and its Exposure Control Method” "The surveillance camera of the present invention is a mobile body detection means 16, 19 for detecting the presence or absence of a mobile body entering the monitoring area and its position and size as mobile body information, and the captured image is divided into a plurality of blocks. Luminance calculation means 20 that divides and calculates the average luminance value for each block as the block luminance average value, and the block luminance average value in the time axis direction for the moving object-free block in which the moving object is not imaged based on the moving object information Based on the moving body information, the background luminance calculating means 21 that calculates the background image luminance average value by integrating the luminance average value and the background image luminance average for the moving object-containing block in which the moving object is imaged. Comparing the door, and a control unit 22 for performing exposure control on the basis of the comparison result, technology that includes. "Is disclosed.

JP 2010-204404 A

  In Patent Document 1, a moving body that is a predetermined subject is detected from an image, and exposure control is performed to improve the visibility of the moving body based on the detection result. When the gradation of the image is lost due to the loss of gradation such as blackout or overexposure, the mobile object cannot be detected, so that exposure control cannot be performed properly and the visibility of the mobile object cannot be improved. There is.

  In order to solve the above-described object, for example, the configuration described in the claims is adopted. The present application discloses a plurality of means for solving the above-described problems. For example, an image input unit and an image signal that performs a signal level correction process on the image signal input from the image input unit and outputs a level-corrected image signal A correction unit, a gradation collapse detection unit that detects the presence or absence of a region in which the gradation is crushed from the image signal input from the image input unit, and a predetermined region in the region in which the gradation is crushed from the image signal input from the image input unit Control of input / output characteristics of the signal level correction processing of the image signal correction unit based on the detection result of the predetermined subject presence / absence estimation unit for estimating the presence / absence of the subject, and the detection result of the gradation collapse detection unit and the estimation result of the predetermined subject presence / absence estimation unit This can be solved by providing an image signal correction control unit.

  According to the present invention, when an area including a predetermined subject in a video is crushed, it is possible to stably provide a video with improved visibility by correcting the crushed gradation of the predetermined subject.

1 is a schematic diagram showing an image processing apparatus according to a first embodiment of the present invention. The figure which shows an example of the signal level correction process of the image processing apparatus which concerns on 1st Example of this invention. The figure which shows an example of the gradation crushing information detection process of the image processing apparatus which concerns on 1st Example of this invention. The figure which shows an example of the predetermined object presence-and-presence estimation process using the time zone information which concerns on 1st Example of this invention. The figure which shows an example of the predetermined object presence-and-presence estimation process using the area information of the gradation collapse area which concerns on 1st Example of this invention. 1 is a first diagram illustrating an example of an image signal correction control process of an image processing apparatus according to a first embodiment of the present invention; The figure which shows the example of application of the image signal correction | amendment control processing when the predetermined | prescribed subject does not exist in the dark part in the image | video which concerns on 1st Example of this invention. The figure which shows the example of application of the image signal correction | amendment control process in case the predetermined | prescribed subject exists in the dark part in the image | video concerning 1st Example of this invention. The figure which shows an example of the signal level correction process of the image processing apparatus which concerns on 2nd Example of this invention. The figure which shows an example of the gradation crushing information detection process of the image processing apparatus which concerns on 2nd Example of this invention. The figure which shows an example of the predetermined subject presence / absence estimation process of the image processing apparatus which concerns on 2nd Example of this invention. 1st figure which shows an example of the image signal correction control process of the image processing apparatus which concerns on 2nd Example of this invention. The figure which shows the example of application of the image signal correction | amendment control process when the predetermined | prescribed subject does not exist in the gradation collapsed area | region of the intermediate brightness | luminance in the image | video concerning 2nd Example of this invention. The figure which shows the example of application of an image signal correction | amendment control process when a predetermined | prescribed subject exists in the gradation collapsed area | region of the intermediate brightness | luminance in the image | video which concerns on 2nd Example of this invention. Schematic diagram illustrating an image processing apparatus according to a third embodiment of the present invention. The figure which shows an example of the predetermined object presence-and-presence estimation process using the time zone information which concerns on 3rd Example of this invention. The figure which shows an example of the predetermined object presence-and-presence estimation process using the area information of the over-out area | region which concerns on 3rd Example of this invention. The 1st figure which shows the 1st example of the process sequence regarding the image signal correction control process of the image processing apparatus which concerns on 3rd Example of this invention. The figure which shows the example of application of an image signal correction | amendment control process in case the predetermined | prescribed subject exists in the overexposed area | region in the image | video concerning 3rd Example of this invention. The figure which shows the example of application of an image signal correction | amendment control process in case the predetermined | prescribed subject exists in the black-out area | region in the image | video concerning 3rd Example of this invention. The 1st figure which shows the 2nd example of the process sequence regarding the image signal correction control process of the image processing apparatus which concerns on 3rd Example of this invention. The figure which shows the example of application of the image signal correction | amendment control process in case the predetermined | prescribed subject exists in both the black-out area | region and the overexposure area | region in the image | video concerning 3rd Example of this invention. The figure which shows the example of application of an image signal correction | amendment control process in case the predetermined | prescribed subject exists in the overexposed area | region in the image | video concerning 3rd Example of this invention. Schematic diagram illustrating an image processing apparatus and an imaging apparatus according to a fourth embodiment of the present invention.

Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a schematic diagram showing an image processing apparatus according to a first embodiment of the present invention. In FIG. 1, 0101 is an image input unit, 0102 is an image signal correction unit, 0103 is a control unit, 0103_1 is a gradation collapse detection unit, 0103_2 is a predetermined subject presence / absence estimation unit, 0103_3 is an image signal correction control unit, and 0104 is an input unit. It is.

  In the image processing apparatus shown in FIG. 1, an image input unit 0101 is connected to an imaging device or a video device via a capture board connected to a video cable, a LAN, or a USB, and inputs an image signal.

  The image signal correction unit 0102 corrects predetermined input / output characteristics for the video signal output from the image input unit 0101 and the signal value for each pixel of the video signal based on the correction characteristics acquired from the image signal correction control unit 0103_3. The signal level is corrected and a corrected video signal is output. An example of this process will be described later with reference to FIG.

  The gradation collapse detection unit 0103_1 calculates a feature amount such as a histogram value of a signal value in a predetermined area or a feature amount such as an edge amount calculated by comparing a target pixel and a neighboring pixel from a video signal output from the image input unit 0101. Based on the calculation result, gradation collapse information such as a region where the gradation is collapsed in the video and the signal level where the gradation collapse occurs is detected. An example of this processing will be described later with reference to FIG.

  Predetermined subject presence / absence estimation unit 0103_2 calculates feature quantities such as average luminance, histogram data of signal values within a predetermined area, and edge amount from the video signal output from image input unit 0101, and uses the calculation result to calculate a shooting time zone, An estimated value of information related to the shooting scene such as the weather at the time of shooting is calculated. Alternatively, an estimated value of information related to a shooting scene is acquired from another device from the input unit 0104. Evaluation of whether or not a predetermined subject exists in the area in which the gradation is crushed in the video using the estimated value of information regarding the shooting scene and the gradation crushed information acquired from the gradation crushed detection unit 0103_1. The value is calculated as a predetermined subject presence / absence estimation evaluation value. An example of this processing will be described later with reference to FIG. Here, the predetermined subject is a subject that the user pays attention to, for example, a moving object, a person, a person's face, a vehicle, a vehicle license plate, or the like for video surveillance, and a preceding vehicle, a sign, or the like for in-vehicle use. .

  The image signal correction control unit 0103_3 uses the gradation collapse information output from the gradation collapse detection unit 0103_1 and the predetermined subject presence / absence estimation evaluation value output from the predetermined subject presence / absence estimation unit 0103_2 to determine a predetermined area in the area where the gradation is collapsed. When it can be determined that the subject exists, the correction characteristic is determined so as to increase the correction strength of the signal level correction processing with respect to the collapsed signal level, and is output to the image signal correction unit 0102. An example of this process will be described later with reference to FIG.

  An input unit 0104 is connected to other imaging devices such as a camera, a video processing server, a PC, a timer, or an information processing device via a LAN, USB, etc., and receives information related to a shooting scene such as a shooting time zone and weather during shooting. input.

  This makes it possible to correct the signal level only when there is a high possibility that the predetermined subject exists even in a situation where it is difficult to detect the predetermined subject in an area where the gradation is crushed due to the crushed gradation generated in the video. By performing the processing, the visibility of a predetermined subject can be improved, and the accuracy of image recognition can be improved. On the other hand, when the predetermined subject is not included in the area where the gradation is crushed, the signal level correction process is weakened, so that deterioration in image quality such as noise enhancement can be suppressed. In this way, it is possible to provide a stable video for shooting various scenes.

  The image input unit 0101 includes a lens, an image sensor, and an A / D converter, and generates an image signal by performing an imaging process, or reads an image signal stored in advance in an image recording unit (not shown). The image signal may be input by performing the above.

  In the above description, the gradation collapse detection unit 0103_1 and the predetermined subject presence / absence estimation unit 0103_2 are configured to calculate the feature amount from the video signal output from the image input unit 0101, respectively. A configuration in which one of the units 0103_2 calculates and uses the feature amount calculated by the other, or the image input unit 0101 performs a process of calculating the feature amount from the video signal and outputs it together with the video signal. 0103_1 and the predetermined subject presence / absence estimation unit 0103_2 may acquire and use the feature amount, and the overall calculation amount can be reduced by common processing for calculating the feature amount. The image processing apparatus shown in FIG. 1 is described focusing on the processing unit directly related to the present invention, but other image quality enhancement processing, signal format conversion processing, compression / decompression processing, and the like are performed before and after each processing. Or you may carry out on the way.

  1 is implemented by an application or GPU on a CPU if it is a personal computer, a microcomputer or DSP, or a dedicated LSI if it is an embedded device, for example. Signal processing such as signal level correction processing is performed on a DSP, a dedicated LSI or GPU, and control processing such as gradation loss detection processing, predetermined subject presence / absence estimation processing, and image signal correction control processing is performed by a microcomputer or CPU. Therefore, cost and performance may be optimized.

  FIG. 2 is a diagram illustrating an example of signal level correction processing of the image processing apparatus according to the first embodiment of the present invention. In the present invention, the signal level correction process is performed by the image signal correction unit 0102.

  In FIG. 2, 0201 is a video signal input from the image input unit 0101, 0202_1 is a representative subject area existing in a dark part in the video signal, and 0202_2 is a representative object existing in a bright part in the video signal. This is the subject area. The image signal correction unit 0102 has different input / output characteristic information for the dark part and the bright part in advance, and applies the corresponding input / output characteristic information depending on whether the target pixel exists in the dark part or the bright part. Correct the signal level. Since the area indicated by 0202_1 is a subject area existing in a dark part, the visibility of a dark subject can be improved by performing correction for raising a signal using input / output characteristic information for the dark part. Further, since the area indicated by 0202_2 is a subject area existing in a bright part, the visibility of a bright subject can be improved by performing correction for suppressing a signal using the input / output characteristic information for the bright part. Here, the input / output characteristics of the dark part and the input / output characteristic of the bright part are corrected according to the correction intensity which is the correction characteristic acquired from the image signal correction control unit 0103_3, and the final input / output characteristic is determined. For this reason, when the correction strength is small, the correction can be weakened by bringing the input and output characteristics close to each other.

  As a result, the gradation can be adaptively corrected according to the brightness of the subject to improve the visibility in the underexposure area and overexposure area, and whether or not correction is necessary as required by giving the correction intensity as an input It is possible to change the strength.

  Note that the determination of whether the target pixel exists in the dark part or the bright part is performed by, for example, calculating the low frequency component of the luminance signal from the average value of the luminance of the local region including the target pixel in the video signal 0201. This can be done by thresholding the low frequency component.

  FIG. 3 is a diagram showing an example of the gradation collapse information detection process of the image processing apparatus according to the first embodiment of the present invention. In the present invention, the gradation collapse information detection processing is performed by the gradation destruction detection unit 0103_1.

  In FIG. 3, 0301 is a video signal input from the image input unit 0101, and 0302 is a gradation collapsed region in which a dark part in the video signal is blacked out. The gradation collapse detection unit 0103_1 calculates a histogram of the luminance of the entire image from the video signal 0301, counts the number of pixels equal to or less than a predetermined threshold th set in the vicinity of the minimum luminance, and the number of pixels reaches a predetermined value. If it exceeds, it can be determined that there is a blackened area, and the determination result is output as grayscale information. At the same time, the area and signal level of the area where the gradation is crushed may be output as the gradation crushed information. For example, the area of the area where the gradation is crushed can be calculated from the total number of pixels whose luminance is th or less, and the signal level of the area where the gradation is crushed can be calculated from the average luminance of the pixels whose luminance is th or less.

  In addition, for simplification of description, the description has been made using blackout as an example of detection of gradation loss. However, the same applies to the case of detecting overexposure and loss of intermediate luminance. For example, if the threshold value th is set near the maximum luminance and the number of pixels equal to or greater than th is counted, it is possible to determine the presence of an overexposed region. If an arbitrary threshold value th1 and th2 in the vicinity thereof are set, If the number of pixels smaller than th2 is counted, it is possible to determine the presence of arbitrary intermediate luminance gradation collapse.

FIG. 4 is a diagram illustrating an example of the predetermined subject presence / absence estimation process of the image processing apparatus according to the first embodiment of the present invention. In the present invention, the predetermined subject presence / absence estimation process is performed by the predetermined subject presence / absence estimation unit 0103_2.
FIG. 4A is an example of a predetermined subject presence / absence estimation process using time zone information, and FIG. 4B is an example of a predetermined subject presence / absence estimation process using area information of a gradation collapsed area.

  In FIG. 4A, the predetermined subject presence / absence estimation unit 0103_2 inputs time information at the time of shooting from the input unit 0104, thereby acquiring information t regarding the time zone at the time of shooting. If the time information at the time of shooting is not recorded, the time information estimated from the average luminance of the video signal input from the image input unit 0101 may be substituted. The predetermined subject presence / absence estimation unit 0103_2 determines whether or not a predetermined subject exists in the blackout area acquired from the gradation collapse detection unit 0103_1 based on the predetermined input / output characteristics based on the acquired time zone information t. A predetermined subject presence estimation evaluation value E1 for estimating the above is determined. For example, in shooting applications, blackout during the day is likely to occur due to the black pattern of the subject, and blackout at night is likely to occur due to insufficient illuminance at the subject of interest such as a person or vehicle. It is possible to estimate whether or not a predetermined subject is included in the blackout area by setting the input / output characteristics such that E1 increases as the time zone becomes nighttime.

  In FIG. 4B, the predetermined subject presence / absence estimation unit 0103_2 calculates the area N of the blackout region from the video signal input from the image input unit 0101. The calculation of the area N of the blacked-out area can be performed by the same method as the calculation of the area of the area where the gradation is crushed in the above-described gradation crushed detection unit 0103_1. Alternatively, the area of the area where the gradation output from the gradation collapse detection unit 0103_1 is destroyed may be used as N. Based on the predetermined input / output characteristics, the predetermined subject presence / absence estimation unit 0103_2 determines whether or not a predetermined subject exists in the blackout region acquired from the gradation collapse detection unit 0103_1 based on the predetermined input / output characteristics. A predetermined subject presence estimation evaluation value E2 for estimating whether or not is determined is determined. As the predetermined input / output characteristics, for example, in shooting applications, when the blackout area is small, it is likely to occur due to the black pattern of the subject, and when the blackout area is large, the target object such as a person or a vehicle is included. Since there is a high possibility of being a region, it is possible to estimate whether or not a predetermined subject is included in the blacked-out region by setting input / output characteristics such that E2 increases as the area N of the blacked-out region increases.

  In this manner, the predetermined subject presence / absence estimation unit 0103_2 uses the information related to the shooting scene estimated from the video signal output from the image input unit 0101 and the information related to the shooting scene acquired from the other device from the input unit 0104, so that the predetermined subject is detected. It is possible to estimate whether or not a predetermined subject exists even when the gradation of the area to be included is crushed and subject detection by image recognition is difficult.

  FIG. 5 is a first diagram illustrating an example of an image signal correction control process of the image processing apparatus according to the first embodiment of the present invention. In the present invention, the image signal correction control process is performed by the image signal correction control unit 0103_3.

  In FIG. 5, the image signal correction control unit 0103_3 is determined from the predetermined subject existence estimation evaluation value E1 in the blacked-out region determined from the time zone information t output from the predetermined subject presence / absence estimating unit 0103_2 and the black determined from the area N of the blacked-out region. The predetermined subject existence estimation evaluation value E2 in the collapsed area is acquired, and the signal level generated by the collapsed gradation detection unit 0103_1, that is, the correction intensity of the dark part is output as the integral E1 × E2 increases. The correction characteristic is determined so as to increase and is output to the image signal correction unit 0102. In this way, by using a predetermined subject presence estimation evaluation value calculated from information related to a plurality of shooting scenes, it is possible to estimate with high accuracy whether or not a predetermined subject exists in a blackened area, Further, it is possible to improve the visibility when a predetermined subject exists in the blackout area based on the estimation result.

  Note that, in the case where a whiteout region or a middle luminance collapse region exists at the same time, the predetermined subject presence estimation evaluation value in each region is similarly calculated by the predetermined subject presence / absence estimation unit 0103_2, and the image signal correction control unit 0103_3 calculates By acquiring the result and determining the correction characteristics of the signal level where the corresponding gradation loss has occurred, when a predetermined subject exists in the overexposed region or the intermediate luminance collapse region, the visibility of that region is improved Is possible.

FIG. 6 is a second diagram illustrating an example of the image signal correction control process of the image processing apparatus according to the first embodiment of the present invention.
FIG. 6A is an application example of the image signal correction control process when a predetermined subject does not exist in the dark part in the video, and FIG. 6B shows an image when the predetermined subject exists in the dark part in the video. It is an application example of signal correction control processing.

  FIG. 6A is an image taken on the road during the day, and the dark part in the image is caused by taking a picture of a black vehicle. In this case, both the predetermined subject existence estimation evaluation value E1 in the blackout area determined from the time zone information t and the predetermined subject existence estimation evaluation value E2 in the blackout area determined from the area N of the blackout area are small values. Therefore, the predetermined subject presence / absence estimation unit 0103_2 determines the correction characteristics so that the correction intensity of the dark part is reduced. For this reason, it is possible to suppress adverse effects such as noise enhancement by performing dark level signal level correction by signal level correction processing, although they are unnecessary.

  FIG. 6B is an image taken on the road at night, and a dark portion is generated in the image due to underexposure due to insufficient illuminance. In this case, both the predetermined subject existence estimation evaluation value E1 in the blacked out area determined from the time zone information t and the predetermined subject existence estimated evaluation value E2 in the blacked out area determined from the area N of the blacked out area are both large values. Therefore, the predetermined subject presence / absence estimation unit 0103_2 determines the correction characteristics so that the correction intensity of the dark part is increased. Therefore, it is possible to improve the visibility of a subject of interest such as a blacked-out vehicle by performing signal level correction in a dark part by signal level correction processing.

  As described above, according to the present embodiment, it is estimated with high accuracy whether or not a predetermined subject exists in an area where the gradation is broken in the video, and when the predetermined subject exists, the visibility is improved. However, if it does not exist, excessive correction can be suppressed, and it is possible to provide an image with improved visibility and image quality adapted to various scenes.

  In this embodiment, the present invention is applied to a fog image correction apparatus. The configuration of the present embodiment is the same as the schematic diagram showing the image processing apparatus according to the first embodiment of the present invention shown in FIG. 1, and the signal level correction processing in the image signal correction unit 0102 and the gradation collapse detection unit 0103_1. The operations of the gradation collapse information detection processing in FIG. 5, the predetermined subject presence / absence estimation processing in the predetermined subject presence / absence estimation unit 0103_2, and the image signal correction control processing in the image signal correction control unit 0103_3 are different.

  FIG. 7 is a diagram illustrating an example of signal level correction processing of the image processing apparatus according to the second embodiment of the present invention. In the present invention, the signal level correction process is performed by the image signal correction unit 0102.

  In FIG. 7, 0701 is a video signal input from the image input unit 0101. The image signal correction unit 0102 has input / output characteristics for expanding the intermediate luminance part in advance as fog image correction. Although the gradation of the subject converges so as to approach a certain luminance due to the influence of fog, how the signal converges depends on the fog state and the exposure state. Therefore, which luminance range of the intermediate luminance part is expanded can be determined by detecting a value that can be regarded as the maximum and minimum values of the luminance of the local region including the target pixel, for example. A plurality of input / output characteristics are prepared in advance in a lookup table or function expression, and the maximum brightness value is larger and the minimum brightness value is smaller than the maximum brightness value and minimum brightness value in the local area including the target pixel. By selecting such input / output characteristics for each area, it is possible to perform gradation expansion processing according to the fog density for each area. In addition, the input / output characteristics calculated for each local region are corrected according to the correction strength, which is the correction characteristics acquired from the image signal correction control unit 0103_3, and the input and output match when the correction strength is small. Correction can be weakened by approaching the characteristics.

  This makes it possible to improve the visibility in foggy scenes by adaptively correcting the gradation according to the fog density for each region, and whether correction is necessary or not by giving correction characteristics as input And strength can be controlled.

  FIG. 8 is a diagram showing an example of gradation collapse information detection processing of the image processing apparatus according to the second embodiment of the present invention. In the present invention, the gradation collapse information detection processing is performed by the gradation destruction detection unit 0103_1.

  In FIG. 8, 0801 is a video signal input from the image input unit 0101, and 0802 is a region in which the fog in the video signal is very dark and the gradation of intermediate luminance is crushed. The gradation collapse detection unit 0103_1 calculates a luminance histogram in a predetermined area in the image, for example, the area 0802, from the video signal 0301, and det_fl which is a value which can be regarded as the maximum luminance and a value which can be regarded as the minimum. When det_fd is detected and the subject contrast c calculated as the difference det_fl-det_fd falls below a predetermined value, it can be determined that the gradation of the intermediate luminance is crushed by fog in the predetermined area. Therefore, the image is divided into a plurality of small regions, and the subject contrast c in each region is calculated. If there is a region where c is below a predetermined value, it is determined that there is a region where the gradation of intermediate luminance is crushed. Then, the determination result is output as gradation collapse information. At the same time, the area and signal level of the area where the gradation is crushed may be output as the gradation crushed information. For example, the signal level of the area where the gradation is crushed can be calculated as a plurality of signal levels from the average luminance of each of the small areas where c is below a predetermined value. Further, the area of the region where the gradation is crushed at each signal level can be calculated from the total number of small regions where the signal level is equal and c is below a predetermined value.

  FIG. 9 is a diagram illustrating an example of the predetermined subject presence / absence estimation process of the image processing apparatus according to the second embodiment of the present invention. In the present invention, the predetermined subject presence / absence estimation process is performed by the predetermined subject presence / absence estimation unit 0103_2.

  In FIG. 9, the predetermined subject presence / absence estimation unit 0103 </ b> _ <b> 2 obtains information on the weather at the time of shooting by inputting the weather information at the time of shooting from the input unit 0104. If the weather information at the time of shooting cannot be input, the weather information estimated from the on-screen distribution information of the subject contrast c of the video signal input from the image input 0101 and the color temperature information may be substituted. In addition, a subject contrast c in a region where the gradation output from the gradation collapse detection unit 0103_1 is destroyed is calculated from the video signal input from the image input unit 0101. The calculation of the subject contrast c can be performed by the same method as the calculation of the subject contrast c in the above-described gradation collapse detection unit 0103_1. Alternatively, the subject contrast c acquired from the gradation collapse detection unit 0103_1 may be used. Based on the acquired weather information and subject contrast c, the predetermined subject presence / absence estimation unit 0103_2 is based on predetermined input / output characteristics in a low-contrast region due to intermediate luminance gradation crushing acquired from the gradation crushing detection unit 0103_1. A predetermined subject existence estimation evaluation value E3 for estimating whether or not a predetermined subject exists is determined. For example, it is determined that there is a high possibility of gradation collapse in a region including a predetermined subject due to fog as the subject contrast c is small, and the input / output characteristics are such that E3 increases as the subject contrast c decreases. Thus, it can be estimated whether or not a predetermined subject is included in the gradation collapsed area of intermediate luminance. At that time, if weather information is used in combination, and the weather information indicates fog or yellow sand, it is determined that there is a higher possibility that the contrast is lowered due to the fog, and E3 is likely to increase. When clear sky is indicated, it is determined that the subject is originally a picture with no contrast, and the input / output characteristics are set such that E3 does not easily increase.

  In this way, by combining information related to a plurality of shooting scenes and determining a predetermined subject existence estimation evaluation value, it is determined whether or not a predetermined subject exists in an area in which gradation collapse of intermediate luminance occurs due to the influence of fog. It can be estimated with high accuracy.

  FIG. 10 is a first diagram illustrating an example of the image signal correction control process of the image processing apparatus according to the second embodiment of the present invention. In the present invention, the image signal correction control process is performed by the image signal correction control unit 0103_3.

  In FIG. 10, the image signal correction control unit 0103_3 acquires the predetermined subject presence estimation evaluation value E3 in the low contrast region output from the predetermined subject presence / absence estimation unit 0103_2, and outputs the gradation collapse detection unit 0103_1 as E3 increases. The correction characteristic is determined so that the fog image correction intensity at the signal level where the gradation is crushed becomes large, and is output to the image signal correction unit 0102. As described above, the visibility when the predetermined subject exists in the region whose gradation is crushed by the influence of fog based on the estimation result of whether or not the predetermined subject exists in the region where the intermediate luminance is crushed in gradation is obtained. It is possible to improve.

FIG. 11 is a second diagram illustrating an example of the image signal correction control process of the image processing apparatus according to the second embodiment of the present invention.
FIG. 11A shows an application example of the image signal correction control process in a case where a predetermined subject does not exist in the intermediate luminance gradation collapse area in the video, and FIG. 11B shows the intermediate luminance gradation destruction area. This is an application example of the image signal correction control process when a predetermined subject exists.

  FIG. 11A shows an image of a building with a low contrast when the weather is fine, and gradation collapse of intermediate luminance in the image is caused by shooting a picture such as a wall having no contrast. In this case, since the predetermined subject presence estimation evaluation value E3 in the low contrast region is a small value, the predetermined subject presence / absence estimation unit 0103_2 determines the correction characteristics so that the fog image correction strength is reduced. For this reason, it is possible to suppress adverse effects such as noise enhancement due to the expansion of the intermediate luminance gradation by the signal level correction processing, although it is unnecessary.

  Further, FIG. 11B is an image taken outdoors during foggy weather, and the gradation collapse of intermediate luminance in the image is caused by the influence of fog. In this case, since the predetermined subject presence estimation evaluation value E3 in the low contrast region is a large value, the predetermined subject presence / absence estimation unit 0103_2 determines the correction characteristic so that the fog image correction strength is increased. Therefore, it is possible to improve the visibility of a subject of interest such as a distant person whose gradation is crushed by the influence of fog by performing signal level correction that extends the gradation of intermediate luminance by signal level correction processing.

  Thus, according to the present embodiment, even when the image processing apparatus is applied to a fog image correction apparatus, it is possible to provide a video with improved visibility and image quality adapted to various scenes.

  In this embodiment, the present invention is applied to an image pickup apparatus.

  FIG. 12 is a schematic diagram showing an image processing apparatus according to the third embodiment of the present invention. In FIG. 12, 1201 is an imaging unit, 1202 is an image signal correction unit, 1203 is a control unit, 1203_1 is a gradation collapse detection unit, 1203_2 is a predetermined subject presence / absence estimation unit, 1203_3 is an image signal correction control unit, 1204 is an input unit, 1205 is a correction importance setting unit, 1206 is a dynamic range expansion unit, and 1207 is an image memory unit. The schematic diagram showing the image processing apparatus according to the third embodiment of the present invention shown in FIG. 12 is obtained by applying the schematic diagram showing the image processing apparatus according to the first embodiment of the present invention shown in FIG. The image signal correction unit 1202, the gradation collapse detection unit 1203_1, the predetermined subject presence / absence estimation unit 1203_2, and the input unit 1204 perform the same operations as in the first embodiment.

  In the imaging apparatus shown in FIG. 12, the imaging unit 1201 appropriately includes a lens group including a zoom lens and a focus lens, an iris, a shutter, an imaging element such as a CCD or a CMOS, a CDS, an AGC, an AD converter, and the like. The optical image received by the image sensor is photoelectrically converted and output as an image signal. At that time, an output image signal may be generated by performing various camera image processing such as exposure value evaluation value detection processing, digital gain processing, local brightness calculation processing for each pixel, and noise correction processing. .

  The image signal correction control unit 1203_3 uses the gradation collapse information output from the gradation collapse detection unit 1203_1 and the predetermined subject presence / absence estimation evaluation value output from the predetermined subject presence / absence estimation unit 1203_2 to generate a predetermined subject in the area where the gradation is destroyed. When the image level is determined to be present, the correction amount of the exposure amount of the imaging unit 1201, the correction characteristic of the signal level correction processing for the signal level where the image is crushed, and the dynamic range so that the gradation of the signal level where the image is crushed are expanded. The composition characteristic of the image composition processing of the enlargement unit 1206 is determined. In addition, when gradation crushing occurs at a plurality of signal levels, the exposure amount correction amount or crushing for each signal level is determined according to the predetermined subject presence / absence estimation evaluation value and correction importance at each signal level. The correction characteristic of the signal level correction process for the generated signal level and the synthesis characteristic of the video composition process of the dynamic range expansion unit 1206 are determined. The correction importance is determined from a predetermined subject presence / absence estimation evaluation value or a value acquired from the correction importance setting unit 1205 is used. An example of this process will be described later with reference to FIG. 14 and another example with reference to FIG.

  The correction importance setting unit 1205 includes user input devices such as cursor keys and buttons, or is connected to an information processing device such as a video processing server or a PC via a LAN, USB, etc., and image signal correction control set by the user The correction importance for each signal level when correcting the gradation collapse at a plurality of signal levels in the unit 1203_3 is input. In addition, it connects to a monitor unit or the like and displays an interface screen for the user to set.

  The dynamic range expansion unit 1206 combines video signals with different exposure periods at a predetermined ratio and outputs a combined video. In order to realize this, as an example, a video signal of a certain exposure period input from the image signal correction unit 1202 is stored in the image memory unit 1207, and a video signal of another exposure period input from the image signal correction unit 1202 is stored. And the weighted video signal of the previous exposure period read from the image memory unit 1207 may be added. By repeating this addition process, it is possible to synthesize video signals with an arbitrary number of different sensitivities, and capture images from low-illuminance subjects to high-illuminance subjects without losing gradation on the image. Can realize a wide dynamic range function. Further, as a combining method, it is possible to take a method of selecting which exposure period video signal is used for each pixel instead of weighting and adding the video signals of different exposure periods. When the wide dynamic range function is not used, the video signal may be output as it is without performing the synthesis process.

  The image memory unit 1207 includes an SDRAM and a DDR memory, accumulates a video signal of a certain exposure period input from the dynamic range expansion unit 1206, and outputs it according to the timing requested from the dynamic range expansion unit 1206.

  As a result, even when the present invention is applied to an image pickup apparatus including the image pickup unit 1201, gradation collapse occurs in the video, and it is difficult to detect a predetermined subject in a region where the gradation is lost due to image recognition or the like. Even in the situation, the visibility of a predetermined subject can be improved by performing exposure amount correction, signal level correction processing, and dynamic range expansion processing only when there is a high possibility that the predetermined subject exists. Conversely, when a predetermined subject is not included in the area where the gradation is crushed, the deterioration of image quality such as noise enhancement is suppressed by weakening exposure correction, signal level correction processing, and dynamic range expansion processing. Therefore, it is possible to provide a stable video for shooting various scenes. In particular, in the case of the present embodiment, when the signal level is corrected, the exposure of the imaging unit 1201 and the image synthesis characteristics of the dynamic range expansion unit 1206 are also controlled. Even when the image is completely blacked out due to insufficient sensitivity, it is possible to improve visibility by correcting the signal level when a predetermined subject is included in the area.

  The dynamic range expansion unit 1206 is configured to input the video signal output from the image signal correction unit 1202. However, the image signal correction unit 1202 inputs the video signal after the video synthesis process output from the dynamic range expansion unit 1206. It does not matter as a structure to do. In this case, since the dynamic range expansion unit 1206 inputs a linear signal to the subject light amount before the nonlinear signal level correction process, there is an advantage that it is easy to improve the image quality during the video composition process. Further, the dynamic range expansion unit 1206 may be configured to include the image memory unit 1207 in the imaging unit 1201 and perform video composition processing in the process in which the imaging unit 1201 outputs an image signal. In this case, the image capturing unit 1201 performs higher-speed driving and longer bit processing than the subsequent processing, so that it is possible to improve the image synthesis processing rate and improve the image quality by reducing the quantization error.

  12 is executed by an application or GPU on a CPU if it is a personal computer, a microcomputer or DSP, a dedicated LSI, or the like if it is an embedded device. Signal processing such as signal level correction processing and dynamic range expansion processing is performed on a DSP, dedicated LSI, or GPU, and control processing such as gradation collapse detection processing, predetermined subject presence / absence estimation processing, and image signal correction control processing is performed by a microcomputer or It may be possible to optimize the cost and performance by implementing with the CPU.

FIG. 13 is a diagram illustrating an example of the predetermined subject presence / absence estimation process of the image processing apparatus according to the third embodiment of the present invention. In this example, an example of a predetermined subject presence / absence estimation process in the case where overexposure occurs in a video is shown. In the present invention, the predetermined subject presence / absence estimation process is performed by the predetermined subject presence / absence estimation unit 1203_2.
FIG. 13A is an example of a predetermined subject presence / absence estimation process using time zone information, and FIG. 13B is an example of a predetermined subject presence / absence estimation process using overexposed area information.

  In FIG. 13A, the predetermined subject presence / absence estimation unit 1203_2 receives time information at the time of shooting from the input unit 1204, thereby acquiring information t regarding the time zone at the time of shooting. When the time information at the time of shooting is not recorded, the time information estimated from the average luminance of the video signal input from the imaging unit 1201 may be used instead. The predetermined subject presence / absence estimation unit 1203_2 determines whether or not there is a predetermined subject in the overexposure area acquired from the gradation collapse detection unit 1203_1 based on the predetermined input / output characteristics based on the acquired information t regarding the time zone. A predetermined subject presence estimation evaluation value E4 for estimating the above is determined. For example, in shooting applications, daytime overexposure is likely to occur due to backlighting of the subject of interest such as a person or vehicle, and nighttime overexposure may occur due to a subject of low attention such as lighting. When the time zone is high, it is possible to estimate whether or not a predetermined subject is included in the overexposure region by setting the input / output characteristics such that E4 increases as the time zone becomes daytime. In FIG. 13B, the predetermined subject presence / absence estimation unit 1203_2 calculates the area M of the overexposed region from the video signal input from the imaging unit 1201. The calculation of the area M of the over-exposure region is performed by losing the gradation in the gradation crushing detection unit 0103_1 in the example of the gradation crushed information detecting process of the image processing apparatus according to the first embodiment of the present invention illustrated in FIG. It is possible by the same method as the calculation of the area of the region. Alternatively, the gradation collapse detection unit 1203_1 may output the area of the overexposed area in the same manner as the gradation collapse detection unit 0103_1 and use it as M. Based on the predetermined input / output characteristics, the predetermined subject presence / absence estimation unit 1203_2 determines whether there is a predetermined subject in the overexposure area acquired from the gradation collapse detection unit 1203_1 based on the predetermined input / output characteristics. A predetermined subject presence estimation evaluation value E5 for estimating whether or not is determined is determined. For example, in a shooting application, when the overexposed area is small, it is likely to occur due to a low-focused subject such as a point light source, and when the overexposed area is large, it is an area that includes the subject of interest such as a person or a vehicle. If there is a high possibility, it is possible to estimate whether or not a predetermined subject is included in the blackout area by setting the input / output characteristics such that E5 increases as the area M of the overexposure area increases.

  In this manner, the predetermined subject presence / absence estimation unit 1203_2 includes a predetermined subject by using information on a shooting scene estimated from the video signal output from the imaging unit 1201 and information on a shooting scene acquired from another device from the input unit 1204. It is possible to estimate whether or not a predetermined subject exists even when the gradation of the region is overexposed and it is difficult to detect the subject by image recognition.

  The image signal correction control unit 1203_3 uses the predetermined subject presence estimation evaluation values E4 and E5 obtained in this way, and the gradation collapse output output by the gradation collapse detection unit 1203_1 increases as the integral E4 × E5 increases. The correction level of the exposure amount of the imaging unit 1201, the correction characteristic of the signal level correction process of the image signal correction unit 1202 and the video synthesis characteristic of the video synthesis process of the dynamic range expansion unit 1206 so as to correct the generated signal level, that is, the overexposure. Thus, it is possible to improve the visibility when a predetermined subject exists in the overexposed region.

  FIG. 13A shows an example of input / output characteristics in which E4 increases as the time zone indicated by the time zone information t becomes daytime, but it is assumed to be used for applications such as license plate monitoring. , White outbursts at night are likely to occur due to the reflection of light on highly reflective subjects including license plates, and daytime overexposure occurs for subjects with low attention other than license plates. Since it is highly likely to occur due to backlighting, the input / output characteristics may be such that E4 increases as the time zone becomes nighttime. In this way, by calculating a predetermined subject presence / absence estimation evaluation value based on a predetermined input / output characteristic according to the application, it is possible to accurately estimate whether or not a predetermined subject that is noticed by the user exists in an overexposed region. It is.

  FIG. 14 is a first diagram illustrating a first example of a processing sequence related to an image signal correction control process of the image processing apparatus according to the third embodiment of the present invention. In the present invention, the image signal correction control process is performed by the image signal correction control unit 1203_3.

  In the processing sequence related to the image signal correction control process shown in FIG. 14, in ST1401, the presence or absence of a region where the gradation is crushed as the gradation smashing information output from the smashing detection unit 1203_1, and the signal level i where the smashing occurs. To get. In the case where a plurality of grayscale areas having different signal levels exist, the signal level i in which the respective grayscale levels are generated is acquired.

  In ST1402, the predetermined subject presence / absence estimation result Ei at the signal level i where the collapse has occurred is acquired from the predetermined subject presence / absence estimation unit 1203_2.

  In ST1403, it is determined whether or not the corresponding predetermined subject presence / absence estimation result Ei has been acquired for all the crushing signal levels i, and if all are acquired, the process proceeds to ST1405, and if not, the process proceeds to ST1404.

In ST1404, the value of the signal level i in which the collapse has occurred is updated to the value of i for which the predetermined subject presence / absence estimation result Ei has not been acquired, and the flow proceeds to ST1402.

  In ST1405, the importance of correction for each signal level i is determined based on the predetermined subject presence / absence estimation result Ei. Here, importance is assigned so that the priority of correction with respect to each signal level i increases in descending order of the predetermined subject presence / absence estimation result Ei. For example, when blackout occurs at a dark signal level ldk and overexposure occurs at a light signal level llt in the video, the predetermined subject presence / absence estimation results Eldk and Ellt are compared for each signal level, and Ellt is If it is greater than Eldk, 1 is set as the importance of correction for the signal level llt and 2 is set as the importance of correction for the signal level ldk so that the correction for overexposure is given priority. Here, it is assumed that the smaller the correction importance, the higher the correction priority.

  In ST1406, the exposure amount correction amount is determined based on the correction importance level at the signal level i where the collapse has occurred and the predetermined subject presence / absence estimation result Ei. Here, the signal level i having the highest correction importance is selected, and the exposure amount correction amount is determined so that the exposure amount correction amount increases as the predetermined subject presence / absence estimation result Ei increases. In the above-described example, since the correction importance for the signal level llt is highest at 1, the correction amount is determined so that the exposure amount is reduced, so that the whiteout at the signal level llt is improved.

  In ST1407, the correction characteristic of the signal level correction is determined based on the importance of correction at the signal level i where the collapse has occurred and the predetermined subject presence / absence estimation result Ei. Here, the signal level i having the highest correction importance is selected, and the correction characteristic is determined so that the correction intensity of the signal level correction amount at the signal level i increases as the predetermined subject presence / absence estimation result Ei increases. In the above-described example, since the importance of correction for the signal level llt is highest at 1, the correction characteristic is determined so that the correction strength of the signal level correction processing for the signal level llt is increased, thereby improving whiteout at the signal level llt. Like that.

  Even in such a situation where gradation collapse occurs in the video and it is difficult to detect a predetermined subject in a region where gradation is lost due to image recognition or the like, the exposure is performed only when there is a high possibility that the predetermined subject exists. The visibility of a predetermined subject can be improved by performing an amount correction and a signal level correction process. Also, in the case of an exclusive operation in which one of the visibility is improved and one of the visibility is reduced by controlling the exposure, such as when overexposure and overexposure occur simultaneously, depending on which area It is possible to estimate whether there is an important subject and improve the visibility of the more important subject with priority.

  As an example of different processing for determining the exposure amount correction amount and the signal level correction strength using the correction importance level, the correction importance level is set as a value having the same ratio as the predetermined subject presence / absence estimation result Ei for each signal level i. The correction amount of the exposure amount and the correction of the signal level calculated by weighting and adding the correction amount of the exposure amount and the correction strength of the signal level calculated from the predetermined subject presence / absence estimation result Ei for each signal level i. You may make it use intensity | strength. In this case, when the overexposure and underexposure occur in the video at the same time, if the possibility that a predetermined subject exists in the overexposure area is much higher than the possibility that a fixed subject exists in the underexposure area, Control the exposure to suppress overexposure, and control the exposure to suppress overexposure when the possibility that a predetermined subject exists in the overexposure area is slightly higher than the possibility that a fixed subject exists in the overexposure area Thus, it is possible to make a picture considering the balance.

  In ST1405, instead of calculating the correction importance based on the predetermined subject presence / absence estimation result Ei, the correction importance set by the user input from the correction importance setting section 1205 may be used. In this case, it is possible to prioritize and improve the visibility of the area where the gradation is crushed at the signal level that the user wants to see.

  FIG. 15 is a second diagram illustrating a first example of the image signal correction control process of the image processing apparatus according to the third embodiment of the present invention.

  15A is an application example of the image signal correction control process when a predetermined subject exists in a whiteout region in the video, and FIG. 15B is a predetermined example in a blackout region in the video. This is an application example of image signal correction control processing when there is a subject.

  FIG. 15A shows an image of a room taken in the daytime, and the overexposed areas in the image are caused by simultaneously photographing a very bright outdoor from a window. In this case, the predetermined subject existence estimation evaluation value Ellt in the whiteout region is a large value, and the predetermined subject presence estimation evaluation value Eldk in the blackout region is a small value. Therefore, in order to determine the correction amount of the exposure amount and the correction characteristic of the signal level correction process so that the importance of correction with respect to the signal level llt is increased, the correction strength with respect to the overexposed region is large, and the correction strength of the dark portion is decreased. It is possible to improve the visibility by giving priority to the subject in the whiteout region. FIG. 15 (b) is an image taken on the road at night. The overexposure area in the image is generated by the streetlight as a point light source, and the blackout area in the image is caused by underexposure due to insufficient illuminance. Has occurred. In this case, the predetermined subject existence estimation evaluation value Ellt in the whiteout region is a small value, and the predetermined subject existence estimation evaluation value Eldk in the blackout region is a large value. Therefore, in order to determine the correction amount of the exposure amount and the correction characteristic of the signal level correction process so that the correction importance for the signal level ldk is high, the correction strength for the overexposed region is small, and the dark portion correction strength is large. It is possible to improve the visibility by giving priority to the subject in the blackout area.

  As described above, even when overexposure and underexposure occur at the same time, it is possible to estimate a region where a more important subject exists and to improve the visibility of the more important subject with priority.

  FIG. 16 is a first diagram illustrating a second example of a processing sequence related to the image signal correction control process of the image processing apparatus according to the third embodiment of the present invention. In the present invention, the image signal correction control process is performed by the image signal correction control unit 1203_3.

  In the processing sequence related to the image signal correction control processing shown in FIG. 16, in ST1601, the presence or absence of a region where the gradation is crushed in the video as the gradation crushed information output from the gradation crushed detection unit 1203_1, and the signal level i where the crushed has occurred. To get. In the case where a plurality of grayscale areas having different signal levels exist, the signal level i in which the respective grayscale levels are generated is acquired.

  In ST1602, the predetermined subject presence / absence estimation result Ei at the signal level i whose gradation is crushed is acquired from the predetermined subject presence / absence estimation unit 1203_2.

  In ST1603, it is determined whether or not the corresponding predetermined subject presence / absence estimation result Ei has been acquired for all the crushing signal levels i. If all are acquired, the process proceeds to ST1605, and if not, the process proceeds to ST1604.

  In ST1604, the value of the signal level i that has been crushed is updated to the value of i for which the predetermined subject presence / absence estimation result Ei has not been acquired, and the flow proceeds to ST1602.

  In ST1605, a blackout area and a whiteout area are both present, the predetermined subject presence / absence estimation result Eldk for the signal level ldk of the area where the blackout area is larger than a predetermined threshold th1, and an overexposed area. It is determined whether or not the predetermined subject presence / absence estimation result Ellt for the signal level llt is greater than a predetermined threshold th2. If both are greater than the predetermined threshold th1, the process proceeds to ST1606, otherwise the process proceeds to ST1608.

  In ST1606, the dynamic range expansion unit 1206 determines the exposure control amount of the imaging unit 1201 for performing video composition processing. Here, a first exposure time is set for improving blackout in the first frame, and a second exposure time is set for improving overexposure in the second frame. The first exposure time is determined so that the exposure time becomes longer as the predetermined subject presence / absence estimation result Eldk with respect to the signal level ldk increases, and the second exposure time is increased with the predetermined subject presence / absence estimation result Ellt with respect to the signal level llt. It determines so that exposure time may become short.

  In ST1607, the dynamic range expansion unit 1206 determines signal processing characteristics for performing video composition processing. Here, when the video of the first frame and the video of the second frame are synthesized, the characteristic that the synthesis ratio of the video of the first frame increases as the predetermined subject presence / absence estimation result Eldk with respect to the signal level ldk increases. And

  In ST1608, the importance of correction for each signal level i is determined based on the predetermined subject presence / absence estimation result Ei. Here, importance is assigned so that the priority of correction with respect to each signal level i increases in descending order of the predetermined subject presence / absence estimation result Ei.

  In ST1609, a correction amount for the exposure amount is determined based on the importance of correction at the signal level i where the gradation is crushed and the predetermined subject presence / absence estimation result Ei. Here, the signal level i having the highest correction importance is selected, and the exposure amount correction amount is determined so that the exposure amount correction amount increases as the predetermined subject presence / absence estimation result Ei increases.

  In ST1610, a correction characteristic for signal level correction is determined based on the importance of correction at the signal level i where the gradation is crushed and the predetermined subject presence / absence estimation result Ei. Here, the signal level i having the highest correction importance is selected, and the correction characteristic is determined so that the correction intensity of the signal level correction amount at the signal level i increases as the predetermined subject presence / absence estimation result Ei increases.

  Even in such a situation where gradation collapse occurs in the video and it is difficult to detect a predetermined subject in a region where gradation is lost due to image recognition or the like, the exposure is performed only when there is a high possibility that the predetermined subject exists. The visibility of a predetermined subject can be improved by performing an amount correction and a signal level correction process. In particular, when overexposure and underexposure occur at the same time, and there are important subjects in both, the visibility of both can be improved by using the dynamic range expansion process, and overexposure and underexposure occur simultaneously. However, when there is only one important subject, the visibility of only the area where the important subject exists is improved without using the dynamic range expansion process. Evil can be avoided.

  In order to simplify the control, the exposure control amount for the dynamic range expansion unit 1206 to perform video composition processing and the signal processing characteristics of the video composition processing are set to predetermined values, and only the presence or absence of the video composition processing is determined. A method of switching from a comparison result between Eldk and Ellt and a predetermined threshold may be adopted.

  Further, when the predetermined subject presence / absence estimation unit 1203_2 calculates the predetermined subject presence / absence estimation result Eldk for the signal level ldk and the predetermined subject presence / absence estimation result Ellt for the signal level llt, a still subject is assumed as the predetermined subject, and information on the shooting scene is obtained. A method of calculating a movement vector, which is a movement amount of pixels between successive time axes, and using an integral value in the screen to determine such that Eldk and Ellt become smaller as the integral value of the movement vector becomes larger may be employed. . At this time, if there is an important subject in the overexposure area or the underexposure area, and if the subject is moving at high speed, Eldk and Ellt are small, so that signal level correction processing is performed without performing dynamic range expansion processing. Is called. For this reason, it is possible to prevent the occurrence of a residual image of a video by combining a plurality of videos having different exposure times when the dynamic range expansion process is performed.

  FIG. 17 is a second diagram illustrating a second example of the image signal correction control process of the image processing apparatus according to the third embodiment of the present invention.

  17A is an application example of the image signal correction control process in the case where a predetermined subject exists in both the blackout area and the overexposed area in the video, and FIG. This is an application example of the image signal correction control process when a predetermined subject exists in the overexposed region.

  FIG. 17 (a) is an image of the entrance taken during the day. The blackened area in the image is taken from a very dark room and the overexposed area in the image is taken from a very bright outdoor view from the window. Has arisen. In this case, both the predetermined subject existence estimation evaluation value Ellt in the whiteout region and the predetermined subject existence estimation evaluation value Eldk in the blackout region are large values. Therefore, the dynamic range expansion process is effective, and the visibility of the subject in the blackout area and the subject in the overexposure area can be improved at the same time. FIG. 17B is an image of the room taken during the day, and the overexposed areas in the image are caused by simultaneously taking a very bright outdoor from the window. In this case, the predetermined subject existence estimation evaluation value Ellt in the whiteout region is a large value, and the predetermined subject presence estimation evaluation value Eldk in the blackout region is a small value. For this reason, the dynamic range expansion process is disabled, and the exposure correction amount and signal level correction correction characteristics are determined so that the correction strength for the overexposed area is increased. It is possible to improve the property.

  Thus, even when overexposure and underexposure occur at the same time, it is possible to adaptively switch the behavior of the dynamic range expansion process according to whether or not an important subject exists, thereby improving the visibility.

  As described above, according to the present embodiment, it is estimated with high accuracy whether or not a predetermined subject exists in an area where the gradation is broken in the video, and when the predetermined subject exists, the visibility is improved. However, if it does not exist, excessive correction can be suppressed, and it is possible to provide an image with improved visibility and image quality adapted to various scenes. In particular, by controlling the exposure, it is possible to improve the visibility even for a subject in which overexposure or underexposure occurs at the time of imaging.

  FIG. 18 is a schematic diagram showing an image processing apparatus and an imaging apparatus according to the fourth embodiment of the present invention. In FIG. 18, 1801 is an image input unit, 1802 is an image signal correction unit, 1803 is a control unit, 1803_1 is a gradation collapse detection unit, 1803_2 is a predetermined subject presence / absence estimation unit, 1803_3 is an image signal correction control unit, and 1804 is an input unit. , 1805 is a correction importance setting unit, 1806 is a dynamic range expansion unit, 1807 is an image memory unit, 1808 is a communication unit, and 1809 is an image processing apparatus including an image processing program for realizing the image input unit 1801 to the communication unit 1808. , 1810 is an image pickup apparatus having an exposure adjustment function.

  The characteristic difference between the image processing apparatus according to the fourth embodiment of the present invention shown in FIG. 18 and the image processing apparatus according to the third embodiment of the present invention shown in FIG. Instead, it is performed by the imaging device 1810 having an exposure adjustment function, and the exposure amount correction amount can be transmitted as an adjustment value from the image processing device 1809 to the imaging device 1810 having the exposure adjustment function via the communication unit 1808.

  In the image processing apparatus and the imaging apparatus shown in FIG. 18, an image input unit 1801 is connected to an imaging apparatus 1810 having an exposure adjustment function via a capture board connected to a video cable, a LAN, a USB, or the like, and has an exposure adjustment function. An image signal captured by the imaging device 1810 provided is input. The image signal correction control unit 1803_3 uses the gradation collapse information output from the gradation collapse detection unit 1203_1 and the predetermined subject presence / absence estimation evaluation value output from the predetermined subject presence / absence estimation unit 1203_2 to generate a predetermined subject in the area where the gradation is destroyed. Is determined, the exposure amount correction amount of the imaging device 1810 having the exposure adjustment function is determined so as to expand the gradation of the signal level at which the collapse occurs, and is output to the communication unit 1808. The communication unit 1808 transmits the exposure amount correction amount to the imaging device 1810 having an exposure adjustment function by communication via a serial port, a parallel port, or a LAN. As a result, even if the exposure-adjustable imaging device and the image processing device that performs the signal level correction process are different devices, it is possible to accurately determine whether or not a predetermined subject exists in the area where the gradation collapse has occurred in the video. The visibility can be improved when a predetermined subject exists, and excessive correction can be suppressed when the subject does not exist.

  In addition, this invention is not limited to an above-described Example, Various modifications are included.

  The above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described.

  Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment.

  The present invention can be used for an imaging device, a PC application connected to the imaging device, an image processing device, and the like, for example, in consumer, surveillance, in-vehicle, and business applications.

0101: Image input unit, 0102: Image signal correction unit, 0103: Control unit, 0103_1: Tone loss detection unit, 0103_2: Presence subject presence / absence estimation unit, 0103_3: Image signal correction control unit, 0104: Input unit, 0201: Image Video signal input from the input unit 0101, 0202_1: representative subject area existing in the dark part of the video signal, 0202_2: representative subject area existing in the bright part of the video signal, 0301: input from the image input part 0101 0302: Video signal input from the image input unit 0101, 0801: Video signal input from the image input unit 0101, 0802: Video signal input from the image input unit 0101 Region where fog is very dark and gradation of intermediate luminance is crushed, 1201: imaging unit, 1202: image signal correction unit, 203: control unit, 1203_1: gradation collapse detection unit, 1203_2: predetermined subject presence / absence estimation unit, 1203_3: image signal correction control unit, 1204: input unit, 1205: correction importance setting unit, 1206: dynamic range expansion unit, 1207 : Image memory unit, 1801: image input unit, 1802: image signal correction unit, 1803: control unit, 1803_1: gradation collapse detection unit, 1803_2: predetermined subject presence / absence estimation unit, 1803_3: image signal correction control unit, 1804: input , 1805: correction importance setting unit, 1806: dynamic range expansion unit, 1807: image memory unit, 1808: communication unit, 1809: image including an image processing program for realizing from image signal input unit 1801 to communication unit 1808 Processing device 1810: Imaging device with exposure adjustment function

Claims (14)

An image input unit;
An image signal correction unit that performs a signal level correction process on the image signal input from the image input unit and outputs a level-corrected image signal;
A gradation collapse detection unit that detects the presence or absence of a region with a collapsed gradation from the image signal input from the image input unit;
A predetermined subject presence / absence estimation unit that estimates the presence / absence of a predetermined subject in the area where the gradation is crushed from the image signal input from the image input unit;
An image signal correction control unit that controls input / output characteristics of signal level correction processing of the image signal correction unit based on a detection result of the gradation collapse detection unit and an estimation result of the predetermined subject presence / absence estimation unit. An image processing apparatus. The image processing apparatus according to claim 1.
The image signal correction control unit increases the degree of correction in the area where the gradation is lost when the predetermined object presence / absence estimation section estimates that the predetermined object exists within the area where the gradation is lost. An image processing apparatus that controls input / output characteristics of signal level correction processing. The image processing apparatus according to claim 1.
The predetermined subject presence / absence estimation unit estimates the presence / absence of a predetermined subject based on information on a shooting scene obtained by processing the image signal input from the previous image input unit or information on a shooting scene obtained from an external device. An image processing apparatus. The image processing apparatus according to claim 3.
The image processing apparatus according to claim 1, wherein the predetermined subject presence / absence estimation unit acquires a plurality of different types of information related to a shooting scene, and estimates the presence / absence of the predetermined subject by combining the plurality of different types of information. The image processing apparatus according to claim 3.
The information relating to the shooting scene includes at least one of time zone information at the time of shooting, weather information, position or area or shape information in the image of the area where the gradation is crushed, and movement information of the subject in the image. An image processing apparatus. The image processing apparatus according to claim 1.
When the gradation collapse detection unit detects a plurality of gradation collapsed regions having different signal levels from the image signal input from the image input unit, the predetermined subject presence / absence estimation unit detects the plurality of floors having different signal levels. The presence / absence of a predetermined subject is estimated in each of the areas where the tones are lost, and the image signal correction control unit determines the signal level based on the detection result of the gradation collapse detection unit and the estimation result of the predetermined subject presence / absence estimation unit. An image processing apparatus that controls input / output characteristics of a signal level correction process of the image signal correction unit for each of a plurality of different gray-scaled areas. The image processing apparatus according to claim 6.
The predetermined subject presence / absence estimation unit estimates the presence / absence of a predetermined subject corresponding to each signal level based on information on the shooting scene for a plurality of gray-scaled regions having different signal levels. A featured image processing apparatus. The image processing apparatus according to claim 6.
The image signal correction control unit determines a correction importance level of a plurality of gradation-depleted regions having different signal levels based on the detection result of the gradation-smashing detection unit and the estimation result of the predetermined subject presence / absence estimation unit. And a signal level correction process for each of a plurality of gradation-depleted areas having different signal levels using the correction importance, the detection result of the gradation collapse detection unit, and the estimation result of the predetermined subject presence / absence estimation unit. An image processing apparatus that controls the degree of correction. The image processing apparatus according to claim 6.
The image signal correction control unit uses a correction importance level corresponding to a signal level with a crushed gradation, a detection result of the gradation smashing detection unit, and an estimation result of the predetermined subject presence / absence estimation unit, so that the signal level differs. An image processing apparatus that controls a degree of correction of a signal level correction process for each of a plurality of gray-scaled areas. The image processing apparatus according to claim 9.
An image processing apparatus comprising: a correction importance setting unit for setting the correction importance. The image processing apparatus according to claim 1.
The image processing apparatus according to claim 1, wherein the image signal input unit includes an imaging unit, and inputs a signal obtained by imaging by the imaging unit as an image signal. The image processing apparatus according to claim 11.
The image signal correction control unit inputs the exposure state of the imaging unit and the signal level correction process of the image signal correction unit based on the detection result of the gradation collapse detection unit and the estimation result of the predetermined subject presence / absence estimation unit. An image processing apparatus that controls output characteristics. The image processing apparatus according to claim 12, further comprising:
A dynamic range expansion unit that synthesizes a plurality of image signals with different exposure timings captured by the imaging unit by signal processing;
The image signal correction control unit is configured to determine whether or not to perform synthesis processing of a plurality of image signals of the dynamic range expansion unit based on the detection result of the gradation collapse detection unit and the estimation result of the predetermined subject presence / absence estimation unit. An image processing apparatus for controlling the signal processing characteristics of the image processing apparatus. The image processing apparatus according to claim 1.
A communication unit connected to an external exposure variable imaging unit to communicate information,
The image signal correction control unit is configured to detect an exposure state of the external imaging unit and a signal level correction process of the image signal correction unit based on a detection result of the gradation collapse detection unit and an estimation result of the predetermined subject presence / absence estimation unit. Control the input / output characteristics of
The image processing apparatus, wherein the communication unit outputs control information of an exposure state of the external imaging unit by the image signal correction control unit to the external imaging device.

Images