WO2022059139A1

WO2022059139A1 - Image display device and image display method

Info

Publication number: WO2022059139A1
Application number: PCT/JP2020/035284
Authority: WO
Inventors: 勇人菊田; 俊明久保
Original assignee: 三菱電機株式会社
Priority date: 2020-09-17
Filing date: 2020-09-17
Publication date: 2022-03-24
Also published as: JPWO2022059139A1; JP7355252B2

Abstract

According to the present invention, a synthetic image (13) is generated in which, in an image affected by a flare, a light source location (41) is detected from the brightness value of an image pickup input image (110), and in at least a part of a flare region (42) determined from the center of the light source location (41), with regard to a corrected image (12) and an input image (11), as the flare region (42) becomes closer to the center of the light source location (41), a proportion of the corrected image (12) becomes larger, and as the flare region (42) becomes farther away therefrom, a proportion of the input image (11) becomes larger. Furthermore, the synthetic image (13) is generated in a pixel region in which the synthetic image (13) is generated, the corrected image (12) or the input image (11) is generated within the flare region (42) in a pixel region in which the synthetic image (13) is not generated, and a display image displaying the input image (11) is generated outside the flare region (42). Accordingly, a discontinuity in the brightness value of a boundary between the flare region (42) and a region other than the flare area (42) can be suppressed, and thus an image having high visibility can be displayed.

Description

Image display device and image display method

This disclosure relates to an image display device such as an electronic mirror device and an image display method.

In image display devices such as conventional electronic mirror devices, for example, image correction that reduces the brightness value is performed so that the user's visibility is not impaired by the influence of flare generated by a strong local light source such as the headlight of a vehicle behind. It has been devised to do. Further, for example, in Patent Document 1, the brightness value of the concentric region from the center of the high-luminance region by the headlight of the rear vehicle in the image of the rear of the own vehicle is reduced, and the brightness value of the central portion is reduced. Image correction that suppresses the rate is performed to brighten only the central part of the headlights of the rear vehicle so that the driver can easily grasp the position of the rear vehicle.

JP-A-2015-198302

However, when the brightness value of the area affected by the flare is reduced, there is a problem that a discontinuous portion appears at the boundary with the area where the brightness value is not reduced and the visibility is lowered.

The present disclosure has been made to solve the above-mentioned problems, and an object of the present disclosure is to provide an image display device for displaying an image with improved visibility.

In the image display device according to the present disclosure, an image pickup unit that images the outside of a vehicle and pixels in which pixels having a preset first brightness value or higher in the image pickup input image captured by the image pickup unit are gathered adjacent to each other. A light source position detection unit that detects an area as a light source position, and a flare area determination unit that determines a pixel area having a preset second brightness value or more as a flare area from a pixel area around the center of the light source position. A correction image generation unit that generates a correction image by reducing the brightness value of pixels of at least the input image in the flare area of the captured input image that are equal to or higher than the preset third brightness value, and at least the flare area. In a part of, the corrected image and the input image are combined with a composite image generation unit that generates a composite image by increasing the ratio of the corrected image as it is closer to the center of the light source position and increasing the ratio of the input image as it is farther from the center. A composite image is provided in the pixel area where the image is generated, a corrected image or an input image is provided in the flare area among the pixel areas where the composite image is not generated, and an image display unit is provided outside the flare area to display the input image. It is an image.

The image display method according to the present disclosure includes a step of detecting as a light source position a pixel region in which pixels having a preset first brightness value or higher in an image captured by capturing the outside of a vehicle are adjacent to each other. A step of determining a pixel region in which the brightness value of the pixels around the center of the light source position is equal to or higher than a preset second brightness value as a flare region, and an input image of at least the flare region of the captured input images. The step of generating a corrected image by reducing the brightness value of the pixel equal to or higher than the preset third brightness value of, and the corrected image and the input image at the center of the light source position at least in a part of the flare region. The closer it is, the larger the proportion of the corrected image, and the farther it is, the greater the proportion of the input image to generate a composite image. Among them, a correction image or an input image is provided in the flare area, and a display image for displaying the input image is generated outside the flare area.

According to the present disclosure, the ratio of the input image to the corrected image is close to the center of the light source position in at least a part of the flare region determined from the center of the light source position by detecting the light source position from the brightness value of the captured input image. Since the composite image generated by increasing the proportion of the corrected image and increasing the proportion of the input image is displayed, the discontinuity of the brightness value of the displayed image can be suppressed and the image with high visibility can be displayed. ..

FIG. 1 is a schematic configuration diagram of the image display device of the first embodiment. FIG. 2 is a schematic block diagram of the image display device of the first embodiment. 3A to 3C are explanatory views illustrating control of the image processing unit of the first embodiment. 4A and 4B are graphs showing the function F used in the composite image generation unit of the first embodiment. 5A to 5D are examples of processed images in the image display device of the first embodiment. FIG. 6 is an example of a time-by-time image capture input image for explaining the second embodiment. FIG. 7 is a schematic block diagram of the image display device of the second embodiment. FIG. 8 is a schematic block diagram of the image display device of the third embodiment.

Embodiment 1.
The image display device 1 according to the first embodiment will be described with reference to FIG. The image display device 1 includes an image pickup unit 2 that captures an image of the outside of the own vehicle, an image processing unit 3 that processes the captured image, and an image display unit 4 that displays the processed image.

The image pickup unit 2 has one or more cameras that output an image pickup input image 110 that images the outside of the own vehicle. In the present embodiment, an example in which the image pickup unit 2 takes an image of the situation behind the own vehicle will be described. As the camera, for example, a two-dimensional image pickup device using a CMOS (Complementary MOSFET) or a CCD (Charge Coupled Device) element as an image sensor may be used.

The image display unit 4 is mounted in the own vehicle and is arranged at a position suitable for the user to see the image display unit 4. For example, it is arranged near the A-pillars on the left and right sides of the front surface of the own vehicle, near the door trim in front of the side surface of the own vehicle, and the like. The image display unit 4 uses a display device having a backlight composed of, for example, a liquid crystal filter such as an LCD (Liquid Crystal Display) panel, an LED (Light Emitting Diode), and a diffuser. Further, it may be a display device in which LEDs that emit organic light are arranged in pixel units. The image display unit 4 displays a display image obtained by processing the image pickup input image 110 described later.

The image processing unit 3 is partially or wholly composed of a processing circuit. For example, the plurality of functions of the image processing unit 3 may be realized by separate processing circuits, or the functions of the plurality of parts may be collectively realized by one processing circuit. Further, the processing circuit may be configured by hardware or software, that is, a programmed computer. Further, among the functions of each part of the image processing unit 3, a part may be realized by hardware and the other part may be realized by software. As the hardware, for example, FPGA (Field-Programmable Gate Array), microprocessor, microcontroller, DSP (Digital Signal Processor) or the like may be used. The software is, for example, a program that executes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like on a processor.

The image processing unit 3 will be described with reference to FIGS. 2 and 3A to 3C. FIG. 2 is a schematic block diagram of the image display device 1. 3A to 3C are explanatory views for explaining the processing control of the image processing unit, FIG. 3A shows an image pickup input image, FIG. 3B shows a light source position information, and FIG. 3C shows an example of flare region information.
As shown in FIG. 2, the image processing unit 3 includes a light source position detection unit 21, a flare area determination unit 22, a correction image generation unit 23, and a composite image generation unit 24. In the image processing unit 3, the light source position information 31 indicating the light source position 41 detected by the light source position detection unit 21 from the image pickup input image 110 is output to the flare area determination unit 22 and the composite image generation unit 24, and the flare area determination unit 22 outputs the light source position information 31. The flare area information 32 indicating the determined flare area 42 is output to the composite image generation unit 24.
Further, the corrected image generation unit 23 generates a corrected image 12 in which the brightness value of the input image 11 in at least the flare region 42 of the captured input image 110 is reduced, and outputs the corrected image 12 to the composite image generation unit 24. Then, the composite image generation unit 24 generates a composite image 13 in which the composite ratio of the input image 11 and the corrected image 12 is changed based on the distance D from the center of the light source position 41 in at least a part of the flare region 42. The composite image 13 is placed in the region where the composite image 13 is generated, and the corrected image 12 or the corrected image 12 is set in the flare region 42 including the end De of the flare region 42 in the region where the composite image 13 is not generated. The image including the input image 11 is displayed on the image display unit 4 in the area outside the flare area 42.
When a plurality of flare areas 42 are determined, the above-mentioned image processing is performed in each of the flare areas 42, and the input image 11 is displayed on the image display unit 4 in the areas other than the flare areas 42.

The light source position detection unit 21 inputs the image pickup input image 110 shown in FIG. 3A output from the image pickup unit 2, and the pixels having the first luminance value Th1 or more set in advance in the image pickup input image 110 are adjacent to each other. The gathered pixel region is detected as the light source position 41, and the light source position 41 is used as the light source position information 31 shown in FIG. 3B. The light source position information 31 is, for example, two-dimensional map information having the same size as the captured input image 110, and indicates the light source position 41 in which the white portion in FIG. 3B is detected.
For example, the light source position detection unit 21 performs gradation processing on the pixels of the image pickup input image 110 to calculate the luminance value, and detects the high-luminance pixel whose luminance value is equal to or higher than the first luminance value Th1 as the light source position 41. do. For the calculation of the luminance value, for example, when the configuration of the image pickup element is a gradation value of three colors obtained from an RGB color filter, it is converted into a YCbCr color space composed of a luminance component and a color difference component, and the Y component is used as the luminance value. .. The luminance value becomes high when strong light is projected onto the imaging device, when imaging noise or circuit noise appears locally, and the like. In general, a high-intensity light source that causes flare to appear is limited to a light source having a large pixel area. Therefore, the light source position detection unit 21 calculates the area of the area where the pixels having the first luminance value Th1 or more are adjacent to each other, and sets the area of the area equal to or more than the preset area as one light source position 41. When photographing the rear of the own vehicle, the object to be visually recognized by the user is the rear vehicle 51 behind the own vehicle, and the range in which the rear vehicle 51 exists within the angle of view of the camera is limited. Therefore, the pixel for calculating the luminance value by the light source position detection unit 21 may be limited to the range in which the rear vehicle 51 exists.

The flare area determination unit 22 determines the flare area 42 by using the light source position information 31 obtained from the light source position detection unit 21, and uses it as the flare area information 32. The flare region information 32 is, for example, two-dimensional map information having the same size as the captured input image 110 as shown in FIG. 3C, and is information on a pixel region having high brightness around the center of the light source position 41. For example, the white portion in FIG. 3C is the determined flare region 42.

A method for determining the flare region 42 will be described. First, peripheral pixels are extracted in order from the center of the light source position 41 obtained in the light source position information 31, and the luminance values of the surrounding pixels are processed and calculated in the same manner as the light source position detection unit 21. The center of the light source position 41 is, for example, the center of gravity of the pixel region detected as the light source position 41 by the light source position detection unit 21.
Specifically, the pixel region in which the luminance value of the pixels around the center of the light source position 41 detected by the light source position detection unit 21 is equal to or greater than the preset second luminance value Th2 is determined as the flare region 42. ..

Next, the corrected image generation unit 23 refers to at least the input image 11 in the flare region 42 of the captured input image 110, and reduces the brightness value of the pixel having the third brightness value Th3 or more set in advance. Twelve is generated.
The corrected image 12 may be one in which the luminance value is reduced only for the pixels having the third luminance value Th3 or more to remove the high luminance component, and both the third luminance value Th3 or more and the third luminance value Th3 or not are reduced. It may be something to make.
For example, since the pixels obtained by imaging the headlight 52 of the rear vehicle 51 have a very high luminance value, the pixels around the headlight 52 have a large luminance value due to the influence of flare.
Since the luminance value increased due to the influence of flare increases as the distance from the center of the light source position 41 increases, by focusing on the pixels around the center of the light source position 41, the third luminance value Th3 or more Flare can be suppressed by reducing the brightness value of the pixel. The setting of the minimum value of the third brightness value Th3 is affected by the sensitivity of the image sensor in the image pickup unit 2 and the allocation to the digital value by HDR (High Dynamic Range) conversion, so it depends on the performance of the equipment to be used and the image pickup environment. Determined as a parameter based on.

The corrected image generation unit 23 may perform a process of reducing the luminance value of the pixel having the above-mentioned third luminance value Th3 or more, for example. For the generation of the corrected image 12, for example, a method of preparing a table for determining an output value based on the luminance value of the captured input image 110 by a LUT (Look Up Table) and generating the corrected image 12 can be mentioned. In this case, the luminance value of the input pixel is output for the luminance value smaller than the third luminance value Th3, and the luminance value of the input pixel is reduced when the luminance value is equal to or greater than the third luminance value Th3. Set the output so that it is the value that was set.
The luminance value increased due to the influence of flare becomes higher as the distance from the center of the light source position 41 becomes closer, so that a large reduction is required. Therefore, as the LUT parameter to be prepared becomes higher in brightness than the third luminance value Th3, for example, linear, Gaussian, and exponential function parameters may be given to reduce the luminance value. The LUT parameter may be a one-dimensional input parameter using the luminance value of the pixel as an input value, or may be a multidimensional input parameter including the value of each RGB color and the chromaticity. A LUT may be constructed only in the flare region 42 determined by the flare region determination unit 22 to generate the corrected image 12. Further, the corrected image 12 may be generated beyond the flare region boundary, which is the boundary between the flare region 42 and the region other than the flare region 42. That is, the correction image 12 may be generated by the correction image generation unit 23 in the entire image pickup input image 110, or not in the entire image pickup input image 110. It may be carried out at least in the pixel region for generating the composite image 13 described later.

The composite image generation unit 24 uses the input image 11, the light source position information 31, the flare area information 32, and the corrected image 12, which are at least a part of the captured input image 110, and the closer to the center of the light source position 41, the closer the corrected image 12 is. The ratio is increased, and the ratio of the captured input image 110 is increased as the ratio is increased. For example, a composite image 13 in which the corrected image 12 and the input image 11 are combined over the entire flare region 42 is generated and output to the image display unit 4.

The method of generating the composite image 13 in the composite image generation unit 24 will be described. For example, when the image obtained by synthesizing the input image 11 and the corrected image 12 in the flare area 42 at a constant ratio and the input image 11 outside the flare area 42 are displayed together, a discontinuity of the luminance value appears at the boundary of the flare area. It occurs and visibility is reduced. Therefore, the ratio of combining the input image 11 and the corrected image 12 is changed based on the distance D from the center of the light source position 41. That is, the closer to the center of the light source position 41, the smaller the composite ratio Ba of the input image 11, the larger the composite ratio Bb of the corrected image 12, and the farther from the center of the light source position 41, the larger the composite ratio Ba of the input image 11. The synthesis ratio Bb of 12 is reduced.

For example, FIGS. 4A and 4B are graphs showing the function F (D, De, Tc) used in the composite image generation unit 24, and the composite ratio Ba of the input image 11 and the distance D from the center of the light source position 41. Shown in relation.
The function F (D, De, Tc) includes the distance D from the center of the light source position 41, the composition ratio Ba of the input image 11, the distance De from the center of the light source position 41 to the end of the flare region 42, and the adjustment parameter Tc. It becomes a function. The composite ratio Bb of the corrected image 12 is a value obtained by subtracting the composite ratio Ba of the input image 11 from 1.
The distance D from the center of the light source position 41 is a variable, for example, the Euclidean distance of the line segment connecting the pixel for calculating the composition ratio and the pixel at the center of the light source position 41. The distance De is the distance between the pixel at the end of the flare region 42 and the pixel at the center of the light source position 41 when the line segment is extended toward the end of the flare region 42. Normally, the flare region 42 is not concentric from the center of the light source position 41, and the distance De from the center of the light source position 41 to the end of the flare region 42 varies depending on the selected line segment. The adjustment parameter Tc is, for example, a constant for adjusting the composite ratio Ba and the composite ratio with Bb at the end of the flare region 42, and adjusts the composite ratio Ba of the input image 11 at the position where D = De to adjust the flare region 42. It alleviates the discontinuity of the brightness value at the end of. That is, the function F (D, De, Tc) is a function of the composition ratio Ba of the input image 11 that approaches 0 as the output of the end portion of the flare region 42 is adjusted by Tc and moves toward the center of the light source position 41. As shown in FIG. 4A, the function F (D, De, Tc) sets the composite ratio Ba of the input image 11 at the center D = 0 of the light source position 41 to 0, and sets the composite ratio Ba of the end D = De of the flare region 42 to 0. The composition ratio Ba may be 1, and as shown in FIG. 4B, the composition ratio Ba of the input image 11 from the center of the light source position 41 to an arbitrary distance Dn is 0, and the composition ratio Ba from Dn to the end De of the flare region 42 is arbitrary. The composition ratio Ba of the input image 11 may be changed up to the distance De + ΔD obtained by adding ΔD. Further, the function F (D, De, Tc) may use an exponential function obtained by inversion of the Gaussian distribution, or may use a linear function.

As described above, the composite image generation unit 24 may generate the composite image 13 in the flare region 42, or may generate the composite image 13 in the region including the end of the flare region and its periphery. The composition ratio may be generated by changing it for each line segment, and the composition ratio may be changed at the end of the flare region and its periphery such as the pixel region from Dn to De + ΔD, and the end of the flare region 42. It may include a pixel area exceeding. In this case, in the region where the composition ratio is not changed, the composition ratio Ba of the input image 11 may be 0 as shown in FIG. 4B, or the input image 11 and the correction image 12 may be added together at an arbitrary ratio.

An example of image processing in the present embodiment will be described with reference to FIGS. 5A to 5D. FIG. 5A is an example of an image pickup input image 110 in which the rear right side of the own vehicle is imaged. Due to the influence of flare caused by the headlight 52 of the rear vehicle 51, the shape of the rear vehicle 51 in the vicinity of the headlight 52 and the road environment are difficult to visually recognize. Further, since the headlights 52 are arranged one on each side, the two flares are connected and visually recognized as one light source, making it difficult to grasp the position of the rear vehicle 51.
In FIG. 5B, the pixel region in which the pixels having the first luminance value Th1 or more in the image pickup input image 110 are adjacently gathered is detected as the light source position 41, and the luminance value of the pixels around the center of the light source position 41 is the second. This is an example in which a pixel region having a luminance value of Th2 or more is determined as a flare region 42.
FIG. 5C is an example of the corrected image 12 in which the image pickup input image 110 is obtained by reducing the luminance value of the pixel having the third luminance value Th3 or more as a whole.
FIG. 5D shows the composite image 13 in the region where the composite image 13 is generated, the corrected image 12 or the composite image 13 in the flare region 42 among the regions where the composite image 13 is not generated, and the region other than the flare region 42. Is an example of a display image in which the input image 11 is displayed. Specifically, in the flare region 42, the closer to the light source position 41, the larger the composite ratio Bb of the corrected image 12, and the farther it is, the larger the composite ratio Ba of the input image 11, and the composite of the input image 11 and the corrected image 12. The image 13 is generated, and the input image 11 is displayed in a region other than the flare region 42.

As described above, in the image affected by the flare, the image display device 1 according to the present embodiment has a second brightness from the pixel region around the center of the light source position 41 detected from the brightness value of the captured input image 110. A pixel region having a value Th2 or more is determined to be a flare region 42, and the brightness value of a pixel having a third brightness value Th3 or more of at least the input image 11 in the flare region 42 of the captured input image 110 is reduced and corrected. Image 12 is generated. Then, at least in a part of the flare region 42, the corrected image 12 and the input image 11 are combined by increasing the proportion of the corrected image 12 as it is closer to the center of the light source position 41 and increasing the proportion of the input image 11 as it is farther away. Image 13 is generated. Further, the composite image 13 is placed in the region where the composite image 13 is generated, and the corrected image 12 or the corrected image 12 includes the input image 11 in the flare region 42 in the region where the composite image 13 is not generated. Since the input image 11 is displayed on the image display unit 4 outside the flare area 42, the discontinuity of the brightness value at the boundary between the flare area 42 and the non-flare area 42 is suppressed, and the image has high visibility. Can be displayed.
Further, since it is not necessary to reduce the luminance value in the entire captured input image 110, it becomes easy to visually recognize notable information such as the shapes of surrounding buildings and the rear vehicle 51.

Further, by significantly reducing the corrected image 12 as the brightness value of the input image 11 increases, the influence of flare is suppressed and visual information such as the shape of the vehicle 51 behind the flare region boundary and the road environment becomes easier to visually recognize.

Further, by changing the composition ratio Ba of the input image 11 and the composition ratio Bb of the corrected image 12 for each line segment, the discontinuity of the brightness value at the boundary of the flare region is suppressed even if the flare shape is complicated. , Highly visible images can be displayed.

In the composite image generation unit 24, the generation of the composite image 13 from the center of the light source position 41 to the end De of the flare region 42, or from Dn to De + ΔD described above may be processed for each pixel, and the flare region may be processed. The function F (D, De, Tc) may be patterned and processed using a table based on the area of the pixel region of 42 or the light source position 41. It can be simplified by processing the pixel areas collectively.

Further, in the flare area determination unit 22, an example of determining a pixel area having a second luminance value Th2 or more as a flare region 42 from a pixel region around the center of the light source position 41 is shown, but the second luminance value is shown. Th2 may be changed based on the distance from the center of the light source position 41. The luminance value of the flare region 42 increases as the distance from the center of the light source position 41 increases. Therefore, for example, if the second luminance value Th2 is lowered sequentially from the center of the light source position 41, the flare region 42 can be more accurately formed. It can be determined. At this time, the luminance value that determines the end portion of the flare region 42 is Th2i, which is smaller than the second luminance value Th20 set near the center of the light source position 41. In this way, by setting the second luminance value Th2 larger as the distance from the center of the light source position 41 is shorter, even if the pixels around the light source position 41 are affected by flare and the luminance value is increased, the luminance value is increased. The flare region 42 can be appropriately determined.

Further, since the flare region 42 determined by the flare region determination unit 22 is a region continuous from the center of the light source position 41, if it is determined that the flare region 42 is not the flare region 42 in the above-mentioned determination of the flare region 42, the flare region 42 is determined. It can be determined that the region outside the determined pixel region is not the flare region 42. Therefore, it is better to start from the center of the light source position 41 and make a determination toward the outside.

Further, in order to simplify the determination of the flare region 42 based on the luminance value, the flare region 42 may be determined in a limited direction. For example, the end portion of the flare region 42 in eight directions is determined by advancing in order from the center of the light source position 41 with respect to a total of eight-direction axes of up / down / left / right and diagonal 45-degree axes. At that time, for the pixel region other than the on-axis, pay attention to the two neighboring axes, and draw the pixels that are the ends of the flare region 42 of the two axes into a straight line or an arc centered on the center of the light source position 41. The region located on the light source position 41 side of the connected line may be determined as the flare region 42. By such a method, the determination of the flare region 42 can be simplified, and the generation of the composite image 13 in the composite image generation unit 24 can also be simplified.
Further, the processing load can be reduced by limiting the pixel region for determining the flare region 42 within the range of the distance from the center of the preset light source position 41. For example, a distance proportional to the area of the pixel region of the light source position 41 obtained from the light source position information 31 may be set, or a plurality of adjacent pixels may be averaged and set as a group.

Embodiment 2.
The image display device 1 according to the present embodiment includes an image pickup unit 2, an image processing unit 3, and an image display unit 4 in the same manner as the schematic configuration of the first embodiment shown in FIG. The image processing unit 3 holds the front frame information 33 such as the light source position information 31 and the flare area information 32 obtained in the previous frame captured in the previous time zone, which can be referred to in the image processing of the current frame captured at the present time. The front frame information holding unit 25 is provided.

The flare area determination unit 22 determines a pixel region in which the luminance value around the center of the light source position 41 is equal to or greater than the second luminance value Th2 as the flare region 42. Since the light source position 41 is detected from the pixel region having the first luminance value Th1 or more in the image pickup input image 110, when noise appears in the input image 11, the flare region 42 in which the noise is erroneously detected as the light source position 41 is determined. There is a risk of Further, for example, when the road surface reflected light 53 of the headlight 52 of the rear vehicle 51 is projected and the pixel region having the second luminance value Th2 or more is large, it is determined to be the flare region 42, but when it is small, the flare region 42 is determined. Is not determined, and there is a risk that the displayed image may flicker. In the image display device 1 of the present embodiment, the camera used for the image pickup unit 2 that acquires the image pickup input image 110 usually takes images at intervals of 30 to 60 frames per second and recognizes noise and road surface reflected light 53. Focuses on the fact that is not all frames, and the display image is visually recognized even if noise is deleted or the flare area 42 is erroneously determined by referring to the information of the previous frame during image processing in the current frame. It does not reduce the sex.

FIG. 6 describes an example in which the road surface reflected light 53 is projected on the image pickup input image 110. FIG. 6 is an example of the image pickup input image 110 captured in the order of time T1 to T7, and the road surface reflected light 53 suddenly appears at time T4 to T6. Further, the luminance value of the road surface reflected light 53 changes and becomes maximum at time T5. Due to the influence of such a momentary change in the luminance value, if the road surface reflected light 53 is not determined to be the flare region 42 at the time T4 and T6, and is determined to be the flare region 42 at the time T5, the input image 11 is used as it is. When the composite image 13 combined with the corrected image 12 is displayed, the display image displayed on the image display unit 4 will flicker. Therefore, the light source position information 31 and the flare area information 32 of the front frame are held in the front frame information holding unit 25, and the composite image generation unit 24 generates the composite image 13 of the current frame using these information.

In the present embodiment, the image pickup input image 110 or the input image 11 before the current time captured by the image pickup unit 2 is used as a front frame (for example, if the current time is time T5, the front frame of the current frame at time T5 is time T1 to One of the frames of 4). It may also refer to multiple previous frames including the frame before the previous frame.
FIG. 7 is a schematic block diagram of the image display device 1 of the present embodiment. The image processing unit 3 includes a front frame information holding unit 25 that holds the light source position information 31 acquired by the light source position detecting unit 21 and the flare area information 32 acquired by the flare area determination unit 22 in the front frame. The processing performed by the light source position detection unit 21, the flare area determination unit 22, and the correction image generation unit 23 is the same as that of the first embodiment.

For the storage of the previous frame information 33 in the previous frame information holding unit 25, for example, a ring buffer having a memory area secured so that the information from the time T1 to the time T7 can be sequentially held is used.

The composite image generation unit 24 inputs the input image 11, the light source position information 31, the flare area information 32, the corrected image 12, and the front frame information 33 obtained from the front frame.

For example, when the position of the flare region 42 determined in the current frame is determined not to be the flare region 42 from the light source position information 31 of the front frame information holding unit 25, the composite image generation unit 24 synthesizes the input image 11. The composite image 13 is generated by increasing the ratio Ba to, for example, 0.7 or more.
In this way, when the noise of the captured input image 110, the road surface reflected light 53 appearing in a hurry, and the like are reflected in the image with reference to the front frame information 33, and the influence of flare is considered to be small, the brightness of the input image 11 is increased. By approaching the value, the flicker of the displayed image can be suppressed.

Further, the front frame information holding unit 25 may hold a plurality of front frame information. For example, if the pixel at the position determined to be the flare region 42 in the time T6 is determined to be the flare region 42 only in T5 in the previous frames of the times T1 to T5 held in the front frame information 33, it is determined to be the flare region 42. The percentage of unfilled frames is 4/5. In this case, for example, the composition ratio Ba of the input image 11 having D = 0, which is the center of the light source position 41 in FIG. 4A, is set to 4/5. Further, the composition ratio Ba of the input image 11 may be determined by using a function having the ratio of the number of previous frames that is not determined to be the flare region 42 as a variable or a LUT having the number of frames as the input value.
As described above, the ratio of the input image 11 and the ratio of the corrected image 12 in the composite image generation unit 24 are such that the larger the number of the front frames that are not determined to be the flare region 42 in the front frame captured by the image pickup unit 2, the larger the number of the input image 11. Increase the proportion and decrease the proportion of the corrected image.

Although an example has been described in which the front frame information 33 held by the front frame information holding unit 25 is the light source position information 31 and the flare area information 32 in the front frame, the light source position 41 and flare detected by the light source position detecting unit 21 have been described. The front frame information holding unit 25 may hold the information from which the information of the flare area 42 determined by the area determination unit 22 is obtained.
For example, the light source position 41 in the front frame is held by the front frame information holding unit 25, and the ratio of the input image 11 and the ratio of the corrected image 12 in the composite image generation unit 24 are set to the light source position in the front frame imaged by the image pickup unit 2. The ratio of the input image 11 may be increased and the ratio of the corrected image 12 may be decreased so that 41 is not detected.

Even in such an image display device 1, the light source position 41 is detected from the brightness value of the captured input image 110, and the input image 11 and the corrected image 12 are formed in at least a part of the flare region 42 determined from the center of the light source position 41. The composite image 13 generated by increasing the proportion of the corrected image 12 as it is closer to the center of the light source position 41 and increasing the proportion of the input image 11 as it is farther away is displayed, and the input image is displayed in the area other than the flare area 42. Since 11 is displayed, it is possible to suppress the discontinuity of the brightness value of the displayed image and display an image with high visibility.
Further, referring to the front frame information 33, when the influence of flare is considered to be small, the flicker of the displayed image is suppressed by bringing the brightness value closer to the input image 11, and an image with higher visibility is displayed. can.

Embodiment 3.
The image display device 1 according to the present embodiment includes an image pickup unit 2, an image processing unit 3, and an image display unit 4 in the same manner as the schematic configuration of the first embodiment shown in FIG. As shown in 8, the front frame information holding unit 26 is provided with the composite image 13 of the front frame output from the composite image generation unit 24.

The front frame information holding unit 26 stores the composite image 13 output from the composite image generation unit 24 in a plurality of front frames. For information storage, for example, a memory area is secured so that information from time T1 to time T7 can be sequentially held, and a ring buffer or the like is used.

The light source position detection unit 21 uses the captured input image 110 of the current frame and the composite image 13 of the plurality of front frames held by the front frame information holding unit 26, and presents from the transition of the light source position 41 in the composite image 13 of the front frame. The light source position 41 of the frame is estimated. For example, in the case of a camera that captures 60 frames per second, the transition of the light source can be grasped because the same light source is repeatedly imaged in the composite image 13 of a plurality of previous frames. Then, by comparing the brightness values of the composite images 13 of the plurality of front frames, it is possible to find and remove high-luminance pixels that are not affected by flare such as noise and road surface reflected light 53.
For example, a composite image 13 of a plurality of front frames for a preset time is taken out from the front frame information holding unit 26, and the transition of the position of the center of gravity of the high-luminance region calculated for each frame is observed. For example, in the case of the headlight 52 of the rear vehicle 51, the position of the center of gravity moves according to the traveling direction of the road. Can be estimated from.
If the movement of the position of the center of gravity in the high-luminance region cannot be estimated in a plurality of front frames, it is determined that the region is not affected by flare caused by noise of the image pickup input image 110, road surface reflected light, or the like, and the light source position 41. Can be prevented from being detected.

From the composite image 13 of the plurality of front frames, the flare in the input image 11 of the current frame is changed from the fluctuation of the size of the flare region 42 due to the movement of the headlight 52 of the rear vehicle 51 depending on the directivity characteristic of the sensitivity of the camera. The region 42 may be determined.
When the fluctuation trend cannot be estimated from the composite image 13 of the plurality of front frames, it is determined that the region is not affected by the flare caused by the noise of the image pickup input image 110, the road surface reflected light 53, and the like, and the region is determined to be the flare region 42. You can avoid it.

Even in such an image display device 1, the light source position 41 is detected from the brightness value of the captured input image 110, and the input image 11 and the corrected image 12 are formed in at least a part of the flare region 42 determined from the center of the light source position 41. The composite image 13 generated by increasing the proportion of the corrected image 12 as it is closer to the center of the light source position 41 and increasing the proportion of the input image 11 as it is farther away is displayed, and the input image is displayed in the area other than the flare area 42. Since 11 is displayed, it is possible to suppress the discontinuity of the brightness value of the displayed image and display an image with high visibility.
Then, by using the front frame information 34 with reference to the composite image 13 of the front frame, the positions of the light source position 41 and the flare region 42 can be estimated, and the processing can be simplified.

Note that, in the above embodiments 1 to 3, as the input image 11, the captured input image 110 may be taken out as it is, or an image whose brightness, hue, etc. have been adjusted may be used.

In addition to the above, it is possible to freely combine each embodiment, modify any component of each embodiment, or omit any component of each embodiment.

Further, although the image display device 1 has been described, the image processing unit 3 constituting the image display device 1, the image display method implemented by the image display device 1, and the image processing method implemented by the image processing unit 3 are also the present invention. Form a part of. Further, a program that causes a computer to perform processing in the image processing unit 3 or the image processing method described above, and a computer-readable recording medium that records the program, such as a non-temporary recording medium, also form a part of the present invention. ..

1 image display device, 2 image pickup unit, 3 image processing unit, 4 image display unit, 11 input image, 12 corrected image, 13 composite image, 21 light source position detection unit, 22 flare area determination unit, 23 corrected image generation unit, 24 Composite image generation unit, 25, 26 front frame information holding unit, 31 light source position information, 32 flare area information, 33, 34 front frame information, 41 light source position, 42 flare area, 51 rear vehicle, 52 headlight, 53 road surface reflection Hikari, 110 image capture input image

Claims

An image pickup unit that captures the outside of the vehicle and
A light source position detection unit that detects as a light source position a pixel region in which pixels of a preset first luminance value or higher in an image pickup input image captured by the image pickup unit are adjacently gathered.
From the pixel area around the center of the light source position, a flare area determination unit that determines a pixel area having a second luminance value or more set in advance as a flare area.
A corrected image generation unit that generates a corrected image by reducing the brightness value of pixels of at least the input image in the flare region of the captured input image that are equal to or higher than a preset third brightness value.
At least in a part of the flare region, the corrected image and the input image are combined by increasing the proportion of the corrected image as it is closer to the center of the light source position and increasing the proportion of the input image as it is farther from the center. The composite image generator to be generated and
The composite image is in the pixel region where the composite image is generated, the corrected image or the input image is in the flare region, and the input image is outside the flare region in the pixel region where the composite image is not generated. Image display unit that displays
Image display device equipped with.
The image display device according to claim 1, wherein the composite image generation unit generates the composite image in the flare region or in a region including the flare region end and its periphery.
The image display device according to claim 1 or 2, wherein the flare area determination unit sets the second luminance value larger as the distance from the center of the light source position is shorter, and determines the flare area.
The image display device according to any one of claims 1 to 3, further comprising a front frame information holding unit for holding front frame information in a front frame imaged in a time zone before the present time.
The claim that the front frame information is at least one of the light source position information obtained by the light source position detection unit, the flare region information obtained by the flare region determination unit, and the composite image obtained by the composite image generation unit. The image display device according to 4.
The front frame information is the light source position information, and the ratio of the input image and the ratio of the corrected image in the composite image generation unit are the number of the front frames in which the light source position is not detected by the light source position information. The image display device according to claim 5, wherein the larger the number, the larger the ratio of the input image and the smaller the ratio of the corrected image.
The front frame information is the flare region information, and the ratio of the input image and the ratio of the corrected image in the composite image generation unit are large in the number of the front frames that are not determined to be the flare region by the flare region information. The image display device according to claim 5, wherein the ratio of the input image is increased and the ratio of the corrected image is decreased.
The front frame information is the composite image, and the light source position detecting unit obtains the light source position of the current frame from the transition of the light source position in the composite image of the front frame held by the front frame information holding unit. The image display device according to claim 5.
The front frame information is the composite image, and the flare region determination unit obtains the flare region of the current frame from the transition of the flare region in the composite image of the front frame held by the front frame information holding unit. The image display device according to claim 5.
A step of detecting as a light source position a pixel region in which pixels having a preset first luminance value or more in an image captured by imaging the outside of a vehicle are adjacent to each other.
A step of determining a pixel region in which the luminance value of the pixels around the center of the light source position is equal to or higher than a preset second luminance value as a flare region.
A step of reducing the luminance value of pixels of at least the input image in the flare region of the captured input image to be equal to or higher than a preset third luminance value to generate a corrected image.
At least in a part of the flare region, the corrected image and the input image are combined by increasing the proportion of the corrected image as it is closer to the center of the light source position and increasing the proportion of the input image as it is farther from the center. Steps to generate and
The composite image is in the pixel region where the composite image is generated, the corrected image or the input image is in the flare region, and the input image is outside the flare region in the pixel region where the composite image is not generated. And the steps to generate a display image to display
An image display method characterized by comprising.