JP6327388B2 - Captured image display device, captured image display method, and captured image display program - Google Patents

Description

    The present invention relates to a captured image display device, a captured image display method, and a captured image display program for displaying an image captured by a camera or the like.

  In a moving body such as an automobile, there is a technique for displaying a peripheral image captured by a camera or the like on a monitor in the vehicle and assisting the driver to recognize it visually.

  Patent Document 1 discloses a technique for synthesizing a video image of an infrared camera on a portion of a visible light camera image having a predetermined luminance or less than a predetermined luminance for the purpose of assisting the driver to recognize the front. ing.

Japanese Patent Laid-Open No. 11-308609

  When an infrared camera image is combined with a visible light camera image in accordance with a predetermined luminance, a dark range where the driver cannot recognize the front, dazzling due to a headlight of an oncoming vehicle, and the like are targeted. The situation in which it is difficult for the driver to visually recognize the outside of the vehicle and to recognize the outside of the vehicle by the image of the visible light camera is not limited to the above, and may be difficult even in the daytime such as fog, rain, and snow.

  When it is difficult to recognize the outside of the vehicle due to fog, rain, snow, or the like, the judgment based on the brightness cannot be distinguished from the case where the outside of the vehicle is clearly recognized.

  The present invention has been made to solve such a problem, and is capable of appropriately assisting visibility at the time of poor visibility regardless of day or night, a captured image display device, a captured image display method, and a captured image display program. The purpose is to provide.

  In order to achieve the above object, a captured image display device (10) according to the present invention includes a first imaging unit (51, 102) that acquires an image in the visible light range, and a first image acquisition unit that acquires an image in the infrared light range. The contrast information for each section is acquired from the images captured by the second photographing section (52, 102) and the first photographing section (51, 102), and the contrast information for each section having a large white image ratio in the image is acquired. A contrast information acquisition unit (105) for determining that the ratio is less than a predetermined threshold, and an image captured by the second imaging unit (52, 102) for the image of the section in which the contrast ratio is less than the predetermined threshold Are combined, and a display unit (108, 60) for displaying an image synthesized by the combining unit (106).

  In the captured image display method according to the present invention, contrast information for each section having a large white image ratio in the image is obtained from the image captured by the first capturing unit (51, 102) that acquires an image in the visible light range. A step of acquiring, a step of determining that a contrast ratio for each section is less than a predetermined threshold, and a second image capturing for acquiring an image in an infrared light range for an image of a section having a contrast ratio of less than a predetermined threshold. A step of combining the images taken by the units (52, 102) and a step of displaying the combined images.

  In addition, the program according to the present invention is a program (100) included in the captured image display device (10), wherein the image is captured from an image captured by the first capturing unit (51, 102) that acquires an image in the visible light range. A step of acquiring contrast information for each section having a large white image ratio in the step, a step of determining that the contrast ratio of each section is less than a predetermined threshold, and an image of a section having a contrast ratio of less than the predetermined threshold. A step of synthesizing images captured by the second imaging unit (52, 102) that acquires an image of the outside light range and a step of displaying the synthesized image are executed.

    According to the present invention, it is possible to appropriately perform visibility assistance when visibility is poor regardless of day or night.

1 is a configuration block diagram of a captured image display device according to the present invention. 5 is a flowchart showing an operation example of a captured image display method according to the present invention. It is a conceptual diagram which shows the example of a division | segmentation of a picked-up image.

  Hereinafter, embodiments of the present invention will be described. FIG. 1 is an example of a configuration block diagram of a captured image display apparatus 10 according to an embodiment of the present invention. The captured image display device 10 can be applied to various mobile objects, and can be used for various mobile objects such as automobiles, railways, aircraft, ships, and bicycles. In the present embodiment, an example in which the vehicle is mounted on an automobile will be described.

  The captured image display device 10 includes a visible light camera 51, an infrared camera 52, a control unit 100, and a display panel 60, as shown in FIG. The captured image display device 10 may be built in as a function of a car in advance, or may be configured by a retrofit device or a tablet terminal that is brought into the car and used.

  The visible light camera 51 captures an image with a visible light wavelength outside the vehicle, and transmits the captured image data to the imaging control unit 102. The visible light wavelength is generally in the range of 380 nm to 750 nm, and is a range that can be recognized by the human eye. It is appropriate that the visible light camera 51 is disposed at a position where an image outside the vehicle can be photographed, and is disposed at a position where the forward direction of the automobile can be photographed. The arrangement direction of the visible light camera 51 is not limited to the front in the traveling direction of the automobile, and a necessary direction may be photographed, and the photographing direction may be movable.

  The infrared camera 52 captures a far-infrared light wavelength image outside the vehicle, and transmits the captured image data to the imaging control unit 102. The wavelength captured by the infrared camera 52 is not limited to the far-infrared light wavelength, and may be a wavelength in another infrared region as necessary. May be included. The other infrared region is mid-infrared light or near-infrared light. Far-infrared light wavelength is generally in the range of 4 μm to 1000 μm and cannot be recognized by the human eye, but is used for human detection at night because it captures far-infrared rays emitted from objects that serve as heat sources. Is done. It is appropriate that the infrared camera 52 is disposed at a position where an image outside the vehicle can be photographed and a range overlapping the visible light camera 51 can be photographed. Similarly to the visible light camera 51, the arrangement direction of the infrared camera 52 is not limited to the front in the traveling direction of the automobile, and a necessary direction may be photographed, and the photographing direction may be movable.

  The control unit 100 includes a CPU (Central Processing Unit) that performs various data processing, a DSP (Digital Signal Processor), a memory, and the like. The control unit 100 may use a dedicated control device mounted on the vehicle, a control device mounted on a navigation device, or a control device such as an external terminal.

  The control unit 100 includes an imaging control unit 102, an image processing unit 104, and a display control unit 108 based on the function. Each function may be realized by a single control device or a single processor, or may be realized by a plurality of distributed control devices or a plurality of processors.

  The imaging control unit 102 controls the imaging operation of the visible light camera 51 and the infrared camera 52. Specifically, the start and stop of shooting, control of the shooting angle of view and aperture, acquisition of shot image data, and the like.

  The visible light camera 51 and the imaging control unit 102 constitute a first imaging unit. The infrared camera 52 and the imaging control unit 102 constitute a second imaging unit.

  The image processing unit 104 processes the image data of the visible light camera 51 and the infrared camera 52 acquired by the imaging control unit 102 and sends them to the display control unit 108. The image processing unit 104 includes a contrast information acquisition unit 105 and a synthesis unit 106 as components by function.

  The contrast information acquisition unit 105 acquires contrast information for each predetermined range of image data captured by the visible light camera 51. Further, the contrast information acquisition unit 105 compares the acquired contrast information with a predetermined threshold value, and determines whether or not it is less than the threshold value. The predetermined threshold is stored in a memory constituting the control unit 100 or another storage medium (not shown).

  The combining unit 106 performs a process of combining the image data captured by the visible light camera 51 and the image data captured by the infrared camera 52. Specifically, the same range of image data captured by the infrared camera 52 is synthesized with the range based on the image data of the visible light camera 51 that the contrast information processing unit 105 has determined to be a contrast equal to or lower than a predetermined threshold. It is sent to the display control unit 108.

  The image processing unit 104 determines whether the image data captured by the visible light camera 51 needs to be combined with the image data of the infrared camera 52 based on the determination of the contrast information acquisition unit 105 or the required display form. The video of the camera 52 is sent to the display control unit 108 without being synthesized.

  The display control unit 108 performs processing for displaying the image data sent from the image processing unit 104 on the display panel 60. The display control unit 108 also performs brightness and color tone adjustment control on the image data sent from the image processing unit 104.

  As the display panel 60, various devices such as a liquid crystal panel, an organic EL (Electro Luminescence) panel, and a head-up display device are used to present an image to a driver who is a user. The display control unit 108 and the display panel 60 constitute a display unit.

  Next, a captured image display method which is an operation example of the captured image display device 10 according to the present invention will be described with reference to FIG.

  As a premise of the operation in FIG. 2, during the operation of the captured image display device 10, the imaging control unit 102 causes the visible light camera 51 and the infrared camera 52 to continuously capture images outside the vehicle including the same range. Is in a state of performing. As an example, the image captured by the visible light camera 51 and the infrared camera 52 is a moving image in which still images of 30 frames per second are continuous.

  In the state described above, the contrast information acquisition unit 105 divides an arbitrary frame image captured by the visible light camera 51 into n (step S100), and acquires contrast information for each of the n divided sections (step S101). Here, n is an integer set in advance.

  FIG. 3 shows an example of division in the process of step S100. FIG. 3 shows an example in which an arbitrary one-frame image 200 of an image captured by the visible light camera 51 is divided into 40. Although the number of divisions and the division shape are not limited, it is desirable that the division is fine according to the processing capability of the control unit 100, such as 480 division. In FIG. 3, the one-frame image 200 is divided into 40, and numbers 200_1 and 200_2 are assigned to the divided sections, but some numbering is omitted.

  In step 101, the contrast information acquisition unit 105 acquires contrast information for each of the sections 200_1 to 200_40 indicating the sections obtained by dividing the one-frame image 200.

  The contrast information acquired in step S101 acquires the maximum luminance value Lmax and the minimum luminance value Lmin for each section obtained by dividing the one-frame image 200. The maximum luminance value Lmax and the minimum luminance value Lmin are calculated from Y signals obtained by integrating the luminance values of R (red), G (green), and blue (B) for each pixel included in the divided sections.

  In step S101, a contrast ratio is calculated based on the acquired maximum value Lmax and minimum value Lmin of luminance. The contrast ratio is generally expressed as Lmax / Lmin.

  Next, the contrast information acquisition unit 105 determines whether or not there is a partition in which the contrast ratio for each partition is less than a preset threshold (step S102). The threshold value set here may be a value that allows a state in which there is almost no change in the luminance difference regardless of the luminance value in the section obtained by dividing the one-frame image 200 to be less than the threshold value. When the contrast ratio is expressed by Lmax / Lmin, for example, threshold = 1.1.

  Further, when the contour (edge) of the object is included in the section where the contrast information acquisition unit 105 calculates the contrast ratio, the high-frequency component of the Y signal is included. In addition to the determination of the contrast ratio corresponding to the threshold value, the processing in step S102 may also include the level of the high-frequency component level of the Y signal as a determination criterion. Even in this case, a threshold value that defines the level of the high-frequency component of the Y signal may be set and compared.

  The processing in step S102 by the contrast information acquisition unit 105 determines for each section obtained by dividing the one-frame image 200. However, if the contrast ratio of the adjacent section is less than the threshold value, the same determination is performed again in a range including the adjacent section. May be performed. For example, in FIG. 3, when the contrast ratio in each of 200_1, 200_2, 200_9, and 200_10 indicating the sections obtained by dividing the one-frame image 200 is less than the threshold value, the contrast ratio is again set to the threshold value in the range obtained by adding these sections. You may judge whether it is less than.

  In step S102, when it is determined that there is no section having a contrast ratio less than the threshold (step S102: No), the image processing unit 104 captures an image captured by the infrared camera 52 with respect to an image captured by the visible light camera 51. The data is sent to the display control unit 108 without being synthesized. For this reason, the display control unit 108 causes the display panel 60 to display an image captured by the visible light camera 51 (step S103).

  In step S102, when it is determined that there is a section having a contrast ratio less than the threshold (step S102: Yes), the image processing unit 104 acquires an image corresponding to the same section from the image captured by the infrared camera 52. (Step S104). The image of the infrared camera 52 acquired in step S104 is an image taken at the same time as the image of the visible light camera 51 that has been determined in step S102.

  In step S104, after acquiring an image corresponding to the same section from the image captured by the infrared camera 52, the combining unit 106 performs step S104 on the image of the section whose contrast ratio is less than the threshold in step S102. The images corresponding to the same section acquired in step S105 are synthesized (step S105). The synthesizing process by the synthesizing unit 106 in step S105 uses an existing method such as alpha blending.

  Next, the image processing unit 104 sends the image synthesized by the synthesizing unit 106 to the display control unit 108, and the display control unit 108 creates an image obtained by synthesizing the image of the infrared camera 52 with the image of the visible light camera 51. It is displayed on the display panel 60 (step 106).

  By such processing, an image captured by the infrared camera 52 is combined with an image captured by the visible light camera 51 and displayed in response to various states in which recognition outside the vehicle is difficult regardless of luminance. For this reason, the driver who is a user can confirm the situation outside a vehicle appropriately.

  The present invention is effective in the following situations as a specific example.

Situation example 1: Recognition of a non-illuminated range at night When the periphery of a vehicle is photographed using the visible light camera 51 and the infrared camera 52, a range in which light is not irradiated by the headlight of the vehicle or an external streetlight This makes it difficult to recognize the image of the visible light camera 51 and the driver's visual observation. In the range captured by the visible light camera 51, the contrast ratio is less than the threshold in a range where light is not irradiated by a headlight of an automobile, an external streetlight, or the like, so the image of the infrared camera 52 is synthesized and displayed. .

  The image synthesized and displayed in the situation example 1 can appropriately recognize a person or an object existing in a range where light is not irradiated by a headlight of an automobile or an external streetlight. In such a situation, the contrast information acquisition unit 105 may limit the section to be processed in step S102 to a section having a high black image ratio.

Situation example 2: Recognition of dazzling range at night When the periphery of a vehicle is photographed using the visible light camera 51 and the infrared camera 52, the range in which the headlight of the oncoming vehicle is dazzled is the visible light camera 51. This makes it difficult to recognize the image and the driver's visual observation. In the range captured by the visible light camera 51, the contrast ratio is less than the threshold in the range where the headlight of the oncoming vehicle is hit and dazzled, so the image of the infrared camera 52 is synthesized and displayed.

  The image synthesized and displayed in the situation example 2 can appropriately recognize a person or an object existing in a range where the headlight of the oncoming vehicle is hit and dazzled. In such a situation, the contrast information acquisition unit 105 may limit the section to be processed in step S102 to a section having a large white image ratio. In addition, the contrast information acquisition unit 105 may limit the section to be processed in step S102 to a section with high luminance.

Situation example 3: Recognition of poor visibility range due to bad weather in the daytime When the periphery of a car is photographed using the visible light camera 51 and the infrared camera 52, it may be partially or wholly depending on the weather conditions such as fog, rain, and snow. In this case, it is difficult to recognize the image of the visible light camera 51 and the driver's visual observation. Since the contrast ratio is less than the threshold in a range that is difficult to recognize due to weather conditions such as fog, rain, and snow, the image of the infrared camera 52 is synthesized and displayed.

  The image synthesized and displayed in the situation example 3 can appropriately recognize a person or an object existing in a range difficult to recognize depending on weather conditions such as fog, rain, and snow. In such a situation, the contrast information acquisition unit 105 may limit the section to be processed in step S102 to a section having a large white image ratio.

  The embodiment of the present invention can be variously modified without departing from the gist thereof. For example, in the present embodiment, the threshold value serving as a reference for determining whether the contrast ratio is high or low is a fixed threshold value, but the threshold value may be varied according to the situation. Alternatively, a plurality of threshold values may be set, and the alpha value of the synthesis process by alpha blending of the synthesis process by the synthesis unit 106 may be changed in accordance with the threshold setting range corresponding to the contrast value.

DESCRIPTION OF SYMBOLS 10 Shooting image display apparatus 51 Visible light camera 52 Infrared camera 60 Display panel 100 Control part 102 Shooting control part 104 Image processing part 105 Contrast information acquisition part 106 Composition part 108 Display control part 200 1 frame image

Claims (5)

A first imaging unit for acquiring an image in a visible light range;
A second imaging unit for acquiring an image in the infrared light range;
A contrast information acquisition unit that acquires, from an image captured by the first imaging unit, contrast information for each section having a large white image ratio in the image and determines that the contrast ratio for each section is less than a predetermined threshold. ,
A synthesizing unit that synthesizes an image captured by the second imaging unit with an image of the section having a contrast ratio less than a predetermined threshold;
A display unit for displaying an image synthesized by the synthesis unit;
A captured image display device. The synthesizing unit synthesizes an image corresponding to the same section imaged by the second imaging unit with an image of a section where the contrast ratio in the image is less than a predetermined threshold;
The captured image display device according to claim 1. The contrast information acquisition unit determines that the contrast value of a section including a plurality of adjacent sections is less than a predetermined threshold when there are a plurality of adjacent sections whose contrast ratio is less than a predetermined threshold.
The captured image display device according to claim 1 or 2. Acquiring contrast information for each section having a large white image ratio in the image from an image captured by the first imaging unit that acquires an image in the visible light range;
Determining that the contrast ratio for each section is less than a predetermined threshold;
Synthesizing an image captured by a second imaging unit that acquires an image in an infrared light range with an image of a section having a contrast ratio less than a predetermined threshold;
Displaying the synthesized image;
A captured image display method comprising: In the computer provided in the captured image display device,
Acquiring contrast information for each section having a large white image ratio in the image from an image captured by the first imaging unit that acquires an image in the visible light range;
Determining that the contrast ratio for each section is less than a predetermined threshold;
Synthesizing an image captured by a second imaging unit that acquires an image in an infrared light range with an image of a section having a contrast ratio less than a predetermined threshold;
Displaying the synthesized image;
A program that executes