WO2024176953A1

WO2024176953A1 - Image processing device, image processing method, and program

Info

Publication number: WO2024176953A1
Application number: PCT/JP2024/005412
Authority: WO
Inventors: 雄介岡田; 哲郎奥山; 良直河合
Original assignee: ヌヴォトンテクノロジージャパン株式会社
Priority date: 2023-02-22
Filing date: 2024-02-16
Publication date: 2024-08-29

Abstract

An image processing device (100) comprising: an acquisition unit (exposure time setting unit (10)) which acquires two types of image obtained by two types of imaging means differ in wavelength from each other and showing the same area; and a determination unit (exposure time setting unit (10)) which determines, on the basis of one of the two types of image, a photometric area which is in the other one of the two types of image and in which photometry is performed.

Description

Image processing device, image processing method, and program

This disclosure relates to image processing devices, etc.

Patent document 1 discloses a technique for setting exposure conditions for a shooting area.

International Publication No. 2017/170719

However, when an image captured using the technology disclosed in Patent Document 1 is applied to object detection, the target object may not be detected correctly. Specifically, if the captured image contains bright objects that are not the target object, such as direct sunlight, the sky, or a distant white object (note that here, direct sunlight and the sky are also referred to as objects), automatic exposure control is performed so that the entire image becomes dark due to the influence of that brightness, and the area of the target object becomes dark, so that the target object may not be detected correctly.

The present disclosure provides an image processing device and the like that can perform automatic exposure control suitable for detecting an object to be detected.

The image processing device according to the present disclosure includes an acquisition unit that acquires two types of images showing the same area obtained by two types of imaging means having different wavelengths, and a determination unit that determines a photometric area in which photometry is performed in one of the two types of images based on the other of the two types of images.

The image processing method disclosed herein is an image processing method executed by an image processing device, and includes an acquisition step of acquiring two types of images showing the same area obtained by two types of imaging means having different wavelengths, and a determination step of determining, based on one of the two types of images, a photometric area in which photometry is performed in the other of the two types of images.

The program disclosed herein is a program for causing a computer to execute the image processing method described above.

These comprehensive or specific aspects may be realized as a system, method, integrated circuit, computer program, or computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, method, integrated circuit, computer program, and recording medium.

An image processing device according to one aspect of the present disclosure can perform automatic exposure control suitable for detecting an object to be detected.

1 is a block diagram showing an example of an image processing device according to an embodiment; 11A and 11B are diagrams for explaining a method of calculating the intensity of a component of reflected light. FIG. 4 is a block diagram showing an example of an exposure time setting unit according to the embodiment. 5 is a flowchart showing an example of an operation of a photometry unit according to the embodiment. FIG. 4 is a diagram showing an example of an acquired visible light image. FIG. 2 is a diagram showing an example of an acquired infrared image. FIG. 13 is a diagram showing an example of an area where photometry is not performed. 10 is a flowchart illustrating an example of an operation of an exposure time calculation unit according to the embodiment. FIG. 1 is a diagram showing an example of a visible light image obtained by performing automatic exposure control using a conventional method. 5A to 5C are diagrams illustrating an example of a visible light image obtained by performing automatic exposure control using the image processing device according to the embodiment. 10 is a flowchart illustrating another example of the operation of the photometry unit according to the embodiment. FIG. 13 is a diagram for explaining blocks on which photometry is not performed. FIG. 2 is a block diagram showing an example of an image synthesis unit according to the embodiment. 10 is a flowchart illustrating an example of an operation of an offset correction unit according to the embodiment. 11 is a flowchart illustrating an example of an operation of an output synthesis unit according to the embodiment. FIG. 4 is a block diagram showing an example of a gain correction unit according to the embodiment. 13 is a flowchart illustrating an example of an image processing method according to another embodiment.

The following describes the embodiment in detail with reference to the drawings.

The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure.

(Embodiment)
An image processing device according to an embodiment will be described below.

FIG. 1 is a block diagram showing an example of an image processing device 100 according to an embodiment. In addition to the image processing device 100, FIG. 1 also shows an imaging unit 200 that captures images to be processed by the image processing device 100. Note that the image processing device 100 may also include the imaging unit 200.

The imaging unit 200 is, for example, a camera that has two types of imaging means with different wavelengths and can capture two types of images using the two types of imaging means. For example, the imaging unit 200 is a BW-TOF camera. A BW-TOF camera can simultaneously capture two types of images: a visible light image (e.g., a BW (black and white) image) obtained by receiving visible light in the entire wavelength range, and an infrared image obtained by receiving reflected light of infrared light in the infrared range that has been irradiated using a ToF (Time of Flight) type sensor. In other words, the imaging unit 200 can simultaneously capture two types of images, a visible light image and an infrared image, showing the same area. Note that the imaging unit 200 does not have to be a camera that can capture visible light images and infrared images with one camera, such as a BW-TOF camera, as long as it can capture two types of images showing the same area. For example, the imaging unit 200 may have two separate cameras, specifically, a camera capable of capturing visible light images and a camera capable of capturing infrared images.

Note that the same area is captured in the two types of images, but that area may be the entire image area or a portion of the entire image area. In other words, the entire image area of the two types of images may match, or a portion of the entire image area of the two types of images may match. For example, if the imaging unit 200 is a camera such as a BW-ToF camera that can capture two types of images simultaneously, the entire image area of the two types of images may match. On the other hand, if the imaging unit 200 has two separate cameras, a portion of the entire image area of the two types of images may match.

The imaging unit 200 can measure (range) the distance to an object shown in an infrared image based on the intensity of the reflected light component of the emitted infrared light. In other words, the infrared image contains distance information (depth information) of the object shown in the infrared image. Here, the method for calculating the intensity of the reflected light component is explained using Figure 2.

Figure 2 is a diagram to explain how to calculate the intensity of the reflected light components.

For example, when an object is present within the imaging range of the imaging unit 200, reflected light from the object is received in response to the irradiated light shown in FIG. 2. The intensity of the reflected light component can be calculated according to the amount of received light signal in at least three sections after the irradiated light is applied. Specifically, the at least three sections are sections A0 and A1 in which the signal amount is large due to the reflected light shown in FIG. 2, and section A2 in which the signal amount is small. Sections A0 and A1 each contain a portion of the IR reflected light component and a background component, which is a component of light other than the reflected light, and section A2 contains only the background component. Therefore, the intensity of the reflected light component (IR reflected light component) is the total signal amount of A0 and A1 minus the background component. If A0, A1, and A2 are each the signal amount, the reflected light component can be calculated by (A0-A2)+(A1-A2).

The image processing device 100 is a device that performs image processing on two types of images (a visible light image and an infrared image) that show the same area and are obtained by two types of imaging means with different wavelengths in the imaging unit 200. For example, the image processing device 100 performs image processing to output an image suitable for detecting an object (detection target) that appears in the captured image.

The image processing device 100 includes an exposure time setting unit 10, an image synthesis unit 20, and a gain correction unit 30. The image processing device 100 is a computer including a processor (microprocessor) and a memory. The memory is a ROM (Read Only Memory) and a RAM (Random Access Memory), etc., and can store programs executed by the processor. The exposure time setting unit 10, the image synthesis unit 20, and the gain correction unit 30 are realized by a processor that executes programs stored in the memory, etc.

The exposure time setting unit 10 measures the luminance of the scene from the captured visible light image, sets the exposure time so that the brightness of the area of the detected object captured in the image becomes the desired brightness, and outputs the exposure time to the imaging unit 200. Details of the exposure time setting unit 10 will be described later.

The image synthesis unit 20 synthesizes an image from multiple visible light images captured with different exposure times from the imaging unit 200, so that the detected object does not have blown-out highlights or crushed shadows. Details of the image synthesis unit 20 will be described later.

The gain correction unit 30 measures the luminance of the scene in the current frame from the synthesized image, and performs digital gain correction on the image so that the brightness of the area of the detection target becomes the desired brightness. Specifically, there is a delay time until the exposure time set by the exposure time setting unit 10 is reflected in the output image, and the gain correction unit 30 performs digital gain correction to compensate for the output level fluctuation that accompanies this delay time. The gain correction unit 30 will be described in detail later.

First, we will explain the details of the exposure time setting unit 10.

FIG. 3 is a block diagram showing an example of an exposure time setting unit 10 according to an embodiment.

The exposure time setting unit 10 includes a photometry unit 11 and an exposure time calculation unit 12.

The photometry unit 11 is an example of an acquisition unit that acquires two types of images showing the same area, obtained by two types of imaging means with different wavelengths. The photometry unit 11 is also an example of a determination unit that determines the photometry area in which photometry is performed in one of the two types of images, based on the other image of the two types of images.

Specifically, the photometry unit 11 acquires two types of images, a visible light image and an infrared image, and determines a photometry area in the visible light image based on the infrared image. The photometry area is a group of pixels to be measured, and the photometry unit 11 measures the brightness of the determined photometry area. Here, an example of a method for determining the photometry area will be described with reference to FIG. 4.

FIG. 4 is a flowchart showing an example of the operation of the photometry unit 11 according to the embodiment. It is assumed that the photometry unit 11 acquires, for example, a visible light image as shown in FIG. 5 and an infrared image as shown in FIG. 6.

Figure 5 shows an example of an acquired visible light image.

Figure 6 shows an example of an acquired infrared image.

As shown in Figures 5 and 6, the same area is captured in the visible light image and the infrared image, and the photometry unit 11 recognizes pixels in the infrared image that capture the same positions as each pixel in the visible light image. Note that pixels in the visible light image and pixels in the infrared image that capture the same positions are referred to as corresponding pixels.

In addition, when the imaging unit 200 has a separately provided camera capable of capturing visible light images and a camera capable of capturing infrared images, if some areas of the entire image area of the visible light image and the infrared image match but the entire image area does not match, a process of matching between the two images can be performed by known means to determine corresponding pixels that capture the same position in the visible light image and the infrared image.

The photometry unit 11 determines the photometry area by performing the processes from step S11 to step S14 shown in FIG. 4 for each pixel in the visible light image. Note that the area where photometry is not performed is an area excluding the photometry area where photometry is performed, so when the photometry area is determined, the area where photometry is not performed is also determined. In other words, determining the photometry area also means determining the area where photometry is not performed.

First, the photometry unit 11 determines whether or not the BW intensity (pixel value (brightness value) of a pixel in a visible light image) is greater than a threshold value for the BW intensity (step S11). The threshold value for the BW intensity is set to a value that is somewhat smaller than the brightness value corresponding to the sky or a white object, for example, but is not particularly limited and may be set appropriately depending on the space or purpose in which the imaging unit 200 is used.

When the photometry unit 11 determines that the BW intensity of a pixel in the visible light image is greater than the threshold for the BW intensity (Yes in step S11), that is, when the pixel in question is a pixel that depicts a bright object such as the sky, direct sunlight, or a white object, it determines whether the IR intensity (intensity of the reflected light component relative to the emitted infrared light) of the pixel in the infrared image corresponding to the pixel in the visible light image is less than the threshold for the IR intensity (step S12). The threshold for the IR intensity is set to a value that is somewhat smaller than the value expected as the intensity of the reflected light component from an object within a certain range when it is desired to detect an object within the certain range from the imaging unit 200, but is not particularly limited and may be set appropriately depending on the space in which the imaging unit 200 is used, the purpose, etc.

When the photometry unit 11 determines that the IR intensity of a pixel in the infrared image is smaller than the threshold value for IR intensity (Yes in step S12), that is, when the pixel does not include an object (detection target) within a certain range from the imaging unit 200, the photometry unit 11 excludes the pixel in the visible light image corresponding to the pixel from the photometry target (step S13). In this way, the photometry unit 11 does not determine a pixel that includes a bright object such as the sky, direct sunlight, or a distant white object, and does not include the detection target, as a photometry area. In other words, the photometry unit 11 excludes such pixels from the photometry area, regarding them as pixels that include a high-luminance object (direct sunlight, the sky, or a distant white object, etc.) that is unrelated to the detection target. For example, the white building surrounded by the dashed line in Figures 5 and 6 is a bright but distant object, so the pixel that includes this building is excluded from the photometry area.

Furthermore, if the photometry unit 11 determines that the IR intensity of a pixel in the infrared image is equal to or greater than the threshold for IR intensity (No in step S12), that is, if the pixel is a pixel that captures an object (detection target) within a certain range from the imaging unit 200, the photometry unit 11 includes the pixel in the visible light image corresponding to the pixel in the photometry target (step S14). In this way, the photometry unit 11 determines, as the photometry region, an area in the visible light image that corresponds to an area in the infrared image where the reflected light component acquired by the imaging unit 200 (ToF type sensor) is equal to or greater than a predetermined intensity (threshold for IR intensity). In other words, even if the pixel captures a high-luminance object, the photometry unit 11 does not exclude the pixel from the photometry region if the object is the detection target.

On the other hand, if the photometry unit 11 determines that the BW intensity of a pixel in the visible light image is equal to or less than the threshold value for BW intensity (No in step S11), that is, if the pixel does not show a bright object such as the sky, direct sunlight, or a white object, the photometry unit 11 includes the pixel in question in the photometry area (step S14). In this way, for pixels that do not show a bright object, the photometry unit 11 determines the pixel in question as a photometry area without making a judgment on the IR intensity of the pixel in the infrared image that corresponds to the pixel in question.

FIG. 7 shows an example of an area where no photometry is performed.

The white areas in Figure 7 indicate areas where no photometry is performed, and the black areas indicate photometry areas. Note that in this example, the threshold for the BW intensity is set to be greater than the BW intensity of pixels that show the sky, and pixels that show the sky are not excluded from the photometry area.

The photometry unit 11 measures the brightness of the photometry area determined in this manner. For example, the photometry unit 11 measures a representative value (e.g., average, median, or mode) of the pixel values of the pixel group to be metered as the brightness of the photometry area. The photometry unit 11 outputs the brightness of the photometry area to the exposure time calculation unit 12.

Next, the exposure time calculation unit 12 will be explained using Figure 8.

FIG. 8 is a flowchart showing an example of the operation of the exposure time calculation unit 12 according to the embodiment.

The exposure time calculation unit 12 calculates the exposure time from the ratio of the photometry result of the current frame (brightness of the measured photometry area) to the target pixel value (step S31). Specifically, the exposure time calculation unit 12 calculates the exposure time by (target pixel value/photometry result of the current frame) x (current exposure time). That is, if the photometry result of the current frame is greater than the target pixel value (i.e., if it is bright), the exposure time calculation unit 12 calculates an exposure time shorter than the current exposure time. Also, if the photometry result of the current frame is smaller than the target pixel value (i.e., if it is dark), the exposure time calculation unit 12 calculates an exposure time longer than the current exposure time. For example, the target pixel value is not particularly limited, but may be an intermediate value between the blown-out highlight value and the crushed shadow value.

Then, the exposure time calculation unit 12 outputs the calculated exposure time to the imaging unit 200. As a result, the imaging unit 200 performs exposure using the acquired exposure time and captures a visible light image. If an exposure time shorter than the current exposure time is calculated, a visible light image that is darker than the current frame can be captured, and if an exposure time longer than the current exposure time is calculated, a visible light image that is brighter than the current frame can be captured.

FIG. 9 shows an example of a visible light image obtained by performing automatic exposure control using a conventional method.

FIG. 10 shows an example of a visible light image obtained by performing automatic exposure control using the image processing device 100 according to the embodiment.

In conventional methods, as shown in FIG. 9, automatic exposure control is performed so that the entire image becomes dark due to the influence of the brightness of the white building in the background, and the area in which the detection object (e.g., a vehicle) exists, surrounded by a dashed line, becomes dark, which may result in the detection object not being detected correctly. On the other hand, according to the image processing device 100 of the embodiment, as shown in FIG. 10, the area in which the white building in the background exists is excluded from the photometry area, so automatic exposure control is performed without being influenced by the brightness of the white building in the background, and the area in which the detection object exists, surrounded by a dashed line, does not become dark, making it easier to detect the detection object.

In this way, the intensity of the reflected light component from nearby objects is high and the intensity of the reflected light component from distant objects is low, making it possible to obtain an infrared image in which nearby objects with high reflected light component intensity are easily captured and distant objects with low reflected light component intensity are not easily captured. Areas with high reflected light component intensity are areas in which the detected object may be captured, and can therefore be determined as photometry areas.

Although the method for determining the photometric area using the intensity of the reflected light components has been explained, there is also a method for determining the photometric area without using the intensity of the reflected light components. Below, a method for determining the photometric area without using the intensity of the reflected light components will be explained using Figure 11.

FIG. 11 is a flowchart showing another example of the operation of the photometry unit 11 according to the embodiment.

The photometry unit 11 determines the photometry area by performing the processes from step S21 to step S24 shown in FIG. 11 for each pixel in the visible light image. Note that the area where photometry is not performed is an area excluding the photometry area where photometry is performed, so when the photometry area is determined, the area where photometry is not performed is also determined. In other words, determining the photometry area also means determining the area where photometry is not performed.

First, the photometry unit 11 determines whether or not the BW intensity (pixel value (brightness value) of a pixel in a visible light image) is greater than a threshold value for the BW intensity (step S21). The threshold value for the BW intensity is set to a value that is somewhat smaller than the brightness value corresponding to the sky or a white object, for example, but is not particularly limited and may be set appropriately depending on the space or purpose in which the imaging unit 200 is used.

When the photometry unit 11 determines that the BW intensity of a pixel in the visible light image is greater than the threshold for the BW intensity (Yes in step S21), that is, when the pixel in question is a pixel that depicts a bright object such as the sky, direct sunlight, or a white object, it determines whether the IRBG intensity (intensity of light components other than reflected light: intensity of background components) of the pixel in the infrared image that corresponds to the pixel in the visible light image is greater than the threshold for the IRBG intensity (step S22). The threshold for the IRBG intensity is set to a value that is somewhat smaller than the saturation level obtained when receiving direct sunlight, for example, but is not particularly limited and may be set appropriately depending on the space in which the imaging unit 200 is used, the purpose, etc.

When the photometry unit 11 determines that the IRBG intensity of a pixel in the infrared image is greater than the threshold for IRBG intensity (Yes in step S22), that is, when the pixel in question is a pixel that reflects direct sunlight or the like, it excludes the pixel in the visible light image corresponding to that pixel from the photometry target (step S23). In this way, the photometry unit 11 does not determine, as a photometry area, an area in the visible light image corresponding to an area in the infrared image where the components of light other than reflected light (background components) acquired by the imaging unit 200 (ToF type sensor) are greater than a predetermined intensity (threshold for IRBG intensity).

Furthermore, if the photometry unit 11 determines that the IRBG intensity of a pixel in the infrared image is equal to or lower than the threshold value for IRBG intensity (No in step S22), that is, if the pixel in question is a pixel that does not reflect direct sunlight, etc., the photometry unit 11 includes the pixel in the visible light image corresponding to the pixel in question as a photometry target (step S24). In this way, even if the pixel reflects a highly luminous object, the photometry unit 11 does not exclude the pixel from the photometry target if the pixel does not reflect direct sunlight, etc.

On the other hand, if the photometry unit 11 determines that the BW intensity of a pixel in the visible light image is equal to or less than the threshold value for BW intensity (No in step S21), that is, if the pixel does not show a bright object such as the sky, direct sunlight, or a white object, the photometry unit 11 includes the pixel in question as a photometry target (step S24). In this way, for pixels that do not show a bright object, the photometry unit 11 determines the pixel in the photometry area without making a judgment on the IRBG intensity of the pixel in the infrared image that corresponds to the pixel in question.

The subsequent operation of the photometry unit 11 is the same as when the photometry area is determined using the intensity of the reflected light components, so a detailed explanation is omitted.

In this way, the intensity of light components other than reflected light from high-brightness objects such as direct sunlight (background components) is very high, and areas where the intensity of light components other than reflected light is high are areas where high-brightness objects such as direct sunlight may be captured, so they can be prevented from being determined as photometry areas.

When determining the photometric area using the intensity of the reflected light components, an area in the visible light image corresponding to an area in the infrared image where the light components other than the reflected light are greater than a predetermined intensity may not be determined as the photometric area.

The photometry unit 11 may divide the visible light image into a number of blocks, and exclude from the photometry target those areas in the blocks in which the proportion of areas that are not photometry areas is equal to or greater than a predetermined proportion. This will be explained using FIG. 12.

FIG. 12 is a diagram for explaining blocks where photometry is not performed. The white areas in FIG. 12 indicate areas where photometry is not performed, and the black areas indicate photometry areas.

For example, the photometry unit 11 divides the visible light image into multiple blocks (multiple blocks surrounded by white lines) as shown in FIG. 12. Next, the photometry unit 11 calculates the proportion of the area where photometry is not performed (white area in FIG. 12) for each block. Specifically, the photometry unit 11 calculates the proportion of the number of pixels in the area where photometry is not performed to the number of pixels included in one block for each block. Then, the photometry unit 11 excludes all pixels included in blocks where the proportion of the area that is not a photometry area is equal to or greater than a predetermined proportion from the photometry target. In other words, in a block that includes many pixels that are not a photometry target, all pixels are excluded from the photometry target. This makes it possible to exclude the areas surrounding the areas where photometry is not performed from the photometry target. For example, the predetermined proportion is 50%, but is not particularly limited.

Next, we will explain the details of the image synthesis unit 20.

FIG. 13 is a block diagram showing an example of the image synthesis unit 20 according to the embodiment. The image synthesis unit 20 acquires visible light images captured with three different exposure times from the imaging unit 200. For example, the ratio of the three types of exposure times is constant, and if the exposure time setting unit 10 calculates an exposure time that is shorter than the current exposure time, the three types of exposure times become shorter while the ratio of the three types of exposure times remains constant, and if the exposure time setting unit 10 calculates an exposure time that is longer than the current exposure time, the three types of exposure times become longer while the ratio of the three types of exposure times remains constant.

The image synthesis unit 20 is a processing unit for realizing WDR (Wide Dynamic Range) and includes, for example, offset

correction units

21a, 21b, and 21c and an output synthesis unit 22.

The offset correction unit 21a receives a visible light image (called the L image) taken with the longest exposure time of the three types of exposure times, the offset correction unit 21b receives a visible light image (called the M image) taken with the second longest exposure time of the three types of exposure times, and the offset correction unit 21c receives a visible light image (called the S image) taken with the shortest exposure time of the three types of exposure times. Of the L, M, and S images, the L image is the brightest image, the M image is the next brightest image, and the S image is the darkest image. Here, the offset correction unit 21a will be explained using Figure 14.

FIG. 14 is a flowchart showing an example of the operation of the offset correction unit 21a according to the embodiment.

The offset correction unit 21a performs the processes of steps S41 and S42 shown in FIG. 14 for each pixel in the L image.

First, the offset correction unit 21a subtracts the offset level from the input image (L image) (step S41).

Next, the offset correction unit 21a replaces pixel values equal to or greater than the clip level with the clip level (step S42).

Note that offset

correction units

21b and 21c are the same as offset correction unit 21a except for the input image, so a description will be omitted.

Next, the output synthesis unit 22 will be explained using FIG. 15.

FIG. 15 is a flowchart showing an example of the operation of the output synthesis unit 22 according to the embodiment.

The output synthesis unit 22 performs the processes from step S51 to step S59 shown in FIG. 15 for each pixel at the same position in the L image, M image, and S image.

First, the output synthesis unit 22 determines whether or not the pixel value of a pixel at the same position in the L image, M image, and S image is smaller than a first threshold value (referred to as threshold value 1) in the L image (step S51).

If the output synthesis unit 22 determines that the pixel value of the pixel in the L image is smaller than threshold 1 (Yes in step S51), it outputs the pixel value of the pixel in the L image (step S52).

If the output synthesis unit 22 determines that the pixel value of the pixel in the L image is equal to or greater than threshold 1 (No in step S51), it determines whether the pixel value of the pixel in the L image is smaller than a second threshold (referred to as threshold 2) (step S53). Threshold 2 is a value greater than threshold 1.

If the output synthesis unit 22 determines that the pixel value of the pixel in the L image is smaller than threshold 2 (Yes in step S53), it outputs a value obtained by weighting and adding the pixel value of the pixel in the L image and the pixel value of the pixel in the M image (step S54).

If the output synthesis unit 22 determines that the pixel value of the pixel in the L image is equal to or greater than threshold value 2 (No in step S53), it determines whether the pixel value of the pixel in the M image is smaller than a third threshold value (hereinafter referred to as threshold value 3) (step S55).

If the output synthesis unit 22 determines that the pixel value of the pixel in the M image is smaller than threshold 3 (Yes in step S55), it outputs the pixel value of the pixel in the M image (step S56).

If the output synthesis unit 22 determines that the pixel value of the pixel in the M image is equal to or greater than threshold value 3 (No in step S55), it determines whether the pixel value of the pixel in the M image is smaller than a fourth threshold value (hereinafter referred to as threshold value 4) (step S57). Threshold value 4 is a value greater than threshold value 3.

If the output synthesis unit 22 determines that the pixel value of the pixel in the M image is smaller than threshold value 4 (Yes in step S57), it outputs a value obtained by weighting and adding the pixel value of the pixel in the M image and the pixel value of the pixel in the S image (step S58).

If the output synthesis unit 22 determines that the pixel value of the pixel in the M image is equal to or greater than threshold value 4 (No in step S57), it outputs the pixel value of the pixel in the S image (step S59).

In this way, WDR is achieved.

Next, we will explain the details of the gain correction unit 30.

FIG. 16 is a block diagram showing an example of a gain correction unit 30 according to an embodiment.

The gain correction unit 30 includes a photometry unit 31, a gain calculation unit 32, and a gain application unit 33.

The photometry unit 31 acquires the composite image synthesized by the image synthesis unit 20 and measures the brightness (pixel value) of each pixel. For example, the photometry unit 31 measures a representative value (e.g., the average value, median value, or mode value) of the pixel values of each pixel as the brightness of each pixel.

The gain calculation unit 32 calculates the correction gain from the ratio between the photometry result (measured brightness) and the target pixel value. Specifically, the gain calculation unit 32 calculates the correction gain from (target pixel value/photometry result of the current frame). That is, if the photometry result of the current frame is greater than the target pixel value (i.e., brighter), the gain calculation unit 32 calculates a small correction gain. Also, if the photometry result of the current frame is smaller than the target pixel value (i.e., darker), the exposure time calculation unit 12 calculates a large correction gain. For example, the target pixel value is not particularly limited, but may be an intermediate value between the blown-out highlight value and the crushed shadow value.

The gain application unit 33 applies the calculated correction gain to each pixel of the composite image synthesized by the image synthesis unit 20. When a small correction gain is calculated, the composite image can be corrected to a dark image, and when a large correction gain is calculated, the composite image can be corrected to a bright image.

As explained above, depending on the wavelength of the imaging means, it is possible to capture an image in which nearby objects are easily captured and distant objects are not easily captured. Note that the object to be detected is often a nearby object, and bright objects that are not the object to be detected, such as direct sunlight, the sky, and distant white objects, are often distant objects. For this reason, one of the two types of images obtained by two types of imaging means with different wavelengths can be an image in which the object to be detected is more clearly captured than the other image, and in which bright objects that are not the object to be detected are less easily captured than the other image. Therefore, by not determining the area that is bright in the other of the two types of images and is difficult to capture in the one image as the photometric area, and determining the other area as the photometric area, the area in which a bright object that is not the object to be detected is captured is excluded from the photometric area, and automatic exposure control suitable for detecting the object to be detected can be performed for the other image.

For example, the two types of images are a visible light image and an infrared image, and the object to be detected is more clearly visible in the infrared image than in the visible light image, and bright areas that are not the object to be detected are less visible than in the visible light image. Therefore, by not determining areas that are bright in the visible light image and less visible in the infrared image (i.e., the intensity of the reflected light components is reduced) as the photometric area, and instead determining other areas as the photometric area, areas that contain bright objects that are not the object to be detected are excluded from the photometric area, and automatic exposure control suitable for detecting the object to be detected can be performed for the visible light image.

(Other embodiments)
As described above, the embodiment has been described as an example of the technology according to the present disclosure. However, the technology according to the present disclosure is not limited to this, and can be applied to an embodiment in which appropriate changes, substitutions, additions, omissions, etc. are made. For example, the following modified examples are also included in one embodiment of the present disclosure.

For example, the present disclosure can be realized not only as an image processing device 100, but also as an image processing method including steps (processing) performed by components that make up the image processing device 100.

FIG. 17 is a flowchart showing an example of an image processing method according to another embodiment.

The image processing method is an image processing method executed by the image processing device 100, and includes an acquisition step (step S101) of acquiring two types of images showing the same area obtained by two types of imaging means having different wavelengths, as shown in FIG. 17, and a determination step (step S102) of determining, based on one of the two images, a photometric area in which photometry is performed in the other of the two images.

For example, the present disclosure can be realized as a program for causing a computer (processor) to execute the steps included in the image processing method. Furthermore, the present disclosure can be realized as a non-transitory computer-readable recording medium, such as a CD-ROM, on which the program is recorded.

For example, when the present disclosure is realized as a program (software), each step is performed by running the program using hardware resources such as a computer's CPU, memory, and input/output circuits. In other words, each step is performed by the CPU obtaining data from memory or input/output circuits, etc., performing calculations, and outputting the results of the calculations to memory or input/output circuits, etc.

In the above embodiment, each component included in the image processing device 100 may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

Some or all of the functions of the image processing device 100 according to the above embodiment are typically realized as an LSI, which is an integrated circuit. These may be individually integrated into a single chip, or may be integrated into a single chip that includes some or all of the functions. Furthermore, the integrated circuit is not limited to an LSI, and may be realized using a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may also be used.

Furthermore, if an integrated circuit technology that can replace LSIs emerges due to advances in semiconductor technology or other derived technologies, it is natural that each component included in the image processing device 100 may be integrated into an integrated circuit using that technology.

In addition, this disclosure also includes forms obtained by applying various modifications to the embodiments that a person skilled in the art may conceive, and forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the spirit of this disclosure.

(Additional Note)
The above description of the embodiments discloses the following techniques.

(Technology 1) An image processing device comprising: an acquisition unit that acquires two types of images showing the same area obtained by two types of imaging means having different wavelengths; and a determination unit that determines a photometric area in which photometry is performed in one of the two types of images, based on the other of the two types of images.

Depending on the wavelength of the imaging means, it is possible to capture an image in which nearby objects are easily captured and distant objects are not easily captured. Note that the object to be detected is often a nearby object, and bright objects that are not the object to be detected, such as direct sunlight, the sky, or distant white objects, are often distant objects. For this reason, one of the two types of images obtained by two types of imaging means with different wavelengths can be an image in which the object to be detected is more clearly captured than in the other image, and in which bright objects that are not the object to be detected are less easily captured than in the other image. Therefore, by not determining the area that is bright in the other of the two types of images and is difficult to capture in the one image as the photometric area, and determining the other area as the photometric area, the area in which a bright object that is not the object to be detected is captured is excluded from the photometric area, and automatic exposure control suitable for detecting the object to be detected can be performed on the other image.

(Technology 2) The image processing device described in Technology 1, in which the two types of images are a visible light image and an infrared image, one of the images being the infrared image and the other image being the visible light image.

Infrared images tend to show the object being detected more clearly than visible light images, and bright areas that are not the object being detected are less visible than in visible light images. Therefore, by not determining areas that are bright in the visible light image but are difficult to see in the infrared image as the photometry area, and instead determining other areas as the photometry area, areas that show bright objects that are not the object being detected are excluded from the photometry area, making it possible to perform automatic exposure control for the visible light image that is suitable for detecting the object being detected.

(Technology 3) The image processing device described in Technology 2, in which the infrared image is an image obtained by a ToF sensor.

ToF sensors can capture infrared images that clearly show nearby objects and not so clearly show distant objects.

(Technology 4) The image processing device described in Technology 3, in which the determination unit determines, as the photometric area, an area in the visible light image corresponding to an area in the infrared image in which the reflected light component acquired by the ToF sensor is equal to or greater than a predetermined intensity.

Since the intensity of the reflected light components from nearby objects is high and the intensity of the reflected light components from distant objects is low, it is possible to obtain an infrared image in which nearby objects with high reflected light component intensity are easily captured and distant objects with low reflected light component intensity are not easily captured. Areas with high reflected light component intensity are areas where the detected object may be captured, and can therefore be determined as photometry areas.

(Technology 5) The image processing device according to Technology 3 or 4, in which the determination unit does not determine, as the photometric area, an area in the visible light image corresponding to an area in the infrared image in which light components other than reflected light acquired by the ToF sensor are greater than a predetermined intensity.

The intensity of light components other than reflected light from high-brightness objects such as direct sunlight (background components) is very high, and areas where the intensity of light components other than reflected light is high are areas where high-brightness objects such as direct sunlight may be captured, so they can be prevented from being determined as photometry areas.

(Technology 6) An image processing method executed by an image processing device, the image processing method including: an acquisition step of acquiring two types of images showing the same area obtained by two types of imaging means having different wavelengths; and a determination step of determining, based on one of the two types of images, a photometric area in which photometry is performed in the other of the two types of images.

This makes it possible to provide an image processing method that can perform automatic exposure control suitable for detecting an object to be detected.

(Technology 7) A program for causing a computer to execute the image processing method described in Technology 6.

This makes it possible to provide a program that can perform automatic exposure control suitable for detecting the object to be detected.

This disclosure can be applied to devices for detecting objects to be detected.

REFERENCE SIGNS LIST 10 Exposure

time setting section

11, 31 Photometry section 12 Exposure time calculation section 20

Image synthesis section

21a, 21b, 21c Offset correction section 22 Output synthesis section 30 Gain correction section 32 Gain calculation section 33 Gain application section 100 Image processing device 200 Imaging section

Claims

an acquisition unit that acquires two types of images of the same area obtained by two types of imaging means having different wavelengths;
a determination unit that determines, based on one of the two types of images, a photometric area in which photometry is performed in the other of the two types of images,
Image processing device.
the two types of images being a visible light image and an infrared image;
the one image is the infrared image;
the other image is the visible light image;
The image processing device according to claim 1 .
The infrared image is an image obtained by a ToF (Time of Flight) sensor.
The image processing device according to claim 2 .
The determination unit determines, as the photometric area, an area in the visible light image corresponding to an area in the infrared image in which a component of reflected light acquired by the ToF sensor has a predetermined intensity or more.
The image processing device according to claim 3 .
the determination unit does not determine, as the photometric area, an area in the visible light image corresponding to an area in the infrared image in which a light component other than the reflected light acquired by the ToF sensor is greater than a predetermined intensity.
5. The image processing device according to claim 3.
An image processing method executed by an image processing device, comprising:
An acquisition step of acquiring two types of images of the same area obtained by two types of imaging means having different wavelengths from each other;
and determining, based on one of the two types of images, a photometric area in the other of the two types of images in which photometric metering is performed.
Image processing methods.
A program for causing a computer to execute the image processing method according to claim 6.