JP2024053482A - Imaging system, imaging method, and program - Google Patents

Abstract

An object is to provide a method for preventing the stereo distance measurement range from being narrowed even when there is a large difference in brightness in an imaging system.
[Solution] An imaging system comprising a first imaging means for outputting a first image signal and a second image signal, a second imaging means for outputting a third image signal, a developing means for generating image information based on each image signal, a stereo ranging means for generating first distance information based on the first image and the second image, a determination means for determining the state of an image based on the first image signal and the second image signal, a control means for controlling the exposure time for imaging to be longer than the other based on the state of the image, and an image synthesis means for generating distance image information based on the image information and the first distance information after controlling the exposure time.
[Selected figure] Figure 4

Description

The present invention relates to a method for expanding the dynamic range in an imaging system that uses multiple imaging devices.

In recent years, cameras are increasingly being used as the eyes of cars and robots. Traditionally, cameras use imaging elements such as CMOS (Complementary Metal Oxide Semiconductor) image sensors to detect the intensity of light (brightness information) and output images. Meanwhile, in addition to cameras that capture normal images, distance measuring cameras that detect the distance to a subject are also becoming more common.

One known distance measurement method for cameras using image sensors is stereo ranging, which calculates the distance to an object from the parallax images of two cameras. This technology is being used to calculate the distance to obstacles or objects from two cameras installed in a vehicle, and is beginning to be used for purposes such as autonomous vehicle driving. Another distance measurement method is the image plane phase difference distance measurement method used in digital cameras. By having two light receiving parts within one pixel, it is possible to obtain two images with parallax and perform stereo distance measurement. Cameras using this image sensor enable distance measurement with a single eye.

Incidentally, one of the performance features of imaging devices such as cameras is the dynamic range. The dynamic range is the ratio between the maximum and minimum brightness values that a camera can distinguish. When capturing an image of a subject with a brightness difference that exceeds the dynamic range of a camera, if the exposure is adjusted to the bright parts, the dark parts will be crushed to black and no information can be obtained. Also, if the exposure is adjusted to the dark parts, the bright parts will be blown out and no information can be obtained. For example, when capturing an image of a scene such as the tunnel exit from inside the tunnel as shown in Figure 1, if the exposure is adjusted to the inside 100 of the tunnel, the outside 101 of the tunnel will be blown out, and if the exposure is adjusted to the outside 101 of the tunnel, the inside 100 of the tunnel will be crushed to black. The same is true for distance measuring cameras, and there is a problem that distance measuring information cannot be obtained with brightness outside the dynamic range.

To properly capture information in scenes with such large differences in brightness, the camera's dynamic range needs to be expanded.

Patent Document 1 proposes that when an imaging system captures a scene with a large difference in brightness, the exposure times of the left and right cameras are changed to create a single composite image. By changing the exposure times of the left and right cameras, even if one of the cameras has blown out highlights or crushed shadows, the blown out highlights and crushed shadows can be compensated for by combining the image captured by the other camera.

This technology can be used to expand the dynamic range of images in imaging systems.

Patent No. 05411842

However, when the dynamic range is expanded using the technology of Patent Document 1, although the dynamic range of normal images is expanded, the range in which stereo distance measurement is possible is narrowed.

The object of the present invention is to provide a method for preventing the stereo ranging range from being narrowed even when there is a large difference in brightness in an imaging system.

In order to solve the above problems, the present invention provides
a first imaging means for capturing an image and outputting a first image signal and a second image signal having a parallax therebetween;
a second imaging means for capturing an image and outputting a third image signal having a parallax between the first image signal and the second image signal;
a developing means for generating image information based on the first image signal, the second image signal, and the third image signal;
a stereo distance measuring means for generating first distance information based on the first image and the second image;
a determination means for determining a state of an image captured by the first imaging means based on the first image signal and the second image signal;
a control means for controlling an exposure time for one of the first image signal and the second image signal based on a state of an image captured by the first image capture means so that the exposure time for the other of the first image signal and the second image signal is longer than the exposure time for the other of the first image signal and the second image signal based on a state of an image captured by the first image capture means;
and an image synthesis unit that generates distance image information based on the image information and the first distance information after controlling the exposure time.

According to the present invention, in an imaging system, it is possible to prevent the stereo ranging range from being narrowed even when there is a large difference in brightness.

Example of a situation with a large difference in brightness A diagram showing the image capture range and distance measurement range of a normal stereo camera. A diagram showing the imageable luminance range and distance measurement range when the exposure time is changed for the left and right cameras. FIG. 1 is a block diagram showing an example of the configuration of an imaging system according to a first embodiment. (A) is a top view of the configuration of the image sensor as seen from the direction of light incidence, and (B) is a cross-sectional view of the pixel group 5000 taken along the line I-I'. (A) A schematic diagram showing light incident on each light source conversion unit of the green pixel G1; (B) A schematic diagram showing the image sensor and the image capturing optical system when in focus; (C) A schematic diagram showing the image sensor and the image capturing optical system when defocused in the negative direction; and (D) A schematic diagram showing the image sensor and the image capturing optical system when defocused in the positive direction. FIG. 1 is a diagram showing an operation flow of the imaging system according to the first embodiment. FIG. 1 is a diagram showing an imageable luminance range and a distance measurement range in a normal mode of the imaging system according to the first embodiment; FIG. 1 is a diagram showing an imageable luminance range and a distance measurement range in an HDR mode of an imaging system according to a first embodiment; FIG. 11 is a block diagram of a control system related to an imaging system according to a second embodiment. FIG. 11 is a diagram showing an operation flow of an imaging system according to a second embodiment. FIG. 13 is a diagram showing an imageable luminance range and a distance measurement range in a normal mode of an imaging system according to a second embodiment; FIG. 13 is a diagram showing an imageable luminance range and a distance measurement range in an HDR mode of an imaging system according to a second embodiment;

The following describes in detail an embodiment of the present invention with reference to the drawings. In each drawing, the same reference numbers are used for the same components, and duplicate descriptions are omitted.

First Embodiment
Hereinafter, a first embodiment of the present invention will be described.

The imaging system uses a block matching method that measures distance using the parallax between images captured by the left and right cameras. To use this block matching method, distance can only be measured for subjects that are captured by both the left and right cameras.

Using Figures 2 and 3, we explain how the range in which stereo distance measurement is possible decreases as the dynamic range of the composite image expands. In Figures 2 and 3, the vertical axis indicates the brightness of the subject, with the subject becoming darker as it moves downwards and the subject becoming brighter as it moves upwards.

When the cameras that make up the imaging system are camera A and camera B, both cameras are controlled with the same exposure time during normal shooting, so what is captured with proper exposure is the imageable range 202 and imageable range 205 shown by solid lines in Figure 2. Also, the areas that are blown out or blown out are shown by dotted lines as blown out highlights 201, blown out highlights 204, crushed shadows 203, and crushed shadows 206. At this time, the range where stereo distance measurement is possible is the measurable range 200 shown by diagonal lines. Therefore, if the exposure time of camera A is shortened and the exposure time of camera B is lengthened, it becomes possible to capture bright subjects with camera A and dark subjects with camera B with proper exposure.

When controlled in this way, the imageable ranges of cameras A and B are imageable range 301 and imageable range 304 shown by solid lines in Figure 3, and the unimageable areas are crushed blacks 302 and blown out highlights 303 shown by dotted lines. By shifting the exposure times of cameras A and B, the imageable range becomes the combined imageable range 301 and imageable range 304. However, the only locations that can be measured using the block matching method are measurable range 300, which is the common part of imageable range 301 and imageable range 304, and the measurable range becomes narrower.

Figure 4 is a block diagram showing an example of the configuration of an imaging system according to an embodiment of the present invention.

This imaging system is a system composed of a stereo camera 4000, a processing unit 4100 (information processing device), and a display unit 4200.

The stereo camera 4000 is composed of an imaging device 4010 (first imaging means) and an imaging device 4020 (second imaging means).

The imaging device 4010 is composed of an imaging optical system 4011 and an imaging element 4012.

The imaging optical system 4011 can form an image (optical image) of a subject on the imaging element 4012, and has an exit pupil located a predetermined distance away from the imaging element 4012.

The image sensor 4012 has a pixel region in which pixels having a photoelectric conversion function are arranged two-dimensionally. Each pixel region (not shown) has two photoelectric conversion units (first photoelectric conversion unit, second photoelectric conversion unit) and photoelectrically converts the subject image formed on the image sensor 4012 to generate an image signal based on the subject image. The image sensor 4012 outputs a first image signal based on the signal output from the first photoelectric conversion unit and a second image signal based on the signal output from the second photoelectric conversion unit to the development unit 4101 and phase difference distance measurement unit 4102 in the processing unit 4100. The first image signal and the second image signal have a parallax. Since the image sensor 4012 is configured to output the first image signal and the second image signal as described above, the imaging system in this embodiment is capable of distance measurement using not only stereo distance measurement but also an imaging surface phase difference distance measurement method described later. In addition, the exposure times of the first photoelectric conversion unit and the second photoelectric conversion unit can be controlled to different times.

The imaging device 4020 is composed of an imaging optical system 4021 and an imaging element 4022.

The imaging optical system 4021 is similar to the imaging optical system 4011, so a description thereof will be omitted.

The image sensor 4022 has a pixel region in which pixels having photoelectric conversion units are arranged two-dimensionally, and each pixel has one photoelectric conversion unit, and photoelectrically converts the subject image formed on the image sensor 4022 to generate an image signal (third image signal) based on the subject image. The third image signal has a parallax with respect to the first image signal and the second image signal. The image sensor 4022 outputs the image signal output from the photoelectric conversion unit to the development unit 4101 in the processing unit 4100.

The processing unit 4100 is composed of a development unit 4101, a phase difference distance measurement unit 4102, a stereo distance measurement unit 4103, a highlight/shadow crush determination unit 4104, an exposure control unit 4105, and an image synthesis unit 4106. The processing unit 4100 also includes a CPU (Central Processing Unit) that performs calculations and control (not shown), and a ROM (Read Only Memory) and RAM (Random Access Memory) that serve as main storage devices. Basic setting data and the program according to this embodiment are stored in the ROM. The CPU calls up a program corresponding to the processing content from the ROM, expands it in the RAM, and executes the operation of each block.

The developing unit 4101 generates image data (image information) having luminance information for each of the colors red, green, and blue for each pixel based on the first image signal and the second image signal transmitted from the image sensor 4012. Similarly, image data having luminance information for each of the colors red, green, and blue for each pixel is generated from the image signal transmitted from the image sensor 4022. The generated image data is transmitted to the stereo distance measurement unit 4103, the whiteout/blackout determination unit 4104, and the image synthesis unit 4106.

The phase difference distance measuring unit 4102 generates distance data (second distance information) indicating distance information for each pixel based on the first image signal and the second image signal transmitted from the image sensor 4012. The generated distance data is transmitted to the image synthesis unit 4106. Since the image data and the distance data are generated from the same image signal, the image data and the distance data are synchronized in time.

The stereo distance measurement unit 4103 calculates the parallax using the image data of the imaging device 4010 and the imaging device 4020 developed by the development unit 4101, and calculates distance information (first distance information) from the parallax information. One method for calculating parallax is the block matching method. The block matching method is a method in which a selected area of one image is searched for in the other image to find an area with high similarity to the other image, and the positional deviation from the area with high similarity is regarded as the parallax. The generated distance data is transmitted to the image synthesis unit 4106.

The determination unit 4104 determines whether there is blown-out highlights or crushed blacks in the image data captured by the imaging device 4010 and developed by the development unit 4101. A blown-out highlight state refers to a portion in the image signal where the luminance value is the maximum value (e.g., 255) or near the maximum value. A crushed black state refers to a portion in the image signal where the luminance value is the minimum value (e.g., 0 or a black level equivalent to the luminance when the imaging element is shielded) or near the minimum value. The above-mentioned luminance value near the maximum value means that it is determined to be blown-out highlights when the luminance value is equal to or greater than a predetermined threshold value W1 (a first threshold value or greater, e.g., 240 or greater). Also, the vicinity of the minimum value means that it is determined to be crushed blacks when the luminance is equal to or less than a predetermined threshold value B1 (a second threshold value or less, e.g., 15 or less) that is smaller than the threshold value W1. If neither blown-out highlights nor crushed blacks are detected in the image signal, the exposure control unit 4105 is informed of control in normal mode. In addition, if blown-out highlights, crushed shadows, or both are detected in the image signal, the exposure control unit 4105 is notified to perform control in HDR mode to suppress blown-out highlights and crushed shadows.

The exposure control unit 4105 changes the exposure control depending on the judgment result of the judgment unit 4104. If neither blown-out nor crushed shadows are detected by the judgment unit 4104, control is performed in normal mode, and the stereo camera 4000 is controlled so that the photoelectric conversion units in each pixel in each imaging device have the same exposure. If the judgment unit 4104 detects blown-out or crushed shadows, or both, the stereo camera 4000 is switched to HDR mode and the exposure time is changed for each photoelectric conversion unit of each pixel. In HDR mode, the change in exposure time in the left and right photoelectric conversion units may be controlled in the same way for all pixels (all pixels) in the imaging element 4012, or it may be adjustable so that it can be applied only to areas where blown-out highlights/crushed shadows occur.

In addition, the exposure time of at least one of the first photoelectric conversion unit and the second photoelectric conversion unit in the image sensor 4012 may be controlled to be longer than the other.

The image synthesis unit 4106 synthesizes an image to be displayed on the display unit 4200 using the image data output by the development unit 4101 and the distance measurement data generated by the phase difference distance measurement unit 4102 and the stereo distance measurement unit 4103. The image (distance image information) generated by the image synthesis unit 4106 (which generates a synthetic image) is transmitted to the display unit 4200.

The display unit 4200 is a display means such as an LCD display, and displays the video transmitted from the image synthesis unit 4106.

Here, the principle of distance measurement using the image plane phase difference distance measurement method using the image sensor 4012 and the image sensor 4022 will be explained using Figures 5 and 6.

Figure 5 is a schematic diagram showing the configuration of the image sensor 4012 as an example. Figure 5 (A) is a top view of the image sensor 4012 seen from the light incidence direction. The image sensor 4012 is configured by arranging a plurality of pixel groups 5000 of 2 rows x 2 columns in a matrix shape. The pixel group 5000 has green pixels G1 and G2 that detect green light, red pixels R that detect red light, and blue pixels B that detect blue light. In the pixel group 5000, the green pixels G1 and G2 are arranged diagonally. Each pixel also has a first photoelectric conversion unit 5001 and a second photoelectric conversion unit 5002. As described above, the exposure times of the first photoelectric conversion unit 5001 and the second photoelectric conversion unit 5002 can be controlled to different times.

Figure 5 (B) is a cross-sectional view of the pixel group 5000 in Figure 5 (A) along the line I-I'. Each pixel is composed of a microlens 5003, a light guide layer 5004, and a light receiving layer 5005.

The light guide layer 5004 is a light guide member that includes microlenses 5003 for efficiently guiding the light flux incident on the pixel to the light receiving layer 5005, color filters that pass light in the wavelength band corresponding to the color of light detected by each pixel, and wiring for reading out the image and driving the pixels.

The light receiving layer 5005 is a photoelectric conversion section that photoelectrically converts the light incident through the light guide layer 5004 and outputs it as an electrical signal. The light receiving layer 5005 has a first photoelectric conversion section 5001 and a second photoelectric conversion section 5002.

Figure 6 is a schematic diagram showing the relationship between subject distance and incident light in the image plane phase difference ranging method. Figure 6 (A) is a schematic diagram showing the exit pupil 6000 of the imaging optical system 4011, the green pixel G1 of the imaging element 4012, and the light incident on each light source conversion unit of the green pixel G1. The imaging element 4012 has multiple pixels, but for simplicity, one green pixel G1 will be described.

The microlens 5003 of the green pixel G1 is arranged so that the exit pupil 6000 and the light guide layer 5004 are optically conjugate with each other. As a result, the light beam that passes through the first pupil region 6010, which is a partial pupil region contained in the exit pupil 6000, enters the first photoelectric conversion unit 5001. Similarly, the light beam that passes through the second pupil region 6020, which is also a partial pupil region, enters the second photoelectric conversion unit 5002.

The first photoelectric conversion unit 5001 of each pixel performs photoelectric conversion on the received light beam and outputs a signal. A first image signal is generated from signals output from the multiple first photoelectric conversion units 5001 included in the image sensor 4012. The first image signal indicates the intensity distribution of an image formed on the image sensor 4012 by the light beam that has mainly passed through the first pupil region 6010.

The second photoelectric conversion unit 5002 of each pixel performs photoelectric conversion on the received light beam and outputs a signal. A second image signal is generated from the signals output from the multiple second photoelectric conversion units 5002 included in the image sensor 4012. The second image signal indicates the intensity distribution of the image formed on the image sensor 4012 by the light beam that has mainly passed through the second pupil region 6020.

The relative positional shift amount between the first image signal and the second image signal (hereinafter referred to as the parallax amount) is an amount that depends on the defocus amount. The relationship between the parallax amount and the defocus amount will be explained using Figures 6 (B), (C), and (D).

Figures 6(B), (C), and (D) are schematic diagrams showing the image sensor 4012 and the image pickup optical system 4011. In the figures, 6011 indicates a first light beam passing through the first pupil region 6010, and 6021 indicates a second light beam passing through the second pupil region 6020.

Figure 6 (B) shows the state when in focus, where the first light beam 6011 and the second light beam 6021 converge on the image sensor 4012. At this time, the parallax amount between the first image signal formed by the first light beam 6011 and the second image signal formed by the second light beam 6021 is 0.

Figure 6 (C) shows a state where the image side is defocused in the negative direction of the w axis. At this time, the parallax amount between the first image signal formed by the first light beam 6011 and the second image signal formed by the second light beam 6021 is not 0, but has a negative value.

Figure 6 (D) shows a state where the image side is defocused in the positive direction of the w axis. At this time, the amount of parallax between the first image signal formed by the first light beam 6011 and the second image signal formed by the second light beam 6021 is not 0 but has a positive value.

Comparing Figures 6(C) and (D) reveals that the direction in which parallax occurs changes depending on whether the defocus amount is positive or negative. Furthermore, from the geometric relationship, it can be seen that the amount of parallax occurs according to the amount of defocus. Therefore, the amount of parallax between the first image signal and the second image signal can be detected by a region-based matching method, and the amount of parallax can be converted into the amount of defocus via a predetermined conversion coefficient. Furthermore, by using the imaging formula of the imaging optical system 4011, the amount of defocus on the image side can be converted into the distance to the object.

The above is an explanation of the principle of distance measurement using the image plane phase difference distance measurement method. In this embodiment, the phase difference distance measurement unit 4102 generates distance data from the first image signal and the second image signal using the image plane phase difference distance measurement method.

Figure 7 is a flowchart showing the operation of the imaging system according to the first embodiment. This flowchart is executed by a CPU (not shown) of the processing unit 4100.

In step S1, the operation is switched to a normal mode in which the first photoelectric conversion unit and the second photoelectric conversion unit constituting each pixel of the image sensor 4012 are controlled with the same exposure. The image sensor 4022 is also controlled to have the same exposure setting as the image sensor 4012. FIG. 8 shows the measurable range and the imageable range in the normal mode. As in FIG. 2, the vertical axis shows the brightness of the subject, with the subject darker toward the bottom and brighter toward the top. The imageable ranges of the image sensor 4010 and the image sensor 4020 are the imageable range 802 and the imageable range 805, respectively. The ranges where whiteout occurs are whiteout 801 and whiteout 804, respectively, and the ranges where blackout occurs are blackout 803 and blackout 806. The measurable range is indicated by the measurable range 800. The image signals captured by each image sensor of the stereo camera 4000 are developed by the developing unit 4101 and transmitted to the determining unit 4104, and the process proceeds to step S2.

In step S2, the determination unit 4104 uses the image signal captured in step S1 to determine whether the image signal contains whiteout, blackout, or both. If it is determined that the image signal does not contain whiteout or blackout, the process proceeds to step S3. If it is determined that the image signal contains whiteout or blackout, the process proceeds to step S5.

In step S3, distance information is calculated from the image data captured by the stereo camera 4000 in step S1. The calculation of distance information is performed by the phase difference distance measurement unit 4102 and the stereo distance measurement unit 4103. The method of generating distance data in each distance measurement unit is as described above. For distance measurement locations common to the phase difference distance measurement unit 4102 and the stereo distance measurement unit 4103, distance measurement information from only one of them may be used preferentially, or both pieces of information may be weighted to calculate distance information. Once development of the image data and generation of distance information are completed, proceed to step S4.

In step S4, the image developed in step S3 and the distance information generated are synthesized in the image synthesis unit 4106 to generate an image to be displayed on the display unit 4200. Once the generated synthesized image is transmitted to the display unit 4200, the process proceeds to step S9.

In step S5, the operation mode of the stereo camera 4000 is switched to HDR mode. The exposure control unit 4105 controls the stereo camera 4000 to change the exposure time for each photoelectric conversion unit of each pixel. When the determination unit 4104 transmits a control signal for changing the exposure time for each photoelectric conversion unit to the exposure control unit 4105, the process proceeds to step S6.

In step S6, control is performed to make the exposure times different between the first photoelectric conversion unit and the second photoelectric conversion unit constituting each pixel for operation in HDR mode. By performing this control, an image with a wide dynamic range can be generated. Taking the green pixel G2 in FIG. 5(A) as an example, the exposure times are made different between the first photoelectric conversion unit 5001 and the second photoelectric conversion unit 5002. By making the exposure time of the first photoelectric conversion unit 5001 longer and the exposure time of the second photoelectric conversion unit 5002 shorter, two image data with different exposure periods can be generated. The first output signal output from the first photoelectric conversion unit 5001 can output an image in which the dark areas of the imaging target are brightly imaged, and the second output signal output from the second photoelectric conversion unit 5002 can output an image in which the saturation of the bright areas of the imaging target is suppressed. By combining the first output signal and the second output signal, an image with a wide dynamic range can be obtained in which the dark areas are brightly expressed while the saturation of the bright areas is suppressed. The image signal output in HDR mode is transmitted to the development unit 4101 and the process proceeds to step S7.

In step S7, the development unit 4101 generates an HDR image, and the stereo distance measurement unit 4103 generates distance information using the generated HDR image. When generating an HDR image, one of the images is used as a reference image, and if the exposure is not appropriate for the reference image (existence of white blowout or black crush), the image is synthesized using the luminance value of the pixels in the corresponding area of the other image. In this way, an image captured with appropriate exposure from light to dark areas can be obtained. In addition, the stereo distance measurement unit 4103 calculates parallax using the output HDR image, and calculates distance information from the parallax information. For example, a block matching method is used as a method for calculating parallax. The block matching method is a method in which a selected area of one image is searched for in the other image to find an area with high similarity, and the positional deviation from the area with high similarity is used as the parallax. Figure 9 shows the imaging range and distance measurement range during operation in HDR mode. By setting different exposure times between the first photoelectric conversion unit 5001 and the second photoelectric conversion unit 5002, the image capture range of the imaging device 4010 becomes the range indicated by the image capture range 901, which shows a wider dynamic range compared to the image capture range 802. Also, the range where distance measurement is possible becomes the distance measurement range 900, which shows that the distance measurement range is not narrower compared to the distance measurement range 800. When the generation of the HDR image by the development unit 4101 and the generation of distance information by the stereo distance measurement unit 4103 are completed, proceed to step S8.

In step S8, the HDR image and distance information generated in step S7 are transmitted to the image synthesis unit 4106, and the image synthesis unit 4106 generates an image to be displayed on the display unit 4200. The generated synthesis image is transmitted to the display unit 4200, and once it is displayed on the display unit 4200, the process proceeds to step S9.

In step S9, the operation is switched to a normal mode in which the first photoelectric conversion unit and the second photoelectric conversion unit constituting each pixel of the image sensor 4012 are controlled with the same exposure. When the control of the stereo camera 4000 is switched to the normal mode control, the series of operations is terminated.

In the first embodiment, two photoelectric conversion units are provided in each pixel area of the image sensor, and the dynamic range is expanded by shifting the exposure time of each photoelectric conversion unit when whiteout/blackout occurs. As a result, the dynamic range of the image can be expanded while preventing the stereo distance measurement range from being narrowed by performing stereo distance measurement using an image with an expanded dynamic range.

Second Embodiment
A second embodiment of the present invention will now be described.

Figure 10 is a block diagram showing an example of the configuration of an imaging system according to an embodiment of the present invention.

This imaging system is a system composed of a stereo camera 4300, a processing unit 4100, and a display unit 4200.

The stereo camera 4300 is composed of an imaging device 4010 and an imaging device 4030 (second imaging means).

The imaging device 4010 is similar to that in the first embodiment, so a detailed description is omitted.

The imaging device 4030 is composed of an imaging optical system 4021 and an imaging element 4032.

The imaging optical system 4021 is the same as in the first embodiment, so a description thereof will be omitted.

The imaging element 4032 uses an imaging element having a structure similar to that of the imaging element 4012. The imaging element 4032 outputs a first image signal (third image signal) based on a signal output from the first photoelectric conversion unit (third photoelectric conversion means) to the development unit 4101 and phase difference distance measurement unit 4102 in the processing unit 4100. The imaging element 4032 also outputs a second image signal (fourth image signal) based on a signal output from the second photoelectric conversion unit (fourth photoelectric conversion means) to the development unit 4101 and phase difference distance measurement unit 4102 in the processing unit 4100. The fourth image signal has a parallax with respect to the third image signal.

The processing unit 4100 and display unit 4200 are also omitted because they are similar to those in the first embodiment.

Figure 11 is a flowchart showing the operation of the imaging system according to the second embodiment. This flowchart is executed by a CPU (not shown) of the processing unit 4100.

In step S11, the operation is switched to a normal mode in which the first photoelectric conversion unit and the second photoelectric conversion unit constituting each pixel of the image sensor 4012 and the image sensor 4032 are controlled with the same exposure. FIG. 12 shows the measurable range and the imageable range in the normal mode. As in FIG. 2, the brightness of the subject is shown on the vertical axis, with the subject darker toward the bottom and brighter toward the top. The imageable ranges of the image sensor 4010 and the image sensor 4030 are the imageable range 812 and the imageable range 815, respectively. The ranges where whiteout occurs are whiteout 811 and whiteout 814, respectively, and the ranges where blackout occurs are blackout 813 and blackout 816. The measurable range is indicated by the measurable range 810. The image signals captured by each image sensor of the stereo camera 4000 are developed by the developing unit 4101 and transmitted to the determining unit 4104, and the process proceeds to step S12.

In step S12, the determination unit 4104 uses the image signal captured in step S11 to determine whether the image signal contains whiteout, blackout, or both. If it is determined that the image signal does not contain whiteout or blackout, the process proceeds to step S13. If it is determined that the image signal contains whiteout or blackout, the process proceeds to step S15.

In step S13, distance information is calculated from the image data captured by the stereo camera 4300 in step S11. The calculation of distance information is performed by the phase difference distance measurement unit 4102 and the stereo distance measurement unit 4103. The method of generating distance data in each distance measurement unit is as described above. For distance measurement locations common to the phase difference distance measurement unit 4102 and the stereo distance measurement unit 4103, distance measurement information from only one of the information from the phase difference distance measurement unit 4102 (third distance information) and the information from the stereo distance measurement unit 4103 may be used preferentially. Also, distance information may be calculated by weighting both pieces of information. Once development of the image data and generation of the distance information are completed, proceed to step S14.

In step S14, the image developed in step S13 and the generated distance information are synthesized in the image synthesis unit 4106 to generate an image to be displayed on the display unit 4200. Once the generated synthesized image is transmitted to the display unit 4200, the process proceeds to step S19.

In step S15, the operation mode of the stereo camera 4300 is switched to HDR mode. The exposure control unit 4105 controls the stereo camera 4300 to change the exposure time for each photoelectric conversion unit of each pixel. When the determination unit 4104 transmits a control signal for changing the exposure time for each photoelectric conversion unit to the exposure control unit 4105, the process proceeds to step S16.

In step S16, control is performed to make the exposure times different between the first photoelectric conversion unit and the second photoelectric conversion unit that constitute each pixel in order to operate in HDR mode. The image signal output in HDR mode is transmitted to the development unit 4101, and the process proceeds to step S17.

In step S17, the development unit 4101 generates an HDR image, and the stereo distance measurement unit 4103 generates distance information using the generated HDR image. Furthermore, the stereo distance measurement unit 4103 calculates parallax using the output HDR image, and calculates distance information from the parallax information. For example, a block matching method is used to calculate parallax. The block matching method is a method in which a selected area of one image is searched for in another image to find an area with high similarity, and the positional deviation between the area with high similarity is taken as the parallax. The imageable range and the distance measurement range during operation in HDR mode are shown in FIG. 13. By making the exposure times different between the first photoelectric conversion unit and the second photoelectric conversion unit, the imageable ranges of the image capture device 4010 and the image capture device 4030 are the ranges indicated by the imageable range 911 and the imageable range 912. In other words, it can be seen that the dynamic range is wider than the imageable range 812 and the imageable range 815. In addition, the range in which distance can be measured is the distance measurement range 910, which is wider than the distance measurement range 810. Once the generation of the HDR image by the development unit 4101 and the generation of distance information by the stereo distance measurement unit 4103 are completed, the process proceeds to step S18.

In step S18, the HDR image and distance information generated in step S17 are transmitted to the image synthesis unit 4106, and the image synthesis unit 4106 generates an image to be displayed on the display unit 4200. The generated synthesis image is transmitted to the display unit 4200, and once it is displayed on the display unit 4200, the process proceeds to step S19.

In step S19, the operation is switched to a normal mode in which the first photoelectric conversion unit and the second photoelectric conversion unit constituting each pixel of the image sensor 4012 and the image sensor 4032 are controlled with the same exposure. When the control of the stereo camera 4300 is switched to the normal mode control, the series of operations is terminated.

In the second embodiment, two photoelectric conversion units are provided in each pixel region of the image sensor in the two imaging devices, and the dynamic range is expanded by shifting the exposure time of each photoelectric conversion unit when whiteout/blackout occurs. As a result, by performing stereo ranging using images with expanded dynamic ranges in each of the two imaging devices, it is possible to expand the dynamic range of the image while suppressing the narrowing of the stereo ranging range compared to the first embodiment.

The present invention has been described in detail above based on preferred embodiments, but the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the invention are also included in the present invention. Parts of the above-mentioned embodiments may be combined as appropriate.

The present invention also includes cases where a software program that realizes the functions of the above-mentioned embodiments is supplied to a system or device having a computer capable of executing the program directly from a recording medium or via wired/wireless communication, and the program is executed.

Therefore, the program code that is supplied to and installed on a computer to realize the functional processing of the present invention also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

In this case, as long as it has the functionality of a program, the form of the program is not important, such as object code, a program executed by an interpreter, or script data supplied to the OS.

The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical/magnetic optical storage medium, or a non-volatile semiconductor memory.

Another possible method for supplying the program is to store the computer program forming the present invention on a server on a computer network, and have connected client computers download and program the computer program.

4000, 4300 Stereo camera 4010, 4020, 4030 Imaging device 4011, 4021 Imaging optical system 4012, 4022, 4032 Imaging element 4100 Processing section 4101 Development section 4102 Phase difference distance measuring section 4103 Stereo distance measuring section 4104 Determination section 4105 Exposure control section 4106 Image synthesis section 4200 Display section 5000 Pixel group 5001 First photoelectric conversion section 5002 Second photoelectric conversion section

Claims (10)

a first imaging means for capturing an image and outputting a first image signal and a second image signal having a parallax therebetween;
a second imaging means for capturing an image and outputting a third image signal having a parallax between the first image signal and the second image signal;
a developing means for generating image information based on the first image signal, the second image signal, and the third image signal;
a stereo distance measuring means for generating first distance information based on a first image signal, the second image signal, and a third image signal;
a determination means for determining a state of an image captured by the first imaging means based on the first image signal and the second image signal;
a control means for controlling an exposure time for one of the first image signal and the second image signal based on a state of an image captured by the first image capture means so that the exposure time for the other of the first image signal and the second image signal is longer than the exposure time for the other of the first image signal and the second image signal based on a state of an image captured by the first image capture means;
and an image synthesis unit that generates distance image information based on the image information and the first distance information after controlling the exposure time. The imaging system according to claim 1 , wherein the determining means determines whether the image captured by the first imaging means has whiteout or blackout based on luminance values of the first image signal and the second image signal. the determination means determines that the image captured by the first imaging means has whiteout when a luminance value of at least one of the first image signal and the second image signal is equal to or greater than a first threshold, and determines that the image captured by the first imaging means has blackout when a luminance value of at least one of the first image signal and the second image signal is equal to or less than a second threshold that is smaller than the first threshold,
3. The imaging system according to claim 2, wherein, when the determining means determines that at least one of whiteout and blackout has occurred, the control means controls the exposure time so as to suppress at least one of whiteout and blackout. a phase difference distance measuring means for generating second distance information based on the first image signal and the second image signal,
2. The imaging system according to claim 1, wherein after controlling the exposure time, the image synthesis means generates the distance image information based on the second distance information. the second imaging means outputs a fourth image signal having a parallax with respect to the third image signal by imaging;
the phase difference distance measuring means generates third distance information based on the third image signal and the fourth image signal;
the determining means determines a state of the image captured by the second imaging means based on the third image signal and the fourth image signal;
5. The imaging system according to claim 4, wherein the control means controls, based on a state of the image captured by the second imaging means determined by the determination means, an exposure time for one of the imaging of the third image signal and the imaging of the fourth image signal so as to be longer than the other. 5. The imaging system according to claim 4, wherein the control means is adjustable to control all pixels of the first imaging means or to control only some pixels. The imaging system according to claim 6, characterized in that, when generating the distance image information, the image synthesis means is capable of calculating distance information by weighting each distance data for a range commonly measured by the stereo ranging means and the phase difference ranging means. a developing means for generating image information based on a first image signal output from a first imaging means by imaging and a second image signal having a parallax with respect to the first image signal, and a third image signal output from a second imaging means by imaging and having a parallax with respect to the first image signal and the second image signal;
a stereo distance measuring means for generating first distance information based on the first image signal, the second image signal, and the third image signal;
a determination means for determining a state of an image captured by the first imaging means based on the first image signal and the second image signal;
a control means for controlling an exposure time for one of the first image signal and the second image signal based on a state of an image captured by the first image capture means so that the exposure time for the other of the first image signal and the second image signal is longer than the exposure time for the other of the first image signal and the second image signal based on a state of an image captured by the first image capture means;
and an image synthesis unit that generates distance image information based on the image information and the first distance information after controlling the exposure time.
13. An information processing device comprising: a first imaging step by a first imaging means for outputting a first image signal and a second image signal having a parallax with each other by imaging;
a second imaging step by a second imaging means for outputting a third image signal having a parallax between the first image signal and the second image signal by imaging;
a developing step of generating image information based on the first image signal, the second image signal, and the third image signal;
a stereo ranging process for generating first distance information based on the first image signal, the second image signal, and the third image signal;
a determination step of determining a state of an image captured by the first imaging means based on the first image signal and the second image signal;
a control step of controlling an exposure time for one of the first image signal and the second image signal based on a state of an image captured by the first imaging means so that the exposure time for the other of the first image signal and the second image signal is longer than the other;
an image synthesis step of generating distance image information based on the image information and the first distance information after the control step. A program for causing a computer to function as each of the means of the imaging system described in any one of claims 1 to 7.

Images