JP2024053855A - Stereo adapter, stereo camera and stereo camera system - Google Patents

Abstract

[Problem] The problem is to miniaturize a stereo camera. [Solution] The present invention is characterized in that it has prisms P10 and P20 that change the angle of a light ray propagating from an object, and the light ray L propagating from the object propagates inside the prisms P10 and P20, and the prisms P10 and P20 are arranged at a distance d in the optical axis direction. Furthermore, the prism P10 refracts the light ray L inward, and the prism P20 refracts the light ray L outward. Also, the refractive index of the prism P10 is greater than that of the prism P20. [Selected Figure] Figure 1

Description

The present invention relates to a stereo adapter, a stereo camera, and a stereo camera system.

A stereo image processing device (hereafter referred to as a stereo camera) is known as a device for recognizing objects three-dimensionally. A stereo camera uses the differences in how images are captured by multiple cameras placed in different positions to detect the parallax between multiple images based on trigonometry. The stereo camera then uses this parallax to detect the depth and position of an object. By using a stereo camera, the position of an object can be detected accurately.

On the other hand, stereo cameras require the use of two camera modules equipped with lenses and sensors, which poses issues in terms of miniaturization and cost. To address these issues, Patent Documents 1, 2, and 3 have been proposed.

In response to the problem of providing a distance measuring device that can be constructed with fewer parts than conventional devices, Patent Document 1 discloses a distance measuring device that "includes a prism, a lens, and an image sensor. Light from a subject enters the prism from two different directions. The prism sends the first light and second light that enter from the two different directions to the lens. The lens focuses the first light and second light sent from the prism on the imaging surface of the image sensor. The imaging surface includes adjacent right and left regions that are equal divisions of the imaging surface. The first light sent from the lens is focused on the right region, and the second light sent from the lens is focused on the left region. The image sensor sends to a control unit a right image formed by the first light, and a left image formed by the second light. The control unit measures the distance between the subject and the distance measuring device based on the right and left images sent from the image sensor" (see abstract).

In addition, Patent Document 2 discloses an imaging device and a stereoscopic imaging method for capturing stereoscopic images using a stereo adapter that is small, lightweight, inexpensive, and of high image quality, eliminating the degradation in image quality and pseudo-stereoscopic information caused by prisms and taking advantage of the inherent advantages of a prism-type stereo adapter (see abstract), which is "an imaging device for capturing multi-eye stereoscopic images, which includes a color imaging element 105 that photoelectrically converts the subject image formed by the imaging optical system 101 to obtain a subject image signal, a prism-type stereo adapter 200 that is placed in front of the imaging optical system 101 and receives light from the subject at different positions according to the parallax and directs it to different areas of the imaging element 105, and a chromatic aberration correction circuit that corrects chromatic aberration by shifting the image position of each color component relative to the subject image signal in response to the shift in image position of each of the multiple color components of the subject image signal caused by the prism-type stereo adapter" to address the issue of capturing stereoscopic images using a small, lightweight, inexpensive, and high image quality stereoscopic image by taking advantage of the inherent advantages of a prism-type stereo adapter.

Furthermore, Patent Document 3 relates to an optical component for stereoscopic imaging that enables three-dimensional imaging with a single camera, and a stereoscopic imaging device using the same. In particular, in response to the issue of an optical component for stereoscopic imaging consisting of an arrowhead-shaped composite achromatic prism with a fixed apex angle, and a device for stereoscopically capturing three-dimensional images, such as a still camera equipped with such an optical component, an endoscope, etc., which is equipped with such an optical component, the optical component discloses an optical lens for stereoscopic imaging that is equipped with a slit aperture close to the aperture stop and that uses the same, and the optical lens is equipped with the same (see abstract).

JP 2020-91154 A JP 2003-47028 A JP 2000-193883 A

The technology described in Patent Document 1 uses a prism to achieve compactness and low cost, and realizes a stereo camera with only one lens and image sensor. However, when a prism is used, the colors of the light that passes through the prism are separated, which affects parallax.

In contrast, the technology described in Patent Documents 2 and 3 uses two prisms to suppress chromatic aberration. Patent Documents 1, 2, and 3 also state that the stereo camera can be made smaller. However, there are serious problems to be solved in order to realize a small stereo camera using this technology. That is, the stereo camera detects images from two viewpoints, and the greater the distance between these two viewpoints (hereinafter referred to as the baseline length), the higher the detection accuracy becomes. And, as the baseline length increases, detection at a greater distance becomes possible. However, in the case of the technology described in Patent Documents 1, 2, and 3, in order to increase the baseline length, the distance between the prism and the lens must be increased, which leads to a problem that the stereo camera becomes larger. This problem is not described in Patent Documents 1, 2, and 3, and naturally, there is no description of a solution to this problem.

The present invention was made in light of this background, and aims to improve the distance measurement accuracy of stereo cameras.

In order to solve the above-mentioned problems, the present invention is characterized in that it has a first ray angle conversion unit and a second ray angle conversion unit that change the angle of a ray propagating from an object, the ray propagating from the object propagates inside the first ray angle conversion unit and the second ray angle conversion unit, and the first ray angle conversion unit and the second ray angle conversion unit are arranged at a predetermined distance from the optical axis.
Other solutions will be described in the embodiments as appropriate.

The present invention can improve the distance measurement accuracy of a stereo camera.

FIG. 2 is a configuration diagram of an optical system of the stereo camera according to the first embodiment. FIG. 11 is a configuration diagram of an optical system of a stereo camera in a comparative example. 13 is a diagram showing an image of an image on a sensor plane when a person is detected by a stereo camera in a comparative example. FIG. 11 is a diagram showing the relationship between the camera module incidence angle and the stereo camera incidence angle according to the present embodiment. FIG. 11 is a diagram showing the relationship between the camera module incidence angle and the stereo camera incidence angle according to the comparative example. FIG. FIG. 4 is a diagram showing dispersion of light by the configuration shown in the first embodiment. FIG. 13 is a diagram showing the dispersion of light obtained by a configuration of a comparative example. FIG. 13 is a diagram showing the relationship between module distance and base line length. FIG. 13 is a diagram showing a baseline length relative to a stereo camera incident angle. FIG. 1 is a diagram illustrating a configuration of a stereo camera system. FIG. 2 is a diagram illustrating a detailed configuration of a stereo parallax image generating unit. FIG. 1 is a diagram illustrating a hardware configuration of an image processing device. FIG. 11 is a diagram showing a modified example of the stereo camera according to the first embodiment. FIG. 1 is a diagram showing an example of sensor arrangement (part 1). FIG. 2 is a diagram showing an example of sensor arrangement (part 2). FIG. 3 is a diagram showing an example of sensor arrangement (part 3). FIG. 11 is a light path diagram in a comparative example in which the refractive index of a prism is increased. 13A and 13B are diagrams showing examples of images formed on a sensor in a comparative example in which the refractive index of a prism is increased; FIG. 2 is a diagram showing optical paths of a stereo camera according to the present embodiment. 5A and 5B are diagrams illustrating examples of images formed on a sensor by the configuration of this embodiment. FIG. 11 is a diagram illustrating the configuration of an optical system of a stereo camera according to a second embodiment. FIG. 13 is a diagram showing a modified example of the stereo camera according to the second embodiment. FIG. 11 is a configuration diagram of an optical system of a stereo camera according to a third embodiment. FIG. 13 is a diagram illustrating another example of the stereo camera according to the third embodiment.

Hereinafter, the present embodiment will be described with reference to the attached drawings. In the attached drawings, functionally identical elements may be indicated by the same numbers. Note that the attached drawings show an embodiment according to the principles of the present disclosure, but these are for the purpose of understanding the present disclosure and are in no way used to interpret the present disclosure in a restrictive manner. The descriptions in this specification are merely typical examples and do not limit the scope or application examples of the present disclosure in any sense.

In this embodiment, the disclosure has been described in sufficient detail for a person skilled in the art to implement the disclosure, but other embodiments are possible. It should be understood that configuration and structure can be changed and various elements can be replaced without departing from the scope and spirit of the technical ideas of the disclosure. Therefore, the following description should not be interpreted as being limited to this.

[First embodiment]
FIG. 1 is a diagram showing the configuration of an optical system of a stereo camera 1 according to the first embodiment.
The stereo camera 1 includes a camera module 10 including a lens 11 and a sensor 12, and a stereo adapter 13 including a prism P10 and a prism P20. In this embodiment, the sensor 12 refers to an image sensor. The plane T will be described later. α10, α20, β10, and β20 will also be described later.

[Comparative Example]
First, the problem in the comparative example will be described with reference to FIG.
FIG. 2 is a diagram showing the configuration of an optical system of a stereo camera 1z described in Patent Document 1 as a comparative example.
This configuration is characterized by using one prism P0 in addition to the lens 11, which is a light collecting unit that forms an image by the light transmitted through the prism P10 and the prism P20, and the sensor 12, which is a light receiving unit that detects the formed image. Note that the orientation of the prism P0 in FIG. 2 is opposite to that shown in the figure of Patent Document 1, but the result is the same. The light rays L00 to L02 and the light rays L10 to L12 propagated from the photographing object (hereinafter referred to as the object: not shown) are refracted at the surfaces S000, S010, and S011 of the prism P0. Thereafter, the light rays L00 to L02 and the light rays L10 to L12 are incident on the camera module 10 composed of the lens 11 and the sensor 12. Note that in this embodiment, the light rays L00 to L02 and L10 to L12 are collectively referred to as the light rays L as appropriate. At this time, for example, light ray L00 and light ray L10 traveling in the same direction are refracted at surface S000→surface S010 and surface S000→surface S011, respectively, to become two beams of light. Then, each of the two beams of light is incident on the camera module 10 at a different angle of incidence. Incidentally, reference symbol N1 denotes the optical axis of the lens 11.

In FIG. 2, the light ray L incident on the camera module 10, which is directed toward the prism P0, is shown by a dashed line. In other words, the light ray L incident on the prism P0 is shown by a dashed line. The approximate intersection of the dashed line is defined as the virtual viewpoint VP1 and the virtual viewpoint VP2. The virtual viewpoint VP1 and the virtual viewpoint VP2 indicate the effective position of the camera module 10. In other words, the virtual viewpoint VP1 and the virtual viewpoint VP2 are the locations where the two camera modules 10 are installed in a stereo camera that does not use the prism P0. In the configuration shown in FIG. 2, the light ray L is refracted at the surfaces S000 and S010 of the prism P0, and at the surfaces S010 and S011. The refracted light ray L is incident on the camera module 10. Therefore, in the camera module 10, it is the same as detecting the light from the virtual viewpoint VP1 and the virtual viewpoint VP2, respectively. That is, the sensor 12 of the camera module 10 detects images detected at the virtual viewpoints VP1 and VP2. Strictly speaking, the virtual viewpoints VP1 and VP2 correspond to the pupil positions of the lens 11 of the camera module 10 in a stereo camera 1 that does not use the prism P0. In the stereo camera 1z of this configuration (the configuration shown in FIG. 2), the distance between the virtual viewpoints VP1 and VP2 is the base length B. The plane T will be described later. The α20 and β20 in FIG. 2 will be described later.

FIG. 3 is a diagram showing an image on the sensor plane when a person is detected by the stereo camera 1z shown in FIG.
Here, it is assumed that light rays L from a person corresponding to light rays L00 and L10 in FIG. 2 are detected. In this case, on the sensor surface, images of a person are formed in two sensor areas 12A and 12B as viewed from virtual viewpoints VP1 and VP2. The sensor area 12A and the sensor area 12B respectively indicate areas obtained by dividing one sensor 12 into two. The sensor area 12A is a first light receiving area, and the sensor area 12B is a second light receiving area. An image is detected by forming an image on the sensor surface. Next, a calibration process such as distortion correction and brightness correction is performed on the image detected by the sensor 12. Then, the image is divided into two images detected by the sensor area 12A and the sensor area 12B, and the distance to the object can be measured by detecting the parallax between the two images. Next, two problems of the comparative example will be described.

The first problem is the effect of light dispersion. Since the glass material used in the prism P0 has a different refractive index for each wavelength (color), the incident light ray L is emitted at a different angle depending on the color. Then, the light incident on the lens 11 at different angles is incident on different positions on the sensor 12 surface depending on the color. Here, the refraction direction of the light ray L on the surfaces S000 and S010 of the prism P0 and the surfaces S000 and S011 are approximately opposite, so the directions in which the colors of the two images are separated are different. For example, in the case of blue, the refraction by the prism P0 is larger than that of other colors, so the blue components of the two images are incident on the center side of the sensor 12. This results in a small parallax being detected. On the other hand, in the case of red, the refraction by the prism P0 is smaller than that of other colors, so the red components of the two images are incident on the relatively outer side of the sensor 12. This results in a large parallax being detected. In this embodiment, such a phenomenon is called light dispersion. In other words, even if the distance to the same object is the same, different distance measurement results are output depending on the color. This is the first issue. Changing the distance measurement results depending on the color of the target object requires very complicated processing, so there is a need to address this with a simple configuration. In response to this, Patent Documents 2 and 3 use two prisms with different refractive indices that are in contact with each other to suppress the effects of light dispersion.

The second issue is to improve the accuracy of distance measurement. In the stereo camera 1z, the accuracy of distance measurement is determined depending on the focal length, the baseline length B, and the sensor pixel pitch. The higher the accuracy of distance measurement, the farther the object can be detected. Therefore, it also affects the distance measurement. In order to achieve high accuracy of distance measurement using the technology described in Patent Document 1, Patent Document 2, and Patent Document 3, there is an issue of the baseline length B. In order to increase the baseline length B using the technology described in Patent Document 1, Patent Document 2, and Patent Document 3, the size of the stereo camera 1z must be increased. In the case of FIG. 2, in order to increase the baseline length B, which is the distance between the virtual viewpoint VP1 and the virtual viewpoint VP2, it is necessary to increase the distance D between the camera module 10 and the prism P0. However, there is an issue that the size of the stereo camera 1z increases accordingly. This issue is not described in Patent Document 1, Patent Document 2, and Patent Document 3, and naturally there is no description of a solution to this issue.

In contrast, the stereo camera 1 of the first embodiment shown in FIG. 1 can solve the problems of the effects of light dispersion and the baseline length. This will be described in detail below with reference to FIG. 1. Note that the configuration of the camera module 10 is the same as that in FIG. 2, so the same reference numerals are used and the description will be omitted.

As shown in FIG. 1, the stereo camera 1 is composed of a prism P10 and a prism P20, and is characterized in that the two prisms P10 and P20 are spaced apart from each other by a predetermined distance d in the front-rear direction. In this embodiment, the incident side of the light ray L is the front stage, and the side of the sensor 12 is the rear stage. In other words, the prisms P10 and P20 are arranged in series with a distance d from the optical axis N1. The prism P10 is a first light ray angle conversion unit that changes the angle of the light ray L propagated from the object, and is the first prism. The prism P20 is a second light ray angle conversion unit that changes the angle of the light ray L propagated from the object, and is a second prism that is different from the prism P10. In this embodiment, the angle of the light ray L is the angle between the optical axis N1 and the traveling direction of the light ray L. Also, in this embodiment, when the term "angle" is used, the angle is the angle between the optical axis N1 and the traveling direction of the light ray L.

Prism P10 has surfaces S110, S111, and S100. Surface S100 can be divided into surface S100a, which faces surface S110, and surface S100b, which faces surface S111. Surfaces S110 and S100a form a first region, and surfaces S111 and S100b form a second region.

Prism P20 has surfaces S210, S211, and S200. Surface S200 can be divided into surface S200a that faces surface S210, and surface S200b that faces surface S211. Surfaces S210 and S200a form a third region, and surfaces S211 and S200b form a fourth region.

Light rays L00 to L02 propagating from the object are refracted at surface S110 of prism P10, then surface S100a, then surface S200a of prism P20, then surface S210. Similarly, light rays L10 to L12 are refracted at surface S111 of prism P10, then surface S100b, then surface S200b of prism P20, then surface S211. After that, light rays L00 to L02 and light rays L10 to L12 are incident on camera module 10, which is composed of lens 11 and sensor 12. In this way, light ray L propagating from the object propagates inside prism P10 and prism P20.

At this time, light rays L irradiated from the same direction, such as light rays L00 and L10, are refracted from surface S110 to surface S100a to surface S200a to surface S210, and from surface S111 to surface S100b to surface S200b to surface S211. Light rays L irradiated from the same direction, such as light rays L00 and L10, are incident on the camera module 10 at different angles of incidence.

Here, as in FIG. 2, of the light rays L incident on the camera module 10, the light rays L directed toward the prism P10 are extended and shown by dashed lines. The approximate intersection points of the dashed lines are defined as virtual viewpoints VP1 and VP2. These virtual viewpoints VP1 and VP2 indicate the effective positions of the camera module 10. As described above, each of the virtual viewpoints VP1 and VP2 is the position where two camera modules are located in a stereo camera that does not use a prism. This is because the light rays L are refracted by multiple surfaces before entering the camera module 10, and therefore the images detected by the sensor 12 are the images that would be detected at the virtual viewpoints VP1 and VP2, respectively.

As described above, the stereo camera 1 shown in the first embodiment is characterized in that the two prisms P10, P20 are arranged in series with respect to the optical axis N1. Furthermore, the two prisms P10, P20 are spaced apart in the front-to-rear direction by a distance d. With this configuration, compared to a stereo camera that does not use prisms, the camera module 10 can be reduced to one, and the number of parts in the stereo camera 1 can be reduced. This makes it possible to reduce the size and cost of the stereo camera 1.

The two prisms P10 and P20 have different refractive indices and refraction angles. The two prisms P10 and P20 have two major effects. The first effect is dispersion compensation of light. This effect is the same as that of Patent Documents 2 and 3. By optimizing the refractive indices and refraction angles of the two prisms P10 and P20, the effect of dispersion of light can be reduced. The refraction angle is the angle between the light ray L incident on the prisms P10 and P20 and the light ray L emitted from them, and is the angle of the light ray L changed in the prisms P10 and P20.

The second effect is the improvement of the accuracy of distance measurement. In the stereo camera 1 shown in the first embodiment, the light ray L from the measurement object is largely refracted by the prism P10. In other words, the refractive index of the prism P10 is larger than that of the prism P0 shown in FIG. 2. In addition, the prism P10 refracts the light incident on the prism P10 in the direction where the light refracted by the surfaces S110 and S100a and the surfaces S111 and S100b intersect. Similarly, the prism P20 refracts the light incident on the prism P20 in the direction where the light refracted by the surfaces S210 and S200a and the surfaces S211 and S200b intersect. As a result, even in the configuration shown in FIG. 1, the sensor 12 displays an image as shown in FIG. 3.

In other words, as shown in FIG. 1, the direction in which the angle of the incident light ray L is changed is substantially opposite between the surfaces S110 and S100a of the prism P10 and between the surfaces S111 and S100b. Furthermore, the direction in which the angle of the incident light ray L is changed is substantially opposite between the surfaces S210 and S200a of the prism P20 and between the surfaces S211 and S200b.

Then, the light ray L is refracted by the prism P20 in the opposite direction to the prism P10. That is, the light is refracted in the opposite directions between the prism P10 and the prism P20. That is, the angle of the light ray L incident on the surface S110 and the surface S100a of the prism P10 and the light ray L incident on the surface S200a and the surface S210 of the prism P20 is changed in the opposite direction. Similarly, the angle of the light ray L incident on the surface S111 and the surface S100b of the prism P10 and the light ray L incident on the surface S200b and the surface S211 of the prism P20 is changed in the opposite direction. Note that in FIG. 1, the prisms P10 and P20 are shown in cross section and are line-symmetrical with respect to the optical axis N1 (in reality, the prisms P10 and P20 have a shape line-symmetrical with respect to the optical axis N1). The same is true in the figures from FIG. 2 onwards.

Furthermore, in prism P10, the angle of the incident light ray L is changed so that it is directed in the direction of optical axis N1, and in prism P20, the angle of the incident light ray L is changed so that it is directed in the opposite direction to prism P10.

In addition, light ray L transmitted through surfaces S110, S100a of prism P10 and surfaces S200a, S210 of prism P20 is incident on sensor area 12A of sensor 12 (see FIG. 3) by lens 11. Similarly, light ray L transmitted through surfaces S100b, S111 and surfaces S200b, S211 of prism P10 is incident on sensor area 12B of sensor 12 (see FIG. 3) by lens 11.

By using this configuration, it is possible to increase the baseline length B between the virtual viewpoints VP1 and VP2 without increasing the distance d between the two prisms P10 and P20. In other words, as is clear from a comparison between FIG. 1 and FIG. 2, the stereo camera 1 increases the refractive index of the prism P10. In this way, it is possible to increase the distance between the virtual viewpoints VP1 and VP2 (i.e., the baseline length B) compared to the comparative example. However, simply increasing the refractive index of the prism P0 in the configuration of FIG. 2 would result in fewer images captured by the sensor 12 that can be used for distance measurement. This issue will be discussed later with reference to FIG. 12 to FIG. 15.

Note that of the two prisms P10 and P20 used in the first embodiment, prism P10 has a larger refractive index and prism angle than prism P20. This configuration can provide the above-mentioned effects. The advantages of providing prism P20 will be described later. It is also possible to place prism P20 closer to camera module 10. For example, prism P20 can be combined with lens 11. This can reduce the material cost of the prism and make it smaller.

4A and 4B are diagrams showing the relationship between the camera module incidence angle and the stereo camera incidence angle in the configurations shown in FIGS.
FIG. 4A shows the camera module incidence angle and the stereo camera incidence angle according to this embodiment. FIG. 4B shows the camera module incidence angle and the stereo camera incidence angle according to the configuration of the comparative example shown in FIG. 2. In FIG. 4A and FIG. 4B, the horizontal axis shows the camera module incidence angle, which is the incidence angle of the light ray L to the camera module 10, and the vertical axis shows the stereo camera incidence angle, which is the incidence angle to the stereo cameras 1 and 1z. The stereo camera incidence angle is the angle of the light ray L incident on the surfaces S110 and S111 of the prism P10. The stereo camera incidence angle indicates the incidence angle of the light ray L to the plane T (see FIG. 1 and FIG. 2) perpendicular to the optical axis N1. Note that, in this example, the camera module 10 with a camera module incidence angle width of approximately 40 degrees is used, and the constraint is to configure the stereo cameras 1 and 1z with an incidence angle width of ±20 degrees on each of the left and right.

The conditions of the prisms P0, P10, and P20 and the camera module 10 when the results shown in Figures 4A and 4B were measured are shown below. Note that the prism angles α10, β10, α20, and β20 below are shown in Figure 1, and the prism angles α0 and β0 are shown in Figure 2.

Prisms P10, P20, P0
(FIG. 4A) FIG. 1 configuration (first embodiment)
Prism P10, prism angle α10 = 44 degrees, prism angle β10 = 90 degrees, glass material: BK7
Prism P20: Prism angle α20 = 25.3 degrees, Prism angle β20 = 90 degrees, Glass material: BaF10

(FIG. 4B) FIG. 2 configuration (Patent Document 1)
Prism P0 Prism angle α0 = 18 degrees, prism angle β0 = 90 degrees, glass material: BK7

Camera module 10 (common to FIGS. 4A and 4B )
Lens 11: focal length 6.0 mm, central projection Sensor 12: pixel pitch 2.25 μm

The prism angle α10 is the angle of the inclined portion (surfaces S110, S111, S100) that constitutes the prism P10, which has the smallest inclination with respect to the optical axis N1. The prism angle α21 in FIG. 1 is 90 degrees minus α20. The prism angle α21 is the prism angle of the inclined portion (surfaces S210, S211, S200) that constitutes the prism P20, which has the smallest inclination with respect to the optical axis N1. Thus, the prism angle α10 is smaller than the prism angle α21. As a result, as described above, the refractive index of the prism P10 is greater than that of the prism P20. By making the refractive index of the prism P10 greater than that of the prism P20, the light ray L refracted by the prism P10 toward the optical axis N1 enters the camera module 10 without being scattered by the prism P20.

The camera module incidence angle of 0 degrees corresponds to the center of the sensor 12. The camera module incidence angle of -20 degrees to 0 degrees corresponds to the sensor area 12A shown in FIG. 3. The camera module incidence angle of 0 degrees to +20 degrees corresponds to the sensor area 12B shown in FIG. 3.

In FIG. 4B, it can be seen that the camera module incidence angles are -20 degrees to 0 degrees and 0 degrees to +20 degrees, and the stereo camera incidence angles are approximately -10 degrees to +10 degrees. Therefore, it can be seen that parallax can be detected by stereo processing images with camera module incidence angles of -20 degrees to 0 degrees and images with 0 degrees to +20 degrees. Also, in FIG. 4A, it can be seen that the camera module incidence angles are -17 degrees to 0 degrees and 0 degrees to +17 degrees, and the stereo camera incidence angles are approximately -10 degrees to +10 degrees. Therefore, it can be seen that parallax can be detected by stereo processing images with camera module incidence angles of -17 degrees to 0 degrees and images with 0 degrees to +17 degrees.

Next, the effect on light dispersion will be described with reference to Figures 5A and 5B.
FIG. 5A is a diagram showing the dispersion of light by the configuration shown in the first embodiment. FIG. 5B is a diagram showing the dispersion of light obtained by the configuration of the comparative example shown in FIG. 2. Here, the horizontal axis shows the camera module incident angle, and the vertical axis shows the difference in the sensor incident position between blue (435.8 nm) and red (656.3 nm). The difference in the sensor incident position between blue and red is the difference between the incident position (sensor incident position) on the sensor 12 for blue and the incident position on the sensor 12 for red. The larger this value is, the greater the degree of dispersion of light is. Note that the configurations of the prisms P0, P10, and P20 and the camera module 10 when obtaining the results of FIG. 5A and FIG. 5B are the same as those of FIG. 4A and FIG. 4B.

In FIG. 5B, which shows the results of the comparative example, it can be seen that the difference in the sensor incident position is reversed for the camera module incident angle from -20 degrees to 0 degrees and from 0 degrees to +20 degrees. This is because, as described above, the refraction direction of the light ray L incident on the surface S000 → surface S010 of the prism P0 and the surface S000 → surface S011 are approximately opposite, so the directions in which the colors of the two images are separated are different. For this reason, in the configuration of the comparative example, different distance measurement results are output depending on the color even though the object is the same. In the results shown in FIG. 5B, the difference in the sensor incident position is ±10 pixels when the camera module incident angle is from -20 degrees to 0 degrees and from 0 degrees to +20 degrees. In other words, a difference of 20 pixels occurs when the camera module incident angle is from -20 degrees to 0 degrees and when it is from 0 degrees to +20 degrees. As a result, for example, a difference in parallax of 20 pixels occurs between the case where a blue object is detected and the case where a red object is detected. A normal stereo camera detects parallax with an accuracy of one pixel or less. For this reason, a difference in parallax of 20 pixels will cause a large distance measurement error. Note that a normal stereo camera is one that does not use a prism.

In contrast, in the case of the configuration shown in this embodiment in FIG. 5A, it can be seen that the difference in the sensor incident position is small compared to the configuration of the comparative example shown in FIG. 5B. Also, the minute differences in the sensor incident position occurring in each of the plot groups on the left and right sides of the paper are approximately symmetrical (separation directions are the same) with the camera module incident angle of ±10 degrees as the center. As described above, the camera module incident angle of -17 degrees to 0 degrees corresponds to the sensor area 12A, and the camera module incident angle of -0 degrees to +17 degrees corresponds to the sensor area 12B. That is, the horizontal axis of FIG. 5A and FIG. 5B corresponds to the direction in which the sensor area 12A and the sensor area 12B are lined up. That is, in the same direction with respect to the direction in which the sensor area 12A and the sensor area 12B are lined up, the separation direction when red and blue light are incident is the same for the sensor area 12A and the sensor area 12B. Incidentally, the plot group on the left side of the paper in FIG. 5A shows the difference in the sensor incident position in the image of the sensor area 12A. Also, the plot group on the right side of the paper shows the difference in the sensor incident position in the sensor area 12B.

With reference to FIG. 4A, consider the case where an object is detected in the direction of a stereo camera incidence angle of +10 degrees. In this case, the image processing unit 210 (see FIG. 8) only needs to look at 0 degrees and +17 degrees for the camera module incidence angle on the horizontal axis of FIG. 4A. That is, in FIG. 4A, the stereo camera incidence angle of +10 degrees corresponds to 0 degrees and +17 degrees in terms of the camera module incidence angle. Therefore, the light ray L coming from the direction of the stereo camera incidence angle of +10 degrees is an image corresponding to the camera module incidence angles of 0 degrees and +17 degrees. Then, with reference to FIG. 5A, the plot with the camera module incidence angle of 0 degrees (the upper plot of the two plots) and the plot with +17 degrees have approximately the same value. In other words, in the image of the sensor area 12A and the image of the sensor area 12B, the images due to the light ray L incident at the same stereo incidence angle have approximately the same color dispersion. Therefore, when performing parallax calculation, cancellation processing can be performed according to the difference in the sensor incidence position corresponding to each camera module incidence angle. That is, as shown in FIG. 5A, the difference in the sensor incident position of the two images on the sensor 12 occurs in the same direction, so it is possible to cancel color dispersion by parallax calculation. This is because the same tendency of difference in the sensor incident position exists even when the stereo camera incident angle shown in FIG. 4A is from -10 degrees to +10 degrees. The stereo camera incident angle shown in FIG. 4A from -10 degrees to +10 degrees corresponds to the camera module incident angle on the horizontal axis of FIG. 5A from -17 degrees to 0 degrees and 0 degrees to +17 degrees. Therefore, color dispersion can be canceled by similar processing even at these stereo incident angles.

In contrast, according to the results of the comparative example shown in Figure 5B, the values are inverted when the camera module incidence angle is from -20 degrees to 0 degrees and from 0 degrees to +20 degrees, making it difficult to cancel color dispersion in the parallax calculation.

As described above, the configuration of this embodiment makes it possible to reduce the absolute value of the difference in the sensor incident position for each wavelength with respect to the dispersion of light. Therefore, this embodiment enables highly accurate distance measurement. Furthermore, with the configuration of this embodiment, as shown in FIG. 5B, it is possible to generate differences in the sensor incident position in the same direction with respect to the stereo camera incident angle. In this way, highly accurate distance measurement is possible. This indicates that the separation direction when red and blue light are incident in the same direction as the arrangement direction on the sensor 12 of the two images is the same in the two images. This can be achieved by optimizing the refractive index and prism angle of the prisms P10 and P20.

FIG. 6 is a diagram showing the relationship between the module distance and the base line length B (see FIGS. 1 and 2) in the configuration of the first embodiment (the configuration of FIG. 1) and the configuration of the comparative example shown in FIG.
The module distance is the distance from the prism surface on the object side to the camera module 10 .
In Fig. 6, symbol C1 indicates the results for the configuration of the first embodiment (the configuration of Fig. 1), and symbol C2 indicates the results for the configuration of the comparative example (the configuration of Fig. 2). In Fig. 6, the horizontal axis indicates the baseline length B, and the vertical axis indicates the module distance. From Fig. 6, it can be seen that in the comparative example (symbol C2), the module distance increases as the baseline length B increases. In other words, it can be seen that the stereo camera 1 becomes larger as the baseline length B increases. In contrast, in the configuration of this embodiment (symbol C1), the change in module distance relative to the change in baseline length B is smaller than that of the comparative example (symbol C2). In other words, in this embodiment, color dispersion can be suppressed, thereby improving distance measurement accuracy.

In the case of the configuration of the first embodiment (stereo camera 1 in FIG. 1), by increasing the refraction angle of the prism P10, it is possible to achieve a smaller size than the comparative example (stereo camera 1z in FIG. 2). On the other hand, in order to increase the refraction angle of the prism P10, it is necessary to increase the prism angle of the entrance surface and the exit surface of the prism P10. As a result, there is a characteristic that the light ray L incident on the prism P10 is shifted from the virtual viewpoint VP1 and the virtual viewpoint VP2 depending on the stereo camera entrance angle. This will be explained with reference to FIG. 7.

FIG. 7 is a diagram showing the base line length B versus the stereo camera incident angle.
Here, the horizontal axis indicates the stereo camera incidence angle, and the vertical axis indicates the base line length B. Note that FIG. 7 shows the results of the stereo camera 1 shown in FIG. 1, and the configurations of the prisms P10, P20, and the camera module 10 are the same as those in FIG. 4A. According to FIG. 7, it can be seen that the base line length B of ±10 degrees is larger than the base line length B when the stereo camera incidence angle is 0 degrees. According to FIG. 7, the base line length appears to change depending on the incidence angle. In contrast, in the first embodiment, the base line length B is changed depending on the stereo camera incidence angle and the position of the object on the sensor 12 to calculate the distance to the object. This allows for more accurate distance measurement.

(Stereo Camera System Z)
A processing method of the stereo camera system Z will be described with reference to FIGS. 8A and 8B.
FIG. 8A is a diagram showing the configuration of the stereo camera system Z, and FIG. 8B is a diagram showing the detailed configuration of the stereo parallax image generating unit 220. As shown in FIG.
The stereo camera system Z includes a stereo camera 1 and an image processing device 2 that is a stereo image processing device, and detects surrounding three-dimensional objects based on an image M0 obtained by a camera module 10.

First, refer to FIG. 8A.
The image processing device 2 is configured to include, for example, an image processing unit 210, a stereo parallax image generating unit 220, and a stereoscopic object detecting unit 230.

The image processing unit 210 applies a predetermined image processing to the image M1 and the image M2, and supplies the images to the stereo parallax image generating unit 220. The image processing unit 210 is configured with an image separating unit 211, affine processing units 212a and 212b, luminance correction units 213a and 213b, pixel interpolation units 214a and 214b, and luminance information generating units 215a and 215b.

The stereo camera 1 detects an image M0 of the sensor 12 in the camera module 10 as shown in FIG. 3. Then, the image M0 is separated by the image separation unit 211 into two images M1 (first detected image) and M2 (second detected image) according to the virtual viewpoints VP1 and VP2. For example, the image M1 is an image detected by the sensor area 12A in FIG. 3, and the image M2 is an image detected by the sensor area 12B in FIG. 3. In other words, the image M1 is the output signal of the first light receiving area (sensor area 12A), and the image M2 is the output signal of the output signal of the second light receiving area (sensor area 12B).

The affine processing unit 212a applies affine processing to image M1. The affine processing is, for example, a linear coordinate transformation process, but may also include non-linear calculations. As a result of performing this affine processing, the affine processing unit 212a outputs image M3 (third image). Similarly, the affine processing unit 212b applies affine processing to image M2 and outputs image M4 (fourth image).

The luminance correction unit 213a corrects the luminance of each pixel in image M3. For example, the luminance of each pixel in image M3 is corrected based on the difference in gain of each pixel in image M3. Similarly, the luminance correction unit 213b corrects the luminance of each pixel in image M4.

The pixel interpolation unit 214a performs demosaicing processing on image M3. For example, a raw image (Bayer array image) is converted into a color image. Similarly, the pixel interpolation unit 214b performs demosaicing processing on image M4.

The luminance information generating unit 215a generates luminance information for image M3. For example, the luminance information generating unit 215a converts information representing a color image into luminance information for generating a parallax image. Similarly, the luminance information generating unit 215b generates luminance information for image M4. The generated images M3 and N4 are input to the stereo parallax image generating unit 220.

The stereo disparity image generating unit 220 generates a stereo disparity image using the obtained images M3 and M4.

The details of the stereo parallax image generating unit 220 are shown in FIG. 8B.
As shown in FIG. 8B, the stereo parallax image generating unit 220 includes an exposure adjustment unit 221, a sensitivity adjustment unit 222, a geometric correction unit 223, a matching unit 224, a noise removal unit 225, and a pixel shift amount calculation unit 226. Each of the exposure adjustment unit 221 and the sensitivity adjustment unit 222 executes feedback control to the camera module 10 regarding the exposure amount, sensitivity, and the like of the camera module 10. In addition, the geometric correction unit 223 performs geometric correction of the two images M3 and M4, and the matching unit 224 performs matching processing of the left and right images (images M3 and M4). In addition, the noise removal unit 225 removes noise by using a smoothing process or the like, and the pixel shift amount calculation unit 226 calculates the pixel shift amount.

Returning to the explanation of FIG. 8A.
The stereoscopic three-dimensional object detection unit 230 detects a three-dimensional object in the stereoscopic region according to the stereoscopic parallax image generated by the stereoscopic parallax image generation unit 220. At this time, the stereoscopic three-dimensional object detection unit 230 changes the base line length B (see FIG. 1) according to the stereoscopic camera incident angle, as shown in FIG. 7. The stereoscopic three-dimensional object detection unit 230 can also measure the distance of the three-dimensional object based on the changed base line length B. In other words, the stereoscopic three-dimensional object detection unit 230 also applies stereo matching to the detected three-dimensional object to detect parallax and identify the type of the three-dimensional object (pedestrian, bicycle, vehicle, building, etc.). In this way, the image processing device 2 detects parallax based on the output signals (images M1 and M2) of the sensor area 12A and the sensor area 12B.

(Hardware configuration)
FIG. 9 is a diagram showing the hardware configuration of the image processing device 2. As shown in FIG.
The image processing device 2 includes a memory 21, and a calculation device 22 including a central processing unit (CPU), a graphic processing unit (GPU), etc. The image processing device 2 also includes a storage device 23 such as a hard disk (HD) or a solid state drive (SSD). The image processing device 2 also includes an input device 24, an output device 25, and a communication device 26 that communicates with the camera module 10.

A program is stored in the storage device 23, and this program is loaded into the memory 21. The loaded program is then executed by the calculation device 22. This embodies the units 210, 211 to 215b, 220, 221 to 226, and 230 shown in Figures 8A and 8B.

By using the optical system and processing method described above, the stereo camera 1 can be realized with only one camera module 10 and two prisms. This solves the problem of small size and low cost.

Here, the distance detection method in the stereo camera 1 shown in the first embodiment does not depend in principle on the projection method of the lens 11. For example, it may be orthogonal projection (f sin θ), equidistant projection (f θ), equistereographic projection (2f sin(θ/2)), or stereoscopic projection (2f tan(θ/2)), which are projections of a fisheye lens. Note that f indicates the focal length of the lens 11, and θ is the angle of incidence on the camera module.

1, the prism angles β10 and β20 of the surface S100 of the prism P10 and the surface S200 of the prism P20 are set to 90 degrees. However, the present invention is not limited to this and any prism angle may be used.
FIG. 10 is a diagram showing a modified example of the stereo camera 1 according to the first embodiment.
For example, a stereo camera 1a may be configured using prisms P10A and P20A as shown in Fig. 10. In Fig. 10, the stereo adapter 13a is configured by prisms P10A and P20A. In this case, the prism P10A can be considered to include a prism region P10B and a prism region P10C. The prism region P10B is a first region and includes a surface S110 and a surface S100B. The prism region P10C is a second region and includes a surface S111 and a surface S100C.

Similarly, prism P20A can be considered to include prism region P20B and prism region P20C. Prism region P20B is a third region and includes surfaces S200B and S210. Prism region P20C is a fourth region and includes surfaces S200C and S211.

In this embodiment, the prisms P10 and P20 are described as being line-symmetrical with respect to the optical axis N1, but this is not limited thereto, and the prism shapes may be different above and below the paper. In this way, the detection direction of the one-sided viewpoint can be intentionally shifted. Furthermore, the prisms P10 and P20 in FIG. 1 may be shifted in the vertical direction with respect to the paper with respect to the camera module 10. In other words, the prisms P10 and P20 may be arranged in a state where they are shifted in the vertical direction with respect to the paper, with a distance d from the optical axis N1 (in the horizontal direction of the paper). In this way, the detection direction of the one-sided viewpoint can be intentionally shifted.

11A and 11B are diagrams showing examples of the arrangement of the sensor 12. FIG.
In the first embodiment, the horizontally elongated sensor 12 is arranged so that the horizontal direction is the longitudinal direction, but the sensor 12 may be arranged so that the vertical direction is the longitudinal direction as shown in FIG. 11A. That is, in the example shown in FIG. 11A, the longitudinal direction of the sensor 12 is perpendicular to the direction in which the virtual viewpoints VP1 and VP2 are arranged. In this case, there is an advantage that the stereo camera incidence angle in the vertical direction (perpendicular to the ground) can be made wider. In addition, the stereo camera 1 having the sensor 12 shown in FIG. 11A may be rotated, for example, by 90 degrees with respect to the depth axis, with the prisms P10 and P20 and the lens 11 shown in FIG. 1. In that case, an image as shown in FIG. 11B is obtained. Furthermore, in the stereo camera 1 shown in FIG. 1 having the sensor 12 arranged so that the horizontal direction is the longitudinal direction, the camera module 10 is not rotated, and only the prisms P10 and P20 can be rotated by 90 degrees. In this case, an image as shown in FIG. 11C is obtained.

In the cases of Fig. 3 and Fig. 11A, parallax is detected in the horizontal direction (horizontal with respect to the ground), but in the cases of Fig. 11B and Fig. 11C, parallax in the vertical direction can be detected. Furthermore, Fig. 11B has the advantage that the incidence angle of the stereo camera in the vertical direction can be made wider, just like Fig. 11A. Also, Fig. 11C has the advantage that the incidence angle of the stereo camera in the horizontal direction can be made wider.

The effect of the prism P20 will now be further described with reference to FIGS.
FIG. 12 is a diagram showing optical paths when the refractive index of the prism P10 of the comparative example is increased, and FIG. 13 is a diagram showing an example of an image formed on the sensor 12 by the configuration shown in FIG.
The stereo camera 1z1 shown in Fig. 12 is an example in which the refractive index of the prism P10 used in Patent Document 1 is increased in the configuration described in Patent Document 1. Note that the orientation of the prism P10 is different from that of the prism P0 shown in Fig. 2, but the result is the same.
In addition, in Fig. 12 and Fig. 13 described later, the refraction of the optical path is exaggerated for ease of understanding. In Fig. 12 and Fig. 13, the same components as those in Fig. 1 are denoted by the same reference numerals and the description thereof will be omitted.
12, a light ray L32 incident on the prism P10 from near the optical axis N1 is incident on a surface S110, and then refracts to reach the sensor area 12B. Similarly, a light ray L31 incident on the prism P10 from near the optical axis N1 is incident on a surface S111, and then refracts to reach the sensor area 12A.

If light ray L32 incident on surface S110 is not refracted by prism P10, light ray L32 will travel along the optical path shown by the dashed line and reach virtual viewpoint VP1. Similarly, if light ray L31 incident on surface S111 is not refracted by prism P10, light ray L31 will travel along the optical path shown by the dashed line and reach virtual viewpoint VP2. It is desirable that light ray L32 virtually detected at virtual viewpoint VP1 be detected by sensor area 12A. Similarly, it is desirable that light ray L31 virtually detected at virtual viewpoint VP2 be detected by sensor area 12B.

However, in the configuration shown in FIG. 12, light ray L32 incident on surface S110 reaches sensor area 12B, and light ray L31 incident on surface S111 reaches sensor area 12A. The area 121 indicated by dots in FIG. 13 is an image area formed by light that thus reaches sensor area 12B after incident on surface S110, or reaches sensor area 12A after incident on surface S111.

The image of the image on the surface of the sensor 12 in region 121 in Figure 13 cannot be used to detect the position of an object. Therefore, simply increasing the refractive index of the prism P10 narrows the range of the image on the surface of the sensor 12 that can be used to detect the position of an object.

FIG. 14 is a diagram showing optical paths of the stereo camera 1 according to the present embodiment shown in FIG. 1, and FIG. 15 is a diagram showing an example of an image formed on the sensor 12 by the configuration shown in FIG.
In the stereo camera 1 shown in FIG. 14, a prism P20 that refracts light in the opposite direction to the prism P10 is provided behind the prism P10. In this way, the light ray L32 incident on the surface S110 reaches the sensor area 12A by refraction by the prism P20. Similarly, the light ray L31 incident on the surface S111 reaches the sensor area 12B by refraction by the prism P20. This makes it possible to prevent the area 121 in FIG. 13 from being detected, as in the image on the surface of the sensor 12 shown in FIG. 15. Therefore, according to the configuration shown in FIG. 14, the range of images that can be used to detect the position of an object can be expanded more than the configuration shown in FIG. 12.

In addition, according to this embodiment, prisms P10 and P20 are spaced apart in series by a distance d in the direction of optical axis N1, which makes it possible to suppress chromatic dispersion and improve distance measurement accuracy.

[Second embodiment]
FIG. 16 is a diagram showing the configuration of an optical system of a stereo camera 1b according to the second embodiment.
As shown in Fig. 16, the stereo camera 1b includes prisms P10D and P10E, which are obtained by spatially separating the prism P10A shown in Fig. 10 into two prisms, and the same effect as that of the first embodiment can be obtained. In Fig. 16, the stereo adapter 13b is composed of the prisms P10D, P10E, and the prism P20A. The prism P10D shown in Fig. 16 includes the surface S110 and the surface S100B, and the prism P10E includes the surface S111 and the surface S100C. In Fig. 16, the prism P10D including the surface S110 and the surface S100B is the first region. And the prism P10E including the surface S111 and the surface S100C is the second region. And the prism P10D and the prism P10E are spatially separated.

By doing this, only the area through which the light ray L passes can be the prisms P10D and P10E, preventing the prisms P10D and P10E from becoming larger. In other words, by using the prisms P10D and P10E as shown in FIG. 16, it is possible to eliminate the prism portion corresponding to the portion through which the light ray L that does not contribute to the detection of an image in the sensor 12 passes. The portion through which the light ray L that does not contribute to the detection of an image in the sensor 12 passes is the light ray L that passes near the optical axis N1. This makes it possible to avoid an increase in cost and an increase in the size of the stereo camera. In other words, as in FIG. 1, the unnecessary prism portion is eliminated while preventing the stereo camera size from increasing, thereby reducing costs.

FIG. 17 is a diagram showing another example of the prism P20 in the second embodiment.
Also, as shown in FIG. 17, the prism P20A shown in FIG. 10 may be spatially separated into two. In this case, prisms P20D and P20E are present as shown in FIG. 17. The prism P20D includes the surface S200B and the surface S210, and the prism P20E includes the surface S200C and the surface S211. In FIG. 17, the prism P20D including the surface S200B and the surface S210 is the third region, and the prism P20E including the surface S200C and the surface S211 is the fourth region. The prisms P20D and P20E are spatially separated. The prisms P20D and P20E shown in FIG. 17 can achieve the same effect as the prisms P10D and P10E shown in FIG. 16.

It is sufficient that either one of the prisms P10A and P20A shown in FIG. 10 is spatially separated. In other words, it is sufficient that either one of the set of prisms P10D and P10E in FIG. 16 and the set of prisms P20D and P20E in FIG. 17 is arranged.

Furthermore, the prisms P10D and P10E shown in FIG. 16 are configured by spatially separating the prism P10A shown in FIG. 10. However, this is not limited to the above, and for example, the prism P10 shown in FIG. 1 may be configured by spatially separating it. Furthermore, the prisms P20D and P20E shown in FIG. 17 are configured by spatially separating the prism P20A shown in FIG. 10. However, this is not limited to the above, and for example, the prism P20 shown in FIG. 1 may be configured by spatially separating it.

Furthermore, the glass material of prisms P10 and P20 is not limited.

[Third embodiment]
18 is a diagram showing the configuration of an optical system of a stereo camera 1c according to a third embodiment of the present invention. In the stereo camera 1c shown in FIG. 18, the difference from the first embodiment is that the prism P10 and the prism P20 shown in FIG. 1 are replaced with a split diffraction grating G10 and a split diffraction grating G20. Since the other configurations are the same as those in FIG. 1, the same reference numerals as in FIG. 1 are used and the description is omitted. The stereo adapter 13c is composed of a split diffraction grating G10 and a split diffraction grating G20. The light beam L incident on the split diffraction gratings G10 and G20 passes through the split diffraction gratings G10 and G20.

The stereo camera 1c has a split diffraction grating G10 having at least regions R60 and R61, and a split diffraction grating G20 which is a diffraction element having at least regions R70 and R71. The region R60 corresponds to the first region, the region R61 corresponds to the second region. The region R70 corresponds to the third region, and the region R31 corresponds to the fourth region. The stereo camera 1c is characterized in that the two split diffraction gratings G10 and G20 are spaced apart by a distance d in the front-rear direction. The split diffraction grating G10 has a diffraction region in which light is diffracted divided into regions 60 and 61. Similarly, the split diffraction grating G20 has a diffraction region in which light is diffracted divided into regions 70 and 71. The white parts of the split diffraction gratings G10 and G20 are made of a material that transmits light, such as glass or quartz.

The light rays L00 to L02 propagating from the object are diffracted in the region R60 of the split diffraction grating G10 and the region R70 of the split diffraction grating G20. The light rays L00 to L02 then enter the camera module 10, which is composed of the lens 11 and the sensor 12. Similarly, the light rays L10 to L12 propagating from the object are diffracted in the region R61 of the split diffraction grating G10 and the region R71 of the split diffraction grating G20. The light rays L10 to L12 then enter the camera module 10, which is composed of the lens 11 and the sensor 12. At this time, for example, the light rays L60 and L61 from the same object are diffracted from the region R60 to the region R70 and from the region R61 to the region R71. The diffracted light rays L then enter the camera module 10 at different angles of incidence.

Here, region 60 exhibits optical characteristics similar to those of surfaces S110 and S100a shown in FIG. 1, and region R61 exhibits optical characteristics similar to those of surfaces S111 and S100b shown in FIG. 1. Similarly, region 70 exhibits optical characteristics similar to those of surfaces S210 and S200a shown in FIG. 1, and region 71 exhibits optical characteristics similar to those of surfaces S211 and S200b shown in FIG. 1.

Here, as in FIG. 1, of the light rays L incident on the camera module 10, the extended lines of the light rays L heading toward the divided diffraction grating G10 are shown as dashed lines. The approximate intersecting points of the dashed lines are defined as virtual viewpoints VP1 and VP2. These virtual viewpoints VP1 and VP2 indicate the effective positions of the camera module 10. Because the light rays L are diffracted by multiple surfaces, the images detected by the sensor 12 are similar to the images detected at the virtual viewpoints VP1 and VP2, respectively.

As described above, the second embodiment is characterized in that the two split diffraction gratings G10, G20 are spaced apart by a distance d in the front-to-back direction. These two split diffraction gratings G10, G20 are characterized in that the diffraction angle of split diffraction grating G10 is greater than the diffraction angle of split diffraction grating G20. With this configuration, for the same reasons as in the first embodiment, miniaturization can be achieved even if the baseline length B is increased.

FIG. 19 is a diagram showing another example of the stereo camera 1c according to the second embodiment.
The split diffraction gratings G10 and G20 used in FIG. 18 may be diffraction elements such as holographic elements. In addition, in the configuration of this embodiment, the first element (prism P10, split diffraction grating G10) bends the light ray L toward the center. In addition, the second element (prism P20, split diffraction grating G20) bends the light ray L to the opposite side of the first element. Due to this configuration, for example, a stereo camera 1d may be configured in which the first element is a split diffraction grating G10 and the second element is a prism P20, as shown in FIG. 19. The stereo adapter 13d is configured by the split diffraction grating G10 and the prism P20. In addition, the surfaces of the prisms P10 and P20 shown in FIG. 1 may be formed into a grating structure. By doing so, there is an advantage that the dispersion due to color can be reduced. In addition, as in the first embodiment, in the second embodiment, the arrangement direction of the sensor 12 is not limited. The longitudinal direction of the sensor 12 may be the direction in which the virtual viewpoint VP1 and the virtual viewpoint VP2 are arranged. 11A and 11B, the longitudinal direction of the sensor 12 may be perpendicular to the direction in which the virtual viewpoints VP1 and VP2 are arranged, or may be oblique to the direction in which the virtual viewpoints VP1 and VP2 are arranged. However, it is necessary that the light ray L incident on the sensor 12 is perpendicular to the imaging surface of the sensor 12.

In FIG. 19, only the prism P10 in the configuration of FIG. 1 may be replaced with a split diffraction grating G10, and only the prism P20 in the configuration of FIG. 1 may be replaced with a split diffraction grating G20. In other words, at least one of the prisms P10 and P20 in the configuration of FIG. 1 may be a split diffraction grating G10, G20.

<Modification>
The present invention is not limited to the above-described embodiment, and includes various modified examples other than those described above. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

The camera module 10 in this embodiment may be any type. For example, a mobile phone camera, a computer camera, or the like may be used as the camera module 10. In such a case, for example, the prisms P10 and P20 in FIG. 1 may be added to the mobile phone camera, computer camera, or the like.

In addition, the above-mentioned configurations, functions, each unit 210, 211 to 215b, 220, 221 to 226, 230, storage device 23, etc. may be realized in hardware by designing some or all of them as an integrated circuit, for example. In addition, as shown in FIG. 9, the above-mentioned configurations, functions, etc. may be realized in software by a calculation device 22 such as a CPU interpreting and executing a program that realizes each function. Information such as a program, table, file, etc. that realizes each function can be stored in a memory 21, a recording device such as an SSD, or a recording medium such as an IC (Integrated Circuit) card, an SD (Secure Digital) card, or a DVD (Digital Versatile Disc), in addition to being stored in a HD.
In addition, in each embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other.

1, 1a to 1d, 1z, 1z1 Stereo camera 2 Image processing device (stereo image processing device)
10 camera module 11 lens (light collecting part)
12 Sensor (light receiving part)
12A Sensor area (first light receiving area)
12B Sensor area (second light receiving area)
13, 13a to 13d Stereo adapter 220 Stereo parallax image generating unit 230 Stereo vision three-dimensional object detecting unit d Distance (predetermined distance)
G10, G20 split diffraction grating (diffraction element)
R60 Region (First Region)
R61 Region (Second Region)
R70 Region (Third Region)
R71 region (fourth region)
L, L00 to L02, L10 to L12, L31, L32: Light ray N1: Optical axis M0: Image M1: Image (output signal of the first light receiving region)
M2 Image (output signal of the second light receiving region)
P0 Prism P10 Prism (first light ray angle conversion unit)
P10A Prism P10B Prism area (first area)
P10C Prism region (second region)
P10D Prism (first light ray angle conversion part, first region)
P10E Prism (first light ray angle conversion part, second region)
P20 Prism (second light ray angle conversion unit)
P20A Prism P20B Prism area (third area)
P20C Prism area (fourth area)
P20D Prism (second light ray angle conversion section, third region)
P20E Prism (Second light ray angle conversion part, fourth region)
S100 surface S100a surface (first region)
S100b surface (second region)
S100B surface (first region)
S100C surface (second region)
S110 surface (first region)
S111 surface (second region)
S200 surface S200a surface (third region)
S200b surface (fourth region)
S200B surface (second region)
S200C surface (fourth region)
S210 surface (third region)
S211 surface (fourth region)
T plane Z stereo camera system α10 prism angle (the angle of the inclined portion that is the smallest inclination with respect to the optical axis among the inclined portions that configure the first prism)
α21 Prism angle (the angle of the inclined portion that is the smallest inclination with respect to the optical axis among the inclined portions that constitute the second prism)

Claims (16)

A first light ray angle conversion unit and a second light ray angle conversion unit that change the angle of a light ray propagated from an object,
The light ray propagated from an object propagates through the first light ray angle conversion unit and the second light ray angle conversion unit,
The stereo adapter according to claim 1, wherein the first light ray angle conversion unit and the second light ray angle conversion unit are disposed at a predetermined distance from an optical axis. 2. The stereo adapter according to claim 1,
The first light ray angle conversion unit is
having at least a first region and a second region;
The second light ray angle conversion unit is
having at least a third region and a fourth region;
a direction in which the angle of the incident light is changed in the first region and the second region of the first light ray angle conversion unit is substantially opposite to each other;
a direction in which the angle of the incident light is changed in the third region and the fourth region of the second light ray angle conversion portion is substantially opposite to each other. 2. The stereo adapter according to claim 1,
The first light ray angle conversion unit is
having at least a first region and a second region;
The second light ray angle conversion unit is
having at least a third region and a fourth region;
the angles of the light beam incident on the first region of the first light ray angle conversion unit and the light beam incident on the third region of the second light ray angle conversion unit are changed in opposite directions,
a light ray incident on the second region of the first light ray angle conversion unit and a light ray incident on the fourth region of the second light ray angle conversion unit are changed in angles in opposite directions. 2. The stereo adapter according to claim 1,
The first light ray angle conversion unit is provided at a front stage on the incident side of the light ray, and the second light ray angle conversion unit is provided at a rear stage,
a stereo adapter, wherein the first light ray angle conversion unit converts the angle of the incident light ray so that it is directed in the direction of an optical axis, and the second light ray angle conversion unit converts the angle of the incident light ray so that the light ray is directed in the opposite direction to that of the first light ray angle conversion unit. 2. The stereo adapter according to claim 1,
The first light ray angle conversion unit is
having at least a first region and a second region;
A stereo adapter, wherein the first region and the second region of the first light ray angle conversion portion are spatially separated from each other. 2. The stereo adapter according to claim 1,
The second light ray angle conversion unit is
having at least a third region and a fourth region;
2. The stereo adapter according to claim 1, wherein the third region and the fourth region of the second light ray angle conversion portion are spatially separated from each other. 2. The stereo adapter according to claim 1,
the first light ray angle conversion unit is constituted by a first prism,
The stereo adapter according to claim 1, wherein the second light ray angle conversion unit is configured with a second prism that is different from the first prism. 8. The stereo adapter according to claim 7,
A stereo adapter, wherein the refractive index of the first prism is greater than the refractive index of the second prism. 9. The stereo adapter according to claim 8,
a tilt angle of the inclined portion constituting the first prism that is least tilted with respect to the optical axis is smaller than a tilt angle of the inclined portion constituting the second prism that is least tilted with respect to the optical axis. 2. The stereo adapter according to claim 1,
The stereo adapter, wherein at least one of the first light ray angle conversion unit and the second light ray angle conversion unit is a diffraction element. 2. The stereo adapter according to claim 1,
The stereo adapter according to claim 1, wherein the first light ray angle conversion unit and the second light ray angle conversion unit are arranged in series with respect to an optical axis. a first light ray angle conversion unit and a second light ray angle conversion unit that change the angle of a light ray propagated from an object;
a light collecting unit that forms an image using light transmitted through the first light ray angle conversion unit and the second light ray angle conversion unit;
A light receiving unit that detects the formed image;
having
The light ray propagated from an object propagates through the first light ray angle conversion unit and the second light ray angle conversion unit,
The stereo camera according to claim 1, wherein the first light ray angle conversion unit and the second light ray angle conversion unit are disposed at a predetermined distance from an optical axis. The stereo camera according to claim 12,
The first light ray angle conversion unit is
having at least a first region and a second region;
The first light ray angle conversion unit is
having at least a third region and a fourth region;
The light receiving section has a first light receiving region and a second light receiving region,
a light beam transmitted through the first region of the first light ray angle conversion unit and the third region of the second light ray angle conversion unit is incident on the first light receiving region by the light collecting unit,
a light beam passing through the second region of the first light ray angle conversion unit and the fourth region of the second light ray angle conversion unit is incident on the second light receiving region by the light collecting unit. The stereo camera according to claim 13,
A stereo camera characterized in that, with respect to a direction in which the first light receiving area and the second light receiving area are arranged, in the same direction, a separation direction when red and blue light are incident is the same in the first light receiving area and the second light receiving area. a first light ray angle conversion unit and a second light ray angle conversion unit that change the angle of a light ray propagated from an object;
a light collecting unit that forms an image by the light rays that have passed through the first light ray angle conversion unit and the second light ray angle conversion unit;
A light receiving unit that detects the formed image;
A stereo image processing device;
having
The first light ray angle conversion unit is
having at least a first region and a second region;
The first light ray angle conversion unit is
having at least a third region and a fourth region;
The light ray propagated from an object propagates through the first light ray angle conversion unit and the second light ray angle conversion unit,
the first light ray angle conversion unit and the second light ray angle conversion unit are disposed at a predetermined distance from an optical axis,
the light receiving section has a first light receiving region and a second light receiving region,
a light beam transmitted through the first region of the first light ray angle conversion unit and the third region of the second light ray angle conversion unit is incident on the first light receiving region by the light collecting unit,
a light beam transmitted through the second region of the first light beam angle conversion unit and the fourth region of the second light beam angle conversion unit is incident on the second light receiving region by the light collecting unit,
The stereo image processing device includes:
A stereo camera system comprising: a first light receiving region and a second light receiving region; and a second light receiving region, the second light receiving region and a second light receiving region, and a parallax detection region. The stereo camera system according to claim 15,
A distance is estimated based on the parallax.
a stereo camera system characterized in that, when estimating the disparity, a baseline length is changed according to a stereo camera incidence angle, which is an angle of the light ray incident on the first light ray angle conversion unit and is an incidence angle of the light ray on a plane perpendicular to an optical axis.

Images