JP7331222B2 - Optical device and imaging device - Google Patents

Description

The present invention relates to an imaging device.

An imaging optical system that forms an image of an observation target has various physical properties such as a focal length and an angle of view. As the focal length increases, a magnified image of the observation object is formed, so detailed optical information of the distant observation object, in other words, magnified optical information can be obtained. As the angle of view is widened, optical information of the observation target located in a wider range can be obtained. However, there is a trade-off relationship between the focal length and the angle of view. The longer the focal length, the narrower the angle of view, and the shorter the focal length, the wider the angle of view.

Therefore, the focal length is adjusted to obtain the desired optical information depending on the situation. For example, the focal length is adjusted by displacing a zoom lens included in the imaging optical system. Also, the focal length is adjusted by switching between a plurality of single-focal lenses (see Patent Documents 1 and 2).

JP-A-11-311832 JP-A-2004-279556

In Patent Documents 1 and 2, magnified optical information and wide range optical information can be separately provided by switching the focal length. However, it has not been possible to obtain wide and magnified optical information at the same time.

It is an object of the present disclosure made in view of this point to provide an imaging device that produces wide-range and magnified optical information.

The optical device according to the first aspect comprises:
an optical system that forms an image of the incident first light on a predetermined area;
and an optical element that guides second light, which is different from the first light at an angle formed by an optical axis of the optical system and a principal ray incident on the optical system, to the predetermined area.

The imaging device according to the second aspect is
an optical system for forming an image of incident first light on a predetermined area; an optical device that has an optical element that leads to the predetermined area;
an imaging element arranged so that the predetermined area and the light receiving area overlap.

Wide and magnified optical information can be generated according to the present disclosure.

1 is a configuration diagram showing a schematic configuration of an imaging device according to a first embodiment; FIG. FIG. 2 is a view of the imaging device seen from a direction perpendicular to the optical axis, showing a modification of the optical element of FIG. 1; FIG. 2 is a view of the imaging device seen from a direction perpendicular to the optical axis, showing another modification of the optical element of FIG. 1; FIG. 10 is a view of the image pickup apparatus seen from a direction perpendicular to the optical axis, showing still another modification of the optical element of FIG. 1; FIG. 10 is a view of the image pickup apparatus seen from a direction perpendicular to the optical axis, showing still another modification of the optical element of FIG. 1; FIG. 10 is a view of the image pickup apparatus seen from a direction perpendicular to the optical axis, showing still another modification of the optical element of FIG. 1; FIG. 10 is a view of the image pickup device viewed from the normal direction of the light receiving area, showing still another modification of the optical element of FIG. 1 ; FIG. 10 is a view of the image pickup device viewed from the normal direction of the light receiving area, showing still another modification of the optical element of FIG. 1 ; 2A and 2B are diagrams for explaining physical properties of an imaging element and an optical system in FIG. 1; FIG. FIG. 2 is a conceptual diagram explaining an image reaching a light receiving area in FIG. 1; FIG. 10 is a view of the image pickup device viewed from the normal direction of the light receiving area, showing still another modification of the optical element of FIG. 1 ; FIG. 10 is a view of the image pickup device viewed from the normal direction of the light receiving area, showing still another modification of the optical element of FIG. 1 ; FIG. 10 is a view of the image pickup device viewed from the normal direction of the light receiving area, showing still another modification of the optical element of FIG. 1 ; FIG. 10 is a view of the image pickup device viewed from the normal direction of the light receiving area, showing still another modification of the optical element of FIG. 1 ; FIG. 10 is a view of the image pickup device viewed from the normal direction of the light receiving area, showing still another modification of the optical element of FIG. 1 ; FIG. 10 is a view of the image pickup device viewed from the normal direction of the light receiving area, showing still another modification of the optical element of FIG. 1 ; FIG. 2 is a conceptual diagram illustrating a situation in which a superimposed image reaching the light receiving area of FIG. 1 is formed; FIG. 2 is a conceptual diagram for explaining the process of generating a restored image from a superimposed image by the controller of FIG. 1; FIG. 2 is a flowchart for explaining distance measurement processing executed by the controller in FIG. 1; FIG. FIG. 2 is a configuration diagram showing a schematic configuration of an imaging device according to a second embodiment; FIG. FIG. 21 is a conceptual diagram for explaining image components reaching the light receiving area of FIG. 20; FIG. 21 is a conceptual diagram for explaining image components reaching a light receiving area in the modification of FIG. 20; FIG. 21 is a conceptual diagram for explaining image components reaching a light receiving area in another modified example of FIG. 20; FIG. 11 is a configuration diagram showing a schematic configuration of an imaging device according to a third embodiment; FIG. 25 is a configuration diagram for explaining a pixel structure in the imaging device of FIG. 24;

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the constituent elements shown in the drawings below, the same constituent elements are denoted by the same reference numerals.

As shown in FIG. 1 , an imaging device 10 including an optical device 21 according to the first embodiment of the present disclosure includes the optical device 21 and an imaging device 12 . The imaging device 10 may further include a controller 14 . The optical device 21 includes an imaging optical system (optical system) 11 and an optical element 13 .

The imaging optical system 11 forms an image of the incident subject light flux. The imaging optical system 11 forms an image of the incident first light on a predetermined area pa. The first light may be light emitted from an object point located within the angle of view of the imaging optical system 11 alone. The predetermined area pa may be, for example, a virtual plane or curved surface in three-dimensional space whose center intersects the optical axis ox of the imaging optical system 11 . Hereinafter, the angle of view of the imaging optical system 11 alone, in other words, the angle of view of the imaging optical system 11 in a configuration that does not include the optical element 13 is also referred to as the direct angle of view. The imaging optical system 11 is composed of an optical element for imaging light beams radiated from object points at different positions on different image points alone, in other words, without the optical element 13 . The optical elements that make up the imaging optical system 11 are, for example, lenses, mirrors, diaphragms, and the like.

The imaging optical system 11 may not be telecentric on the image side. In other words, the angle of the principal ray of any light beam passing through the imaging optical system 11 with respect to the optical axis may be greater than 0°. Alternatively, the imaging optical system 11 may be image-side telecentric.

The optical element 13 guides the second light incident on the imaging optical system 11 to a predetermined area pa. The second light differs from the first light in the angle formed by the optical axis ox of the imaging optical system 11 and the principal ray incident on the imaging optical system 11 . The second light may be light emitted from an object point positioned outside the angle of view of the imaging optical system 11, in other words, directly outside the angle of view. Therefore, the angle between the principal ray of the second light and the optical axis ox may be larger than the angle between the principal ray of the first light and the optical axis ox. The principal ray includes a ray passing through the center of the aperture stop of the imaging optical system 15, a ray passing through the center of the entrance pupil of the imaging optical system 15, and the center of a luminous flux emitted from any one object point and entering the imaging optical system 15. can be any of the rays of Furthermore, the optical element 13 may form an image of the second light that has passed through the imaging optical system 11 on a predetermined area pa.

The optical element 13 may be a mirror that reflects the second light and guides it to the predetermined area pa. A reflecting surface of the mirror may be parallel to the optical axis ox of the imaging optical system 11 . Alternatively, the reflective surface of the mirror may not be parallel to the optical axis ox.

As shown in FIG. 2, the reflecting surface of the mirror may be tilted with respect to the optical axis ox so as to face the image pickup optical system 11 side. In the outward tilting posture, the angle of view of the entire optical device 21 can be widened compared to the configuration in which the reflecting surfaces of the mirrors are parallel to the optical axis ox.

In the outward tilting configuration, the optical device 21 may include a first lens 22 for adjusting the optical path length between the imaging optical system 11 and the optical element 13, which is a mirror. By arranging the first lens 22, the deviation of the in-focus position from the predetermined area pa due to the longer optical path length can be reduced compared to the configuration in which the reflecting surface of the mirror is parallel to the optical axis ox. The first lens 22 may be a cylindrical lens in a configuration in which the mirror is a mirror having a surface parallel to the direction perpendicular to the optical axis ox, such as a plane mirror. The first lens 22 may be positioned outside the optical axis ox from a line connecting the principal ray of the first light passing through the outer edge of the exit pupil of the imaging optical system 11 and the predetermined area pa.

Also, in an outward tilt configuration, the optical device 21 may be provided with a prism 23, as shown in FIG. The second light may be reflected by the optical element 13, which is a mirror, and further reflected by the prism 23 to be guided to the predetermined area pa. By providing the prism 23, it is possible to widen the tilt angle between the mirror and the optical axis ox in the outward tilt configuration.

In the outward tilt configuration, the mirror optical element 13 may be, for example, a plane mirror, a curved mirror, a DMD, and a Fresnel mirror.

As shown in FIG. 4, the reflecting surface of the mirror may be inclined with respect to the optical axis ox so as to assume an inwardly inclined posture facing the imaging plane side of the imaging optical system 11 . In the inwardly inclined posture, the size of the entire optical device 21 can be reduced compared to a configuration in which the reflecting surface of the mirror is parallel to the optical axis ox.

In the inward tilting configuration, the mirror optical element 13 may be, for example, a plane mirror, a curved mirror as shown in FIG. 5, a DMD as shown in FIG. 6, and a Fresnel mirror.

The reflecting surface of the mirror may be parallel to any side of the rectangular light receiving area ra of the imaging element 12, which will be described later. Alternatively, the reflecting surface of the mirror may intersect any side of the light receiving area ra as shown in FIG. In a configuration in which the reflective surface of the mirror intersects any side of the light receiving area ra, it is possible to improve the separation accuracy by the image separation model described later. In the configuration where the reflecting surface of the mirror intersects one of the sides of the light receiving area ra, two straight lines vertically extending from both ends of the optical element 13, which is a mirror, when viewed from the normal direction of the light receiving area ra. It is preferable to dispose the optical element 13 so that the overlapping area between the area sandwiched therebetween and the light receiving area ra is maximized.

The mirror may be positioned outside the exit pupil of the imaging optical system 11 when viewed from the optical axis ox direction of the imaging optical system 11 . More specifically, the mirror may be arranged with respect to the imaging optical system 11 such that the reflective surface is located outside the exit pupil. Alternatively, the mirror may be positioned inside the exit pupil when viewed from the optical axis ox direction. In particular, in a configuration where the light receiving area ra is smaller than the pupil diameter, the mirror may be positioned inside the exit pupil.

The mirrors may include multiple plane mirrors. Two plane mirrors belonging to at least one set among the plurality of plane mirrors may be positioned so that their reflecting surfaces are opposite and parallel to each other. Alternatively, the plurality of plane mirrors may be two plane mirrors positioned such that the reflective surfaces are orthogonal to each other, as shown in FIG. Furthermore, the two plane mirrors whose reflective surfaces are orthogonal to each other may be parallel to two sides perpendicular to each other of the rectangular light receiving area ra. The plane mirror and the outer edge of the light receiving area ra of the imaging element 12 may be in close contact in the normal direction of the plane mirror. Alternatively, the flat mirror and the outer edge of the light receiving area ra may have a gap instead of being in close contact in the normal direction of the flat mirror.

As shown in FIG. 9, the distance H between each of the two plane mirrors whose reflecting surfaces are parallel to each other and the optical axis ox may be equal. The two plane mirrors parallel to each other, the imaging optical system 11, and the imaging device 12 may be designed and arranged to satisfy CRA≦tan ⁻¹ (H/B). CRA is the angle with respect to the optical axis ox of the chief ray of the luminous flux emitted from the object point pp at an angle twice the direct angle of view by the imaging optical system 11 . B is the back focus of the imaging optical system 11 .

As will be described later, the combination of the arrangement of the imaging element 12 in the imaging apparatus 10 and the configuration described above allows, as shown in FIG. In other words, the first image component im1 corresponding to the object point emitting the first light arrives without passing through the optical element 13. FIG. The first image component im1 corresponding to the object point directly within the angle of view more specifically corresponds to the subject image located directly within the angle of view. In addition, the second image component im2 corresponding to the object point directly outside the angle of view, in other words, the object point emitting the second light reaches the light-receiving area ra after being inverted via the optical element 13 . The second image component im2 corresponding to the object point directly outside the angle of view, more specifically, corresponds to the subject image positioned directly outside the angle of view.

In the above description, the optical element 13 is a mirror having a surface parallel to the direction perpendicular to the optical axis ox, but it may be a mirror having a curved surface when viewed from the optical axis ox. For example, as shown in FIG. 11, the optical element 13 may be a pair of curved mirrors provided on a pair of opposite sides of the rectangular light receiving region ra when viewed from the normal direction of the light receiving region ra. The curved mirror may be parallel to the normal direction of the light receiving area ra. Alternatively, as shown in FIG. 12, the optical element 13 may be a mirror having a circular curved surface that includes a rectangular light receiving area ra when viewed from the normal direction of the light receiving area ra. Alternatively, as shown in FIG. 13, the optical element 13 may be a mirror having an elliptical curved surface that encloses a rectangular light receiving area ra when viewed from the normal direction of the light receiving area ra. A mirror with an elliptical curved surface is preferred in configurations where the light receiving area ra is a rectangle other than a square. Alternatively, as shown in FIG. 14, the optical element 13 may be a mirror having a circular curved surface included in the rectangular light receiving area ra when viewed from the normal direction of the light receiving area ra. Alternatively, as shown in FIG. 15, the optical element 13 may be a mirror having an elliptical curved surface included in the rectangular light receiving area ra when viewed from the normal direction of the light receiving area ra. In the configuration in which the optical element 13 is a mirror having a curved surface included in the rectangular light receiving area ra, a gap between the light receiving area ra and the mirror is eliminated when viewed from the normal direction of the light receiving area ra. obtain. In such a configuration, the elimination of gaps can improve the continuity of optical information in a superimposed image, which will be described later, compared to a configuration having gaps.

The imaging element 12 captures an image formed within the light receiving area ra. The imaging element 12 may be arranged in the imaging device 10 so that the predetermined area pa of the optical device 21 and the light receiving area ra overlap. Therefore, the light receiving area ra of the imaging device 12 may directly correspond to the angle of view. The direct angle of view may be an angle of view corresponding to the range of the object point imaged within the light receiving area ra without the optical element 13 intervening. At least part of the light flux, which is the first light, entering the imaging optical system 11 from within the direct angle of view of the imaging optical system 11 may be imaged on the light receiving area ra. Also, at least a part of the light flux, which is the second light that enters the imaging optical system 11 from outside the direct angle of view of the imaging optical system 11 and passes through the optical element 13, may form an image on the light receiving area ra.

The imaging element 12 may be capable of capturing an image using invisible light such as visible light, infrared light, and ultraviolet light. The imaging element 12 is, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, or the like. The imaging device 12 may be a color image sensor. In other words, the plurality of pixels arranged in the light receiving area ra of the image sensor 12 may be covered with, for example, RGB color filters so as to be evenly distributed within the light receiving area ra. The imaging device 12 generates an image signal corresponding to an image received by imaging. The imaging device 12 may generate image signals at a predetermined frame rate such as 30 fps.

In the imaging element 12 , the outer edge of the light receiving area ra on the side where the optical element 13 is provided may be located outside the outer edge of the exit pupil of the imaging optical system 11 . “Outside the outer edge of the exit pupil” means outside with respect to the optical axis ox of the imaging optical system 11 . As mentioned above, the light receiving area ra may be rectangular.

The imaging device 10 may be provided with a plurality of imaging elements 12 . In a configuration in which a plurality of imaging elements 12 are provided, an optical element 13 may be provided between two imaging elements 12 adjacent to each other, as shown in FIG. By providing the optical element 13 between the two imaging elements 12 adjacent to each other, an object imaged in the gap generated between the light receiving areas ra of the two adjacent imaging elements 12 in a configuration in which the optical element 13 is not provided. A light beam can be imaged by at least one imaging element 12 .

Also, with the above configuration, as shown in FIG. 17, the first image component im1 and the inverted second image component im2 in the configuration where the optical element 13 is a mirror are superimposed in the light receiving area ra. Accordingly, the image sensor 12 captures a superimposed image olim of the first image component im1 and the inverted second image component im2 in the configuration where the optical element 13 is a mirror.

Controller 14 may include at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for specific processing. The dedicated circuit may be, for example, an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like. The controller 14 may apply image processing to the image signal acquired from the imaging element 12 .

As shown in FIG. 18, the controller 14 may perform image processing to separate the superimposed image olim corresponding to the image signal into a first image component im1 and a second image component im2. The controller 14 separates the superimposed image olim by applying image processing methods such as independent component analysis, wavelet method, and image separation model. The image separation model is, for example, a model constructed by creating a superimposed image by superimposing a plurality of images in advance and having the superimposed image learn the plural images as correct answers. In the image separation model, a generator that generates images like the Encoder-Decoder model competes with a discriminator that determines whether the generated image is a fake image, and a Pix-to model that generates a pair of images that reflects the relationship between them. - It may be a model to which Pix is applied. The controller 14 may generate the reconstructed image rcim by combining the separated first image component im1 and the second image component im2.

The controller 14 may use the restored image rcim to measure the distance of the subject captured around the imaging device 10 . The controller 14 performs distance measurement using the restored image rcim, for example, based on DFD (Depth From Defocus). The controller 14 may perform distance measurement using the restored image rcim based on a motion parallax method (SLAM: Simultaneous Localization and Mapping, Motion Stereo), a separation model based on Deep Learning, a foot ranging method, or the like. The foot distance measurement method is a method of calculating three-dimensional coordinates based on image coordinates on the assumption that the lower end of the subject image is positioned on the ground.

The controller 14 may generate the distance image based on the distance corresponding to each address of the restored image rcim. A distance image is an image in which the pixel value of each pixel corresponds to the distance. The controller 14 may give the distance image to the external device.

Next, distance measurement processing executed by the controller 14 in this embodiment will be described using the flowchart of FIG. 19 . The ranging process is started each time an image signal is acquired from the image sensor 12 .

In step S100, the controller 14 separates the second image component im2 from the superimposed image olim corresponding to the acquired image signal. After separation, the process proceeds to step S101.

In step S101, the controller 14 generates the first image component im1 by subtracting the second image component im2 separated in step S100 from the superimposed image olim. After generating the first image component im1, the process proceeds to step S102.

In step S102, the controller 14 generates a restored image rcim by combining the second image component im2 separated in step S101 and the first image component im1 generated in step S101. After generating the restored image rcim, the process proceeds to step S103.

In step S103, the controller 14 uses the restored image rcim generated in step S102 to perform distance measurement for each subject appearing in the restored image rcim. After ranging, the process proceeds to step S104.

In step S104, the controller 14 generates a distance image based on the distance calculated in step S103 and the position of the restored image rcim corresponding to the distance. Also, the controller 14 gives the distance image to the external device. After the distance image is generated, the distance measurement process ends.

The optical device 21 of the first embodiment configured as described above includes an imaging optical system 11 that forms an image of the incident first light on a predetermined area pa, an optical axis ox of the imaging optical system 11 and the imaging optical system. An optical element 13 is provided for guiding second light having a different angle from the first light with respect to the principal ray incident on the optical element 11 to a predetermined area pa. With such a configuration, the optical device 21, while adopting the imaging optical system 11 having a relatively long focal length, captures an image containing optical information in a range wider than the angle of view corresponding to the focal length in the predetermined area ra. can lead Therefore, the optical device 21 can produce wide and magnified optical information.

In addition, in the imaging device 10 of the first embodiment, the optical element 13 is a mirror that reflects the second light and guides it to the predetermined area pa. is parallel to any side of the light receiving area ra. With such a configuration, in the imaging device 10, the direction in which the image reaching the light receiving area ra via the mirror is distorted is one direction or less. Therefore, the imaging apparatus 10 can reduce the image processing load for removing the distortion of the reflected light component due to the mirror included in the captured image, thereby improving the reproducibility of the reflected light component.

In addition, in the imaging device 10 of the first embodiment, the mirrors include a plurality of plane mirrors, and at least one set of two plane mirrors among the plurality of plane mirrors are positioned so that their reflecting surfaces are parallel to each other. With such a configuration, the imaging device 10 can obtain optical information that is directly expanded from the angle of view to both sides about the optical axis ox.

In addition, in the imaging device 10 of the first embodiment, the distance H between the optical axis ox and the reflecting surfaces of the two plane mirrors parallel to each other is equal, and the distance from the object point pp at an angle twice the direct angle of view is The angle CRA of the chief ray of the luminous flux satisfies CRA≦tan ⁻¹ (H/B). With such a configuration, in the imaging device 10, superimposition of three layers of image components due to superimposition of a reflected light component from one plane mirror and a reflected light component from the other plane mirror is prevented, and a subsequent image is obtained. Separation accuracy of image components by processing can be improved.

Further, in the imaging device 10 of the first embodiment, the plane mirror and the outer edge of the light receiving area ra of the imaging element 12 are in close contact in the normal direction of the plane mirror. If there is a gap between the plane mirror and the light receiving area ra of the imaging element 12 in the normal direction, the optical information of the subject imaged in the gap is lost. The imaging device 10 having the above configuration prevents the loss of optical information in such an event.

Further, in the imaging device 10 of the first embodiment, the mirror is positioned outside the exit pupil of the imaging optical system 11 when viewed from the optical axis ox direction. With such a configuration, the imaging device 10 can make the light flux passing through the vicinity of the end of the exit pupil enter the mirror. Therefore, the image pickup apparatus 10 can reduce the decrease in the amount of light due to vignetting of a part of the light flux passing near the exit pupil.

Further, in the imaging apparatus 10 of the first embodiment, the angle of the principal ray of any light beam in the imaging optical system 11 with respect to the optical axis ox is greater than 0°. With such a configuration, in the image pickup apparatus 10 , the image pickup optical system 11 is not telecentric on the image side, so that a light beam from an object point with an angle wider than the direct angle of view can be made incident on the optical element 13 . Therefore, the imaging apparatus 10 can reliably generate optical information in a range widened from the direct angle of view.

In addition, in the imaging apparatus 10 of the first embodiment, the image corresponding to the image signal is divided into the first image component im1 corresponding to the object point directly in the image and the second image component im1 directly corresponding to the object point outside the angle of view. and a controller 14 for separating the image components im2 of . With such a configuration, the imaging device 10 can generate an image in which the superimposed image olim in which a plurality of image components are superimposed is eliminated.

Next, an imaging device according to a second embodiment of the present disclosure will be described. The second embodiment differs from the first embodiment in the configuration of the optical element and the separation processing by the controller. The second embodiment will be described below, focusing on the differences from the first embodiment. In addition, the same code|symbol is attached|subjected to the site|part which has the same structure as 1st Embodiment.

As shown in FIG. 20, an imaging apparatus 100 according to the second embodiment includes an imaging optical system 11, an imaging element 12, and an optical element 130, similar to the first embodiment. The imaging device 100 may further include a controller 14 . The structures and functions of the imaging optical system 11 and the imaging device 12 in the second embodiment are the same as in the first embodiment. The structure of the controller 14 in the second embodiment is the same as in the first embodiment.

In the second embodiment, similar to the first embodiment, the optical element 130 directs at least a part of the light flux incident on the imaging optical system 11 from outside the direct angle of view of the imaging optical system 11 to the imaging element 12 . An image is formed within the light receiving area ra. In the second embodiment, unlike the first embodiment, the optical element 130 performs optical processing on the incident light flux and emits the light flux.

Optical processing is, for example, a change in the band of the incident light beam. Specifically, the optical element 130 attenuates light in a band corresponding to some of the plurality of color filters covering the imaging element 12 in the incident light flux. Therefore, the optical element 130 forms an image of the light in the band other than the light in the band in question within the light receiving area ra.

Optical element 130 may be a mirror that reflects light in bands other than the band of colors to be attenuated. The optical element 130, for example, attenuates R light and reflects GB light.

With the above configuration, as shown in FIG. 21, in the second embodiment, the light-receiving area ra has a first light-receiving area ra directly corresponding to the object point within the angle of view and corresponding to the colors of all the color filters. The R image component im1r, the first G image component im1g, and the first B image component im1b arrive without going through the optical element 130 . In addition, in the light receiving area ra, the second G image component im2g and the second B image component im2b corresponding to the object point directly outside the angle of view and corresponding to the color other than the attenuated color component pass through the optical element 130. reach through.

Alternatively, the optical processing is, for example, addition of a light-dark pattern to the incident light flux according to the incident position. Specifically, as shown in FIG. 22, the optical element 130 has a surface on which the first regions 190 and the second regions 200 are distributed, for example, in a checkered pattern. The first region 190 attenuates the luminance of incident light with a first attenuation factor and emits the light. The first attenuation rate is greater than 0% and less than 100%. The second region 200 attenuates the luminance of incident light at a second attenuation factor and emits the light. The second attenuation rate is 0% or more and less than the first attenuation rate. Therefore, the optical element 130 gives the incident light beam a contrast difference according to the pattern of the first area 190 and the second area 200, and forms an image in the light receiving area ra.

Alternatively, the optical processing is, for example, imparting distortion to an image formed within the light receiving area ra by an incident light beam. Specifically, as shown in FIG. 23, the optical element 130 is a mirror having a cylindrical curved surface whose axis is a line parallel to the optical axis. is imaged within the light receiving area ra. More specifically, the optical element 130 is an image distorted so as to expand in a direction connecting both ends of an arc of a cross-section of the optical element, which is a mirror with a plane perpendicular to the optical axis.

In the second embodiment, similar to the first embodiment, the controller 14 converts the superimposed image olim corresponding to the image signal into the first image component im1 and the second image component im2 by image processing. Image processing may be performed to separate the In the second embodiment, the controller 14 separates the superimposed image olim by an image processing method using an image separation model.

The image separation model in the second embodiment will be explained below. In the image separation model, a first image without optical processing and a second image different from the first image are subjected to image processing equivalent to the optical processing performed by the optical element 130 in advance, and a second image is obtained. This model is constructed by generating a superimposed image superimposed on one image and learning the superimposed image with the first image and the second image as correct answers.

In configurations where the optical processing is band change, the first image is the RGB image components of an arbitrary image. Also, in that arrangement, the second image is the GB image component of an image separate from the arbitrary image. In configurations where the optical processing is a band change, the R image component of any image can also be used for training in addition to the superimposed image.

In configurations where the optical processing is the addition of contrast patterns, the first image is an arbitrary image. Also, in this configuration, the second image is an image obtained by changing the luminance of an image different from the arbitrary image by the contrast pattern of the optical element 130 .

In configurations where the optical processing is to apply distortion, the first image is any image. Moreover, in the configuration, the second image is an image obtained by reflecting an image different from the arbitrary image by a mirror having the same curved surface as the optical element 130 .

In the second embodiment, the controller 14 may generate the restored image rcim by combining the separated first image component im1 and second image component im2, as in the first embodiment. In the second embodiment, as in the first embodiment, the controller 14 may use the restored image rcim to measure the distance of the subject captured around the imaging device 100 . In the second embodiment, as in the first embodiment, the controller 14 may generate a distance image based on the distance corresponding to each address of the restored image rcim and give it to the external device.

The optical device 210 of the second embodiment configured as described above also includes an imaging optical system 11 that forms an image of the incident first light on a predetermined area pa, an optical axis ox of the imaging optical system 11, and the imaging optical system. An optical element 130 is provided for guiding second light, which has a different angle with the principal ray incident on 11 from the first light, to a predetermined area pa. Therefore, the imaging device 100 can also produce wide and magnified optical information.

Also in the imaging device 100 of the second embodiment, the optical element 130 is a mirror that reflects the second light and forms an image within the predetermined area pa, and the reflecting surface of the mirror is the optical axis ox and the imaging element It is parallel to any side of the 12 rectangular light receiving areas ra. Therefore, the imaging apparatus 100 can also reduce the image processing load for removing the distortion of the reflected light component due to the mirror included in the captured image, thereby improving the reproducibility of the reflected light component.

Also in the imaging device 100 of the second embodiment, the mirrors include a plurality of plane mirrors, and at least one pair of two plane mirrors of the plurality of plane mirrors are positioned such that the reflecting surfaces are parallel to each other. Therefore, the imaging apparatus 100 can also acquire optical information that is directly expanded from the angle of view to both sides around the optical axis ox.

Also in the imaging apparatus 100 of the second embodiment, the distance H between the optical axis ox and the reflecting surfaces of the two plane mirrors parallel to each other is equal, and the angle from the object point pp at an angle twice the direct angle of view is The angle CRA of the chief ray of the luminous flux satisfies CRA≦tan ⁻¹ (H/B). Therefore, in the image capturing apparatus 100 as well, superimposition of three layers of image components due to superimposition of the reflected light component from one plane mirror and the reflected light component from the other plane mirror is prevented, and the image component by subsequent image processing is prevented. can improve the separation accuracy of

Also, in the imaging apparatus 100 of the second embodiment, the plane mirror and the outer edge of the light receiving area ra of the imaging element 12 are in close contact in the normal direction of the plane mirror. Therefore, the imaging device 100 also prevents loss of optical information.

Also in the imaging apparatus 100 of the second embodiment, the mirror is positioned outside the exit pupil of the imaging optical system 11 when viewed from the optical axis ox direction. Therefore, the imaging apparatus 100 can also reduce the decrease in light amount due to vignetting of a part of the light flux passing near the exit pupil.

Also in the imaging apparatus 100 of the second embodiment, the angle of the principal ray of any light beam in the imaging optical system 11 with respect to the optical axis ox is greater than 0°. Therefore, the imaging apparatus 100 can also reliably generate optical information in a range widened from the direct angle of view.

Also, in the imaging apparatus 100 of the second embodiment, the image corresponding to the image signal is divided into the first image component im1 corresponding to the object point directly in the image and the second image component im1 directly corresponding to the object point outside the angle of view. and a controller 14 for separating the image components im2 of . Therefore, the imaging apparatus 100 can also generate an image in which the superimposed image olim in which a plurality of image components are superimposed is eliminated.

Further, in the imaging device 100 of the second embodiment, the optical element 130 performs optical processing on the light flux incident on the optical element 130 and emits the light flux. With such a configuration, in the imaging apparatus 100, optical features corresponding to the optical processing are added to the separated image components. Therefore, in the imaging device 100, an image separation model trained to improve separation accuracy can be constructed. Therefore, the imaging device 100 can improve the restoration accuracy of the restored image.

Next, an imaging device according to a third embodiment of the present disclosure will be described. The third embodiment differs from the first embodiment in the configuration of the imaging device and the separation processing by the controller. The third embodiment will be described below, focusing on the differences from the first embodiment. In addition, the same code|symbol is attached|subjected to the site|part which has the same structure as 1st Embodiment.

As shown in FIG. 24, an imaging apparatus 101 according to the third embodiment includes an imaging optical system 11, an imaging element 121, and an optical element 13, similar to the first embodiment. The imaging device 101 may further include a controller 14 . The structures and functions of the imaging optical system 11 and the optical element 13 in the third embodiment are the same as in the first embodiment. The structure of the controller 14 in the third embodiment is the same as in the first embodiment.

In the third embodiment, the imaging element 121 captures an image formed within the light receiving area ra via the imaging optical system 11, similar to the first embodiment. The image sensor 121 may be capable of capturing an image using invisible light such as visible light, infrared light, and ultraviolet light, similar to the first embodiment. The imaging device 121 may be a color image sensor. The imaging element 121 generates an image signal corresponding to an image received by imaging, similar to the first embodiment. The imaging device 121 may generate an image signal at a predetermined frame rate such as 30 fps, similar to the first embodiment. In the imaging element 121 , the outer edge of the light receiving area ra on the side where the optical element 13 is provided may be located outside the outer edge of the exit pupil of the imaging optical system 11 , as in the first embodiment. The light receiving area ra may be rectangular as in the first embodiment.

In the third embodiment, the imaging device 121 may be a dual pixel type image sensor, unlike the first embodiment. As shown in FIG. 25, the imaging element 121, which is a dual-pixel type image sensor, has a first PD (Photo Diode) 171 and a second PD 181 provided in the pixel 161 covered by each microlens 151. This image sensor has a structure in which light can enter only one PD depending on the direction of incidence. For example, in each pixel 161, the first PD 171 can receive only light beams from a direction inclined toward the optical axis ox side, and the second PD 181 can receive only light beams from a direction inclined toward the optical element 13 side. Incident is possible.

With the configuration as described above, the first image component im1 corresponding to the object point in the image directly reaches the first PD 171 in the light receiving area ra without passing through the optical element 13 . Further, the second image component im2 corresponding to the object point outside the direct angle of view is inverted via the optical element 13 and reaches the second PD 181 in the light receiving area ra.

In the third embodiment, similar to the first embodiment, the controller 14 converts the superimposed image olim corresponding to the image signal into the first image component im1 and the second image component im2 by image processing. Image processing may be performed to separate the In the third embodiment, the controller 14 may generate the first image component im1 based only on signals generated by the first PD 171. FIG. "Based only on the signal generated by the first PD 171" defines that the signal generated by the second PD 181 is not used. It may involve using an irrelevant signal. Also, in the third embodiment, the controller 14 may generate the inverted second image component im2 based only on the signal generated by the second PD 181 . The meaning of being based only on the signal generated by the second PD 181 is similar to the meaning of being based only on the signal generated by the first PD 171 .

In the third embodiment, the controller 14 may generate the restored image rcim by combining the separated first image component im1 and second image component im2, as in the first embodiment. In the third embodiment, similarly to the first embodiment, the controller 14 may use the restored image rcim to measure the distance of the subject captured around the imaging device 101 . In the third embodiment, as in the first embodiment, the controller 14 may generate a distance image based on the distance corresponding to each address of the restored image rcim and give it to the external device.

In the imaging device 101 of the third embodiment configured as described above, the optical element 13 is also a mirror that reflects the second light and guides it into the predetermined area pa, and the reflecting surfaces of the mirror are the optical axis ox and It is parallel to any side of the rectangular light receiving area ra of the imaging device 121 . Therefore, the imaging apparatus 101 can also reduce the image processing load for removing the distortion of the reflected light component due to the mirror included in the captured image, thereby improving the reproducibility of the reflected light component.

Also in the imaging device 101 of the third embodiment, the mirrors include a plurality of plane mirrors, and at least one pair of two plane mirrors among the plurality of plane mirrors are positioned so that their reflecting surfaces are parallel to each other. Therefore, the imaging apparatus 101 can also obtain optical information that is directly expanded from the angle of view to both sides around the optical axis ox.

Also in the imaging apparatus 101 of the third embodiment, the distance H between the optical axis ox and the reflecting surfaces of the two plane mirrors parallel to each other is equal, and the angle from the object point pp at an angle twice the direct angle of view is The angle CRA of the chief ray of the luminous flux satisfies CRA≦tan ⁻¹ (H/B). Therefore, in the imaging device 101 as well, superimposition of three layers of image components due to the superimposition of the reflected light component from one plane mirror and the reflected light component from the other plane mirror is prevented, and the image component by subsequent image processing is prevented. can improve the separation accuracy of

Also in the imaging apparatus 101 of the third embodiment, the plane mirror and the outer edge of the light receiving area ra of the imaging element 121 are in close contact in the normal direction of the plane mirror. Therefore, the imaging device 101 also prevents loss of optical information.

Also in the imaging apparatus 101 of the third embodiment, the mirror is positioned outside the exit pupil of the imaging optical system 11 when viewed from the optical axis ox direction. Therefore, the image pickup apparatus 101 can also reduce the decrease in the amount of light due to vignetting of a part of the light flux passing near the exit pupil.

Also in the imaging apparatus 101 of the third embodiment, the angle of the principal ray of any light beam in the imaging optical system 11 with respect to the optical axis ox is greater than 0°. Therefore, the imaging apparatus 101 can also reliably generate optical information in a range widened from the direct angle of view.

Also, in the imaging apparatus 101 of the third embodiment, the image corresponding to the image signal is divided into the first image component im1 corresponding to the object point directly in the image and the second image component im1 directly corresponding to the object point outside the angle of view. and a controller 14 for separating the image components im2 of . Therefore, the imaging apparatus 101 can also generate an image in which the superimposed image olim in which a plurality of image components are superimposed is eliminated from the superimposed image.

A driving space image was prepared as a data set of images for learning. Each image in these datasets was used to create a set of superimposed images in which a portion of the full image was cut and one subimage was superimposed on the other. An image separation model for separating superimposed images was learned using the created 15000 sets as teacher data, and the image separation model of Example 1 was constructed. The superimposed images were separated by the image separation model of Example 1 using the remaining sets as data for confirmation. PSNR and SSIM were calculated by comparing the separated image and the partial image used for the superimposed image. For the image separation model of Example 1, PSNR and SSIM were 20.54 and 0.55, respectively.

Using each image of the same data set as in Example 1, a set of superimposed images was created by cutting a portion of the entire image and superimposing the GB image components of one partial image on the RGB image components of the other partial image. . An image separation model for separating superimposed images was learned using the created X1 set as teacher data, and the image separation model of Example 2 was constructed. The superimposed images were separated by the image separation model of Example 2 using the remaining sets as data for confirmation. PSNR and SSIM were calculated by comparing the separated image and the partial image used for the superimposed image. The image separation model of Example 2 gave PSNR and SSIM of 27.40 and 0.92, respectively.

Using each image of the same data set as in Example 1, a part of the whole image is cut, and one partial image is attenuated by a square checkered contrast pattern corresponding to an angle of view of 25 °, A set of superimposed images superimposed on the other subimage was created. The attenuation rate of one area of the contrast pattern was 70%, and the attenuation rate of the other area was 30%. An image separation model for separating superimposed images was learned using the created X1 set as teacher data, and the image separation model of Example 3 was constructed. The superimposed images were separated by the image separation model of Example 3 using the remaining sets as data for confirmation. PSNR and SSIM were calculated by comparing the separated image and the partial image used for the superimposed image. The image separation model of Example 3 gave PSNR and SSIM of 24.48 and 0.88, respectively.

Using each image of the same data set as in Example 1, a part of the whole image is cut, one partial image is distorted so as to reproduce the distortion aberration, and a set of superimposed images superimposed on the other partial image It was created. An image separation model for separating superimposed images was learned using the created X1 set as teacher data, and the image separation model of Example 4 was constructed. The superimposed images were separated by the image separation model of Example 4 using the remaining sets as data for confirmation. PSNR and SSIM were calculated by comparing the separated image and the partial image used for the superimposed image. The image separation model of Example 4 gave PSNR and SSIM of 25.26 and 0.90, respectively.

In one embodiment, (1) the optical device comprises:
an optical system that forms an image of the incident first light on a predetermined area;
and an optical element that guides second light, which is different from the first light at an angle formed by an optical axis of the optical system and a principal ray incident on the optical system, to the predetermined area.

(2) In the optical device of (1) above,
The angle between the principal ray of the second light and the optical axis is greater than the angle between the principal ray of the first light and the optical axis.

(3) In the optical device of (1) or (2) above,
The optical element is a mirror that reflects the second light and guides it to the predetermined area.

(4) The imaging device
the optical device of (1) or (2);
an imaging element arranged so that the predetermined area and the light receiving area overlap.

(5) The imaging device of (4) above is
The optical element is a mirror that reflects the second light and guides it to the predetermined area.

(6) In the imaging device of (5) above,
A reflective surface of the mirror is parallel to the optical axis of the optical system.

(7) In the imaging device of (5) or (6) above,
The reflecting surface of the mirror is parallel to one side of the rectangular light receiving area of the imaging element.

(8) In the imaging device of (5) to (7) above,
the mirror includes a plurality of plane mirrors;
At least one set of two plane mirrors among the plurality of plane mirrors has reflecting surfaces parallel to each other.

(9) In the imaging device of (8) above,
The imaging optics corresponding to the light-receiving area of the imaging element, where the distance H between the optical axis of the imaging element and the reflecting surfaces of two plane mirrors whose reflecting surfaces are parallel to each other is equal, and the back focus of the optical system is B. The angle CRA of the chief ray of the luminous flux from the object point at an angle twice the direct angle of view of the system satisfies CRA≦tan ⁻¹ (H/B).

(10) In the imaging device of (8) or (9) above,
The plane mirror and the outer edge of the light receiving area of the imaging element are in close contact in the normal direction of the plane mirror.

(11) In the imaging device of (5) to (10) above,
The mirror is positioned outside the exit pupil of the optical system when viewed along the optical axis of the optical system.

(12) In the imaging device of (5) to (11) above,
The angle of the chief ray of any luminous flux in the optical system with respect to the optical axis is greater than 0°.

(13) The imaging device of (4) to (12) above,
An image corresponding to an image signal generated by imaging by the imaging device is composed of a first image component corresponding to an object point within a direct angle of view of the imaging optical system corresponding to a light receiving area of the imaging device, and the direct image. and a second image component corresponding to the out-of-corner object points.

(14) In the imaging device of (4) to (13) above,
The optical element performs optical processing on a light beam incident on the optical element and outputs the light beam.

(15) In the imaging device of (14) above,
Said optical processing is a change in the band of the incident light beam.

(16) In the imaging device of (14) above,
The optical processing is addition of a light-dark pattern to the incident light flux according to the incident position.

(17) In the imaging device of (14) above,
The optical processing is imparting distortion to an image formed within the light receiving area by an incident light flux.

The embodiments of the imaging method using the imaging devices 10, 100, and 101 have been described above. (For example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card) can be used.

Moreover, the implementation form of the program is not limited to an application program such as an object code compiled by a compiler or a program code executed by an interpreter. good. Furthermore, the program may or may not be configured so that all processing is performed only in the CPU on the control board. The program may be configured to be partially or wholly executed by another processing unit mounted on an expansion board or expansion unit added to the board as required.

The figures describing the embodiments of the present disclosure are schematic. The dimensional ratios and the like on the drawings do not necessarily match the actual ones.

Although the embodiments according to the present disclosure have been described with reference to drawings and examples, it should be noted that various modifications or alterations can be made by those skilled in the art based on the present disclosure. Therefore, it should be noted that these variations or modifications are included within the scope of this disclosure. For example, the functions included in each component can be rearranged so as not to be logically inconsistent, and multiple components can be combined into one or divided.

For all elements set forth in this disclosure and/or for any method or process step disclosed, in any combination, except combinations where these features are mutually exclusive. Can be combined. Also, each feature disclosed in this disclosure, unless expressly contradicted, may be replaced by alternative features serving the same, equivalent, or similar purpose. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of identical or equivalent features.

Furthermore, embodiments in accordance with the present disclosure are not limited to any specific configuration of the embodiments described above. Embodiments of the present disclosure extend to all novel features or combinations thereof described in the present disclosure or to all novel method or process steps or combinations thereof described. be able to.

Descriptions such as “first” and “second” in the present disclosure are identifiers for distinguishing the configurations. Configurations that are differentiated in descriptions such as "first" and "second" in this disclosure may interchange the numbers in that configuration. For example, a first image component can exchange identifiers "first" and "second" with a second image component. The exchange of identifiers is done simultaneously. The configurations are still distinct after the exchange of identifiers. Identifiers may be deleted. Configurations from which identifiers have been deleted are distinguished by codes. The description of identifiers such as “first” and “second” in this disclosure should not be used as a basis for interpreting the order of the configuration or the existence of lower numbered identifiers.

Reference Signs List 10, 100, 101 imaging device 11 imaging optical system 12, 121 imaging element 13, 130 optical element 14 controller 151 microlens 161 pixel 171 first PD (Photo Diode)
181 Second PD
190 First region 200 Second region 21, 210, 211 Optical device 22 First lens 23 Prism CRA Chief ray of light emitted from an object point at an angle twice the direct angle of view by the imaging optical system Angle to axis im1 First image component im1b First B image component im1g First G image component im1r First R image component im2 Second image component im2b Second B image component im2g Second G image component olim Superimposed image ox Optical axis pa Predetermined area rcim Restored image pp Object point at twice the direct angle of view ra Light receiving area

Claims (17)

an optical system that forms an image of the incident first light on a predetermined light receiving area of the imaging element;
receiving the second light incident on the optical system, the second light having an angle formed by an optical axis of the optical system and a principal ray incident on the optical system different from the first light; an optical element that guides the image to an area and superimposes it on the image of the first light to cause the imaging element to image it,
An optical device that causes the imaging element to output an image signal in which an image formed by the first light and an image formed by the second light are superimposed. An optical device according to claim 1, wherein
An angle between the principal ray of the second light and the optical axis is greater than an angle between the principal ray of the first light and the optical axis. 3. The optical device according to claim 1, wherein
The optical device, wherein the optical element is a mirror that reflects the second light and guides it to the predetermined light receiving area. an optical device according to claim 1 or 2;
and the imaging device having the predetermined light receiving area. In the imaging device according to claim 4,
The imaging device, wherein the optical element is a mirror that reflects the second light and guides it to the predetermined light receiving area. In the imaging device according to claim 5,
The imaging device, wherein the reflecting surface of the mirror is parallel to the optical axis of the optical system. In the imaging device according to claim 5,
The imaging device, wherein the reflecting surface of the mirror is parallel to one side of the rectangular light receiving area of the imaging device. In the imaging device according to claim 5,
the mirror includes a plurality of planar mirrors;
The imaging device, wherein at least one set of two plane mirrors among the plurality of plane mirrors has reflecting surfaces parallel to each other. The imaging device according to claim 8,
The optical system corresponding to the light-receiving area of the imaging element, where the distance H between the optical axis of the imaging element and the reflecting surfaces of two plane mirrors whose reflecting surfaces are parallel to each other is equal, and the back focus of the optical system is B. an angle CRA of a principal ray of a luminous flux from an object point at an angle twice the direct angle of view of , satisfies CRA≦tan ⁻¹ (H/B). The imaging device according to claim 8,
The imaging device, wherein the plane mirror and the outer edge of the light receiving area of the imaging element are in close contact with each other in the normal direction of the plane mirror. In the imaging device according to claim 5,
The imaging device, wherein the mirror is positioned outside an exit pupil of the optical system when viewed from the optical axis direction of the optical system. In the imaging device according to claim 5,
An imaging device in which an angle of a principal ray of any light flux in the optical system with respect to the optical axis is greater than 0°. In the imaging device according to claim 4,
An image corresponding to an image signal generated by imaging by the imaging device is composed of a first image component corresponding to an object point within a direct angle of view of the optical system corresponding to a light receiving area of the imaging device, and the direct angle of view. and a second image component corresponding to the outer object point. In the imaging device according to claim 4,
The imaging device, wherein the optical element performs optical processing on a light flux incident on the optical element and emits the light flux. 15. The imaging device according to claim 14, wherein
The imaging device, wherein the optical processing is a change in the band of an incident light beam. 15. The imaging device according to claim 14, wherein
The optical processing is addition of a light-dark pattern to an incident light beam according to an incident position. 15. The imaging device according to claim 14, wherein
The optical processing is imparting distortion to an image formed within the light receiving area by an incident light flux.

Images