JP7380146B2 - Information acquisition device - Google Patents

Description

The present invention relates to an information acquisition device.

Conventionally, 3D sensing technology that optically acquires 3D information has been used in various fields.

For example, 3D sensing technology related to gesture recognition, object recognition, and obstacle detection, which is used in robots, drones, VR (Virtual Reality), etc., has been disclosed. In such 3D sensing technology, a two-dimensional sensor such as an image sensor capable of surface measurement may be used.

However, the accuracy of 3D sensing using a two-dimensional sensor is limited by the physical resolution of the two-dimensional sensor.

The present invention has been made in view of the above, and an object of the present invention is to obtain information regarding an imaged surface at a resolution greater than the physical resolution of a two-dimensional sensor used.

In order to solve the above-mentioned problems and achieve the objects, the present invention provides a sensor that acquires light delay information and an optical sensor that outputs a light reception signal according to the intensity distribution of light incident on an imaging surface. , a first optical system that transmits light from the imaged surface to a conjugate surface of the imaged surface, and light from a plurality of regions of the imaged surface that can be transmitted by the first optical system. an information acquisition device comprising: a second optical system for forming an image on the same area of the imaging surface; and a processing circuit for acquiring information regarding the imaging surface based on the time-divided light reception signal. Features.

According to the present invention, it is possible to obtain information regarding the imaged surface at a resolution higher than the physical resolution of the two-dimensional sensor used.

FIG. 1 is a diagram illustrating an example of the configuration of an information acquisition device according to an embodiment. FIG. 2 is a cross-sectional view showing an example of the configuration of the imaging system shown in FIG. FIG. 3 is a diagram for explaining image formation regarding the first region of the imaged surface in the information acquisition device of FIG. 1. FIG. 4 is a diagram for explaining image formation regarding the second region of the imaged surface in the information acquisition device of FIG. 1. FIG. 5 is a diagram for explaining image formation regarding the third region of the imaged surface in the information acquisition device of FIG. 1. FIG. 6 is a diagram illustrating an example of the functional configuration of the information acquisition device of FIG. 1. FIG. 7 is a timing chart showing an example of an information acquisition process executed in the information acquisition device of FIG. 1. FIG. 8 is a diagram for explaining acquisition of information regarding the imaged surface when the second optical system in FIG. 1 divides the intermediate image into three. FIG. 9 is a diagram for explaining acquisition of information regarding the imaged surface when the second optical system in FIG. 1 divides the intermediate image into nine parts. FIG. 10 is a diagram for explaining another example of lighting control in the information acquisition process of FIG. 7. FIG. 11 is a timing chart showing another example of the information acquisition process executed by the information acquisition device of FIG. 1. FIG. 12 is a diagram showing another example of the configuration of the second optical system in FIG. 1. FIG. 13 is a diagram for explaining acquisition of information regarding the imaged surface when the second optical system in FIG. 12 divides the intermediate image into nine parts. FIG. 14 is a cross-sectional view showing another example of the configuration of the imaging system and illumination system in FIG. 1. FIG. 15 is a cross-sectional view showing another example of the configuration of the imaging system and illumination system shown in FIG. 14.

In recent years, sensing technology for acquiring 3D information has been used in various fields. For example, at short distances, 3D information can be used to identify an individual when facial recognition is performed using a smartphone, etc., or 3D information can be used to capture hand movements when inputting commands using gestures. There is a technology to do that. For example, in medium and long distances, it is used for detecting vehicles and people ahead in automatic braking systems of automobiles, and for 3D sensing such as detecting obstacles ahead for automatic driving. They are also used to detect obstacles around automatic guided vehicles (AGVs), and are used as three-dimensional sensors as the eyes of robots. For these reasons, there is a demand to improve sensing accuracy.

Conventionally, as a means for acquiring 3D information, there is a means of using a stereo camera. A stereo camera acquires 3D information by converting parallax information obtained by left and right cameras into depth information. There is also a 3D distance sensor called a TOF (Time of Flight) camera. A TOF camera (TOF sensor) projects infrared light or the like onto a target object, calculates the time difference between reflected light from the target object and reaches the sensor, and converts it into distance. Both methods have advantages and disadvantages, but stereo cameras use two cameras and require calibration, making them complicated to use, while TOF sensors have a simple camera configuration, so it is better to use a TOF camera. There are more opportunities. In this way, 3D sensing applications are used for gesture recognition, object recognition, and obstacle detection in markets such as robots, drones, and VR.

However, conventional 3D sensing, especially 3D sensing using two-dimensional TOF sensors, has a sensor resolution of about 250,000 pixels to VGA (approximately 300,000 pixels), so 3D sensing over a wider angle and wider range is required. Therefore, the resolution was insufficient. Therefore, information between pixels could not be sensed in more detail. Therefore, the information acquisition device 1 described below acquires information regarding the imaged surface at a resolution greater than the physical resolution of the two-dimensional sensor used.

Embodiments of the information acquisition device 1 will be described in detail below with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a diagram illustrating an example of the configuration of an information acquisition device 1 according to an embodiment. As shown in FIG. 1, the information acquisition device 1 includes an imaging system 2, an illumination system 3, a processing circuit 4, and a memory 5.

FIG. 2 is a sectional view showing an example of the configuration of the imaging system 2 in FIG. 1. As shown in FIGS. 1 and 2, the imaging system 2 includes a first optical system 21, a second optical system 22, and an optical sensor 23.

The first optical system 21 forms object information on the imaged surface as an intermediate image. That is, the first optical system 21 transmits light from the imaged surface to the intermediate image surface. The intermediate image formed by the first optical system 21 relates to a region R of the imaged surface that can be transmitted by the first optical system 21 . In other words, the intermediate image formed by the first optical system 21 is a region R within the field of view of the first optical system 21 of the region of the imaged surface. Here, it is assumed that the intermediate image plane is a conjugate plane of the imaged surface via the first optical system 21, and is on the optical path from the imaged surface to the imaging surface of the imaging system 2. Note that the first optical system 21 can also be expressed as transmitting an object or spatial information on the imaged surface to a conjugate position as light intensity information.

Although FIG. 2 shows the first optical system 21 composed of a single lens (single lens) for ease of explanation, the present invention is not limited to this. It is sufficient that at least one optical element included in the first optical system 21 has a power to provide desired imaging performance. Here, power refers to the reciprocal of focal length. For example, the first optical system 21 may be configured with a lens system, a reflection system, or the like having positive or negative power. Of course, as the first optical system 21, a compound lens including at least one single lens can also be used as appropriate.

The second optical system 22 is arranged on the image side of the first optical system 21 on the optical path from the first optical system 21 of the imaging system 2 to the imaging surface. The second optical system 22 divides the light intensity information transmitted by the first optical system 21. That is, the second optical system 22 divides the intermediate image formed by the first optical system 21 into a plurality of intermediate images. Hereinafter, each of the divided intermediate images will be referred to as a divided intermediate image. Note that since the intermediate image plane and the imaged surface are in a conjugate relationship, each of the plurality of divided intermediate images can also be expressed as being formed by light from each region of the corresponding imaged surface.

The second optical system 22 transmits information on the plurality of divided intermediate images to the imaging plane. Specifically, the second optical system 22 forms an image by superimposing information of a plurality of divided intermediate images on an imaging plane. Here, it is assumed that the image plane is a conjugate plane of the intermediate image plane via the second optical system 22. In other words, the second optical system 22 forms a plurality of divided intermediate images on the same area on the imaging plane.

As the second optical system 22, as shown in FIG. 2, a configuration including a first lens array 221, a second lens array 222, and an imaging lens 223 can be used. The first lens array 221 and the second lens array 222 divide the intermediate image formed on the intermediate image plane. At this time, the first lens array 221 functions as a field lens for efficiently guiding the light converged by the first optical system 21 to the second lens array 222. The first lens array 221 and the second lens array 222 have a number of condensing elements corresponding to the number of divisions of the intermediate image. Furthermore, the shape of each condensing element corresponds to the shape of each region of the imaged surface. The imaging lens 223 superimposes a plurality of divided intermediate images on an imaging plane to form an image. Like the first optical system, the imaging lens 223 may be configured by a lens system or a reflection system having positive or negative power.

In the example shown in FIG. 2, the first lens array 221 and the second lens array 222 each have three condensing elements, and are related to the intermediate image formed on the intermediate image plane, that is, the region R of the imaged surface. The intermediate image is divided into three intermediate images: a first divided intermediate image A', a second divided intermediate image B', and a third divided intermediate image C'. Here, the first divided intermediate image A' is an intermediate image corresponding to the first region A of the imaged surface. Further, the second divided intermediate image B' is an intermediate image corresponding to the second region B of the imaged surface. The third divided intermediate image C' is an intermediate image corresponding to the third region C of the imaged surface. The imaging lens 223 forms three divided intermediate images A', B', and C' superimposed on the imaging plane.

The first lens array 221 may be an optical element in which a number of condensing elements each having a positive power corresponding to each region on the imaged surface are arranged in the same number as each region on the imaged surface. Not exclusively. As the first lens array 221, for example, a Fresnel lens array or a diffraction lens array can be used as appropriate. Note that when using a single wavelength infrared laser beam as illumination light, a diffraction lens array may be used.

Note that the first lens array 221 may not be provided. In this case, for example, by configuring the first optical system 21 to have telecentricity on the image side using a plurality of optical elements, it is possible to efficiently form an image on the imaging plane.

Note that in the first lens array 221 arranged on the intermediate image plane according to the area division, information also exists at the boundaries of each area. Therefore, it is preferable that the boundary of the area be as close to 0 as possible. Therefore, it is preferable for the first lens array 221 to have a configuration in which, for example, rectangular condensing elements are arranged vertically and horizontally to reduce the boundaries. Further, the first lens array 221 may have a honeycomb-like configuration in which hexagonal condensing elements are arranged, for example. By continuously arranging a plurality of light condensing elements in a rectangular or honeycomb shape, data loss between each light condensing element can be reduced. The configuration of the second lens array 222 may be determined as appropriate depending on the configuration of the first lens array 221.

In this way, the second optical system 22 images the light from a plurality of regions of the region R of the imaged surface that can be transmitted by the first optical system 21 onto the same region of the imaging surface. is configured to do so. As an example, the second optical system 22 includes a first lens array 221 and/or a second lens array 222 (at least one condensing element array) in which condensing elements respectively corresponding to a plurality of divided intermediate images are arranged. ), and at least one imaging lens 223 (condensing element) that focuses the light corresponding to the plurality of divided intermediate images that have passed through the condensing element array onto the same area of the imaging plane. Further, in the light condensing element array, each light condensing element is arranged in a rectangular shape or a honeycomb shape.

The optical sensor 23 outputs a light reception signal according to the intensity distribution of light incident on the imaging surface 231. As the optical sensor 23, various two-dimensional sensors such as a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor) sensor can be used as appropriate. The optical sensor 23 is arranged on the imaging plane of the second optical system 22. The second optical system 22 forms superimposed images on the imaging surface 231 according to the number of divisions. That is, the second optical system 22 images the light from a plurality of regions of the region R of the imaging surface that can be transmitted by the first optical system 21 onto the same region of the imaging surface 231. Further, the optical sensor 23 acquires light intensity information regarding each of the plurality of divided intermediate images formed on the imaging surface 231.

The illumination system 3 illuminates the imaged surface for each region (A, B, C) of the imaged surface corresponding to a plurality of divided intermediate images (A', B', C'). The illumination system 3 includes an illumination optical system 31 and a light source 32. The illumination optical system 31 is an optical system provided to efficiently guide light from the light source 32 to each region (A, B, C) of the imaged surface. Like the first optical system, the illumination optical system 31 may be configured with a lens system, a reflection system, or the like having positive or negative power. As the light source 32, various light sources such as a light emitting diode (LED), a mercury lamp, a fluorescent tube, an incandescent lamp, etc. can be used as appropriate depending on various characteristics such as wavelength dependence of the imaging system 2.

3 to 5 are diagrams for explaining image formation regarding each area on the imaged surface in the information acquisition device 1 of FIG. 1. FIG. 3 relates to the first area A. FIG. 4 relates to the second area B. FIG. 5 relates to the third area C. In addition, FIGS. 3 to 5 show the ray paths (upper rays and lower rays) of the light irradiating each region on the imaged surface and the light reflected (scattered) by each region on the imaged surface, respectively. It is shown.

As shown in FIGS. 3 to 5, the illumination optical system 31 includes a first illumination optical system CL1, a second illumination optical system CL2, and a third illumination optical system CL3. The light source 32 includes a first light source LS1, a second light source LS2, and a third light source LS3. As shown in FIG. 3, light (illumination light) from the first light source LS1 is irradiated onto the first region A on the imaged surface by the first illumination optical system CL1. The light from the illuminated first area A forms a first divided intermediate image A' by the first optical system 21. The first divided intermediate image A' is formed by the second optical system 22 as a first divided image A'' on the imaging surface 231. As shown in FIG. 4, the light (illumination light) from the second light source LS2 is irradiated onto the second region B on the imaged surface by the second illumination optical system CL2. The light from the illuminated second region B forms a second divided intermediate image B' by the first optical system 21. The second divided intermediate image B' is formed by the second optical system 22 as a second divided image B'' on the imaging surface 231. As shown in FIG. 5, the light (illumination light) from the third light source LS3 is irradiated onto the third region C on the imaged surface by the third illumination optical system CL3. The light from the illuminated third region C forms a third divided intermediate image C' by the first optical system 21. The third divided intermediate image C' is formed by the second optical system 22 as a third divided image C'' on the imaging surface 231.

Note that the illumination system 3 may have a configuration in which at least one light source 32 is shared by two or more of the plurality of illumination optical systems CL1, CL2, and CL3.

The processing circuit 4 controls the overall operation of the information acquisition device 1 by executing a program read from the memory 5. The processing circuit 4 acquires information regarding the imaged surface based on the light reception signal from the optical sensor 23. Specifically, the processing circuit 4 acquires information regarding the imaged surface based on time-divided light reception signals from the optical sensor 23. The processing circuit 4 is communicably connected to the optical sensor 23 of the imaging system 2, the light source 32 of the illumination system, and the memory 5 via a bus or various networks. As the processing circuit 4, for example, a CPU (Central Processing Unit) is used, but various processors such as a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array) can be used as appropriate. It is possible.

As the memory 5, ROM (Read Only Memory), RAM (Random Access Memory), etc. can be used as appropriate. For example, the ROM stores programs, data, etc. used by the processing circuit 4. Further, for example, the RAM is used for loading recording data and storing environmental data. As the memory 5, storage devices such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), and a CD-ROM drive can also be used as appropriate.

Note that as the processing circuit 4 and the memory 5, a general-purpose computer having a CPU, ROM, RAM, etc. may be used. Further, the processing circuit 4 and the memory 5 may be configured integrally with the optical sensor 23 and the light source 32, respectively.

FIG. 6 is a diagram showing an example of the functional configuration of the information acquisition device 1 of FIG. 1. As shown in FIG. The processing circuit 4 realizes the functions of the illumination control section 41, the imaging control section 42, and the information acquisition section 43 by executing the information acquisition program stored in the memory 5.

The illumination control unit 41 sequentially causes the light sources LS1, LS2, and LS3 to emit light, and sequentially illuminates the areas A, B, and C on the imaged surface. Note that the illumination control unit 41 sequentially illuminates each region A, B, and C on the imaged surface by sequentially driving each illumination optical system CL1, CL2, and CL3 according to the configuration of the illumination system 3. Good too.

The imaging control unit 42 causes the optical sensor 23 to output a light reception signal corresponding to the light from each area while the area is illuminated, that is, at a timing synchronized with the timing at which the illumination control unit 41 illuminates each area.

The information acquisition unit 43 acquires, as information regarding each area, a light reception signal from the optical sensor 23 regarding light received while each area is illuminated. In other words, the information acquisition unit 43 acquires the time series of the light reception signal from the optical sensor 23 in a time-divided manner according to the illumination timing of each area, as information regarding each area. Furthermore, the information acquisition unit 43 acquires information regarding the imaged surface based on information regarding each region. The information regarding the imaged surface (light intensity information) includes, for example, image information such as an image regarding a region within the field of view of the first optical system 21 on the imaged surface, the presence or absence of a detected object on the imaged surface, It includes optical delay information of a TOF (Time-of-Flight) sensor, such as the distance to an arbitrary object point on the detected object and the shape of the detected object.

Hereinafter, the operation of the information acquisition device 1 according to the embodiment will be described with reference to the drawings.

FIG. 7 is a timing chart showing an example of an information acquisition process executed in the information acquisition device 1 of FIG. 1. In the example shown in FIG. 7, an arbitrary frame n and a frame n+1 following frame n are shown.

The illumination control unit 41 sequentially causes the light sources LS1, LS2, and LS3 to emit light within the time of each frame, and sequentially illuminates the regions A, B, and C on the imaged surface. The imaging control unit 42 causes the optical sensor 23 to output a light reception signal corresponding to the light from each area while each area is illuminated, that is, at a timing synchronized with the timing at which the illumination control unit 41 illuminates each area. . The information acquisition unit 43 also acquires a light reception signal from the optical sensor 23 regarding light received while each area is illuminated, as information regarding each area.

In the example shown in FIG. 7, at timing t0 to t1 of frame n and timing t3 to t4 of frame n+1, the first area A on the imaged surface is illuminated, and the light from the first area A is illuminated. A light reception signal corresponding to the first area A is acquired as information regarding the first area A. At timing t1 to t2 of frame n and timing t4 to t5 of frame n+1, the second region B on the imaged surface is illuminated, and a light reception signal corresponding to the light from the second region B is transmitted. It is acquired as information regarding area B of No. 2. At timing t2 to t3 of frame n and timing t5 to t6 of frame n+1, the third region C on the imaged surface is illuminated, and a light reception signal corresponding to the light from the third region C is illuminated. It is acquired as information regarding area C of No. 3. Note that the light emission order of each of the light sources LS1, LS2, and LS3 can be set arbitrarily, and the time series of the light reception signals are sequentially acquired according to the set light emission order of each light source.

Note that the process of time-divisionally acquiring the light reception signals may be performed sequentially while each area is illuminated, or may be performed after the time series of the light reception signals regarding multiple areas is acquired. . In this case, the illumination timing of each area may be recorded together with the light reception signal time series.

FIG. 8 is a diagram for explaining acquisition of information regarding the imaged surface when the second optical system 22 in FIG. 1 divides the intermediate image into three. As an example, the information acquisition unit 43 acquires information regarding the imaged surface by connecting image data (light reception signals) regarding each area A, B, and C.

In this manner, the information acquisition device 1 according to the present embodiment sequentially irradiates the divided regions with illumination light, and therefore can reliably obtain information on the regions. Furthermore, in the information acquisition device 1, the timing of illuminating each area and the timing of acquiring the light intensity information that has reached the optical sensor 23 are synchronized. In other words, the information acquisition device 1 time-divides the light reception signal of each frame from the optical sensor 23 into a number corresponding to the number of divisions of the region at a timing synchronized with the illumination timing at which the illumination system 3 illuminates each region. and obtain it. Here, in the imaging system 2 according to the present embodiment, the divided images A'', B'', and C'' regarding each area are formed on the imaging surface 231 in a superimposed manner. Therefore, by shifting the time, the light reception signals of the optical sensor 23 can be separated and acquired as different information, that is, information regarding each area A, B, and C. Here, the image data (light reception signals) regarding each area A, B, and C have a resolution corresponding to the physical resolution (number of pixels) of the optical sensor 23, respectively. In other words, since the small area that is each of the plurality of divided intermediate images can be acquired in the entire effective area of the optical sensor 23, high resolution can be easily achieved. Therefore, according to the information acquisition device 1 according to the present embodiment, information regarding the imaged surface can be acquired with a resolution higher than that of the optical sensor 23.

(Second embodiment)
As the first lens array 221 and the second lens array 222 of the second optical system 22, a condensing element array in which nine condensing elements are arranged can also be used. FIG. 9 is a diagram for explaining acquisition of information regarding the imaged surface when the second optical system 22 in FIG. 1 divides the intermediate image into nine parts. In the example shown in FIG. 9, it is assumed that three condensing elements of the condensing element array are arranged in each of the vertical and horizontal directions. At this time, the area of the imaged surface and the intermediate image are divided into nine parts. In the example shown in FIG. 9, each region A, B, and C of the imaged surface shown in FIG. 8 is divided into three regions. In this case, the information acquisition device 1 acquires information regarding each area in the order of, for example, A1 → A2 → A3 → B1 → B2 → B3 → C1 → C2 → C3, and connects the information obtained by dividing the area into 9 areas. Obtain information regarding the imaging surface. Note that the light from each area A1, A2, and A3 on the imaging surface is acquired between timings t0 and t1 in frame n in FIG. At this time, the illumination system 3 may be configured to be able to sequentially illuminate each area A1, A2, and A3 on the imaging surface. In addition, although the case of 3 divisions and 9 divisions was illustrated, it is not limited to this. The number of divisions can be appropriately set according to various limitations such as desired resolution performance and the size of the optical system.

In this way, the imaging system 2 in which a plurality of divided images corresponding to a plurality of regions within the angle of view of the first optical system 21 on the imaging surface are formed in a superimposed manner on the imaging surface 231; According to the information acquisition device 1 having the illumination system 3 that time-divides a plurality of corresponding divided images and transmits them to the optical sensor 23, since information regarding each region can be acquired separately, the projection surface ( It becomes possible to increase the resolution of information on the imaged surface) in accordance with the number of divisions.

(Third embodiment)
The information acquisition device 1 according to each embodiment can obtain information with the highest S/N when there is no external lighting such as at night or when there are no other light emitters of the illumination system 3 within the imaged area. can. Under these circumstances, if the optical sensor 23 is sensitive to visible light and the optical system on the optical path from the imaged surface to the imaged surface 231 has a constant visible light transmittance, the illumination system 3 Light from the imaged surface reaches the optical sensor 23 without illuminating the imaged surface. Therefore, in such a case, when acquiring information on a certain area on the imaging surface, information on other areas also reaches the optical sensor 23, so light components from other areas are acquired as noise. Ru. For this reason, for example, before irradiating the illumination light onto the imaged surface, the information acquisition unit 43 acquires information regarding the entire area on which the illumination light is projected, that is, the entire viewing angle of the first optical system 21 on the imaged surface. (Information N) is acquired. The information acquisition unit 43 uses the acquired information N as a bias component and subtracts the information N from the information regarding each area A, B, and C that has been acquired independently as described above. According to this configuration, it is possible to improve the S/N.

(Fourth embodiment)
The information acquisition device 1 may be configured to use light with a wavelength including an infrared region. Specifically, each of the light sources LS1, LS2, and LS3 of the illumination system 3 is an illumination light source with an emission wavelength that includes an infrared region. Further, each optical system of the imaging system 2 has power for light having a wavelength in the infrared region. Further, the optical sensor 23 is a sensor capable of detecting light having a wavelength in the infrared region. According to this configuration, the object to be detected can be detected without being affected by external light.

(Fifth embodiment)
In each of the above-described embodiments, an example has been described in which each region is selectively illuminated from the illumination system 3 arranged according to each region on the imaged surface, but the present invention is not limited to this. For example, means for blocking or deflecting light from other areas of the target area can be appropriately used at or near the intermediate image position.

For example, a light shutter function is provided at the position where each divided intermediate image A', B', C' is formed, and light from each area A, B, C is guided to the second optical system 22 in time order. It's okay. FIG. 10 is a diagram for explaining another example of lighting control in the information acquisition process of FIG. 7. In the example shown in FIG. 10, the number of divisions is nine, three in the vertical direction and three in the horizontal direction, similarly to the example shown in FIG. Here, the illumination optical system 31 and the light source 32 are configured to illuminate the entire imaged surface, for example. The illumination system 3 of the information acquisition device 1 further includes an optical shutter 33 shown in FIG. The optical shutter 33 has a plurality of shutters whose number corresponds to the number of divisions of the imaged surface. The optical shutters 33 are configured such that each shutter can operate independently. Each shutter of the optical shutter 33 is arranged so as to correspond to each area on the imaging surface. In other words, the optical shutter 33 selectively allows light from regions of the imaged surface corresponding to the plurality of divided intermediate images to pass through.

In each frame, the illumination control unit 41 switches transmission/blocking of each shutter in an order corresponding to each area A1→A1→A2→A3→B1→B2→B3→C1→C2→C3 of the optical shutter 33. The imaging control unit 42 changes the transmission area of the optical shutter 33 at a timing synchronized with the timing at which the illumination control unit 41 switches the transmission area of the optical shutter 33, that is, the timing at which the optical shutter 33 passes light from each area. The light reception signal is outputted to the optical sensor 23. The information acquisition unit 43 also acquires a light reception signal from the optical sensor 23 regarding the light that has passed through the transmission area of the optical shutter 33 as information regarding each area.

Note that instead of the optical shutter 33, for example, a digital micromirror array or the like may be arranged at the intermediate image position. The digital micromirror array controls the tilt of each mirror arranged to correspond to each region on the imaging surface, and deflects light from each region. Further, for example, instead of the optical shutter 33, a liquid crystal shutter or a mechanical shutter may be provided at an intermediate image position to selectively form an image on the imaging surface 231 for each region on the imaging surface. Even with these configurations, effects similar to those of the above-described embodiments can be obtained.

(Sixth embodiment)
The techniques according to the embodiments described above are applicable to TOF sensors. That is, the information acquisition device 1 includes optical delay information of the TOF sensor, such as the presence or absence of a detected object on the imaged surface, the distance to an arbitrary object point on the detected object, and the shape of the detected object. Information regarding the imaged surface (light intensity information) can also be acquired.

In the configurations shown in FIGS. 3 to 5, the illumination system 3 is configured to be able to irradiate each of a plurality of regions of the imaged surface with, for example, pulsed light as illumination light. The optical sensor 23 is a sensor for acquiring light delay information. The information acquisition unit 43 measures the time from when the illumination system 3 irradiates the imaged surface with pulsed light until the light reflected from the imaged surface is detected by the imaged surface 231. The information acquisition unit 43 can acquire the distance from the reference position (light emitting point/plane or imaging surface 231) of the optical sensor 23 to the detected object based on the measured time. Furthermore, when applied to a two-dimensional TOF sensor, the three-dimensional shape of the detected object can also be acquired based on a plurality of distances to a plurality of object points on the detected object. Furthermore, in time measurement, the movement of the object to be detected can be detected by dividing a plurality of minute periods of time and comparing the difference between the measured values in the sections before and after the intervals. Note that the illumination light is not limited to pulsed light, and frequency-modulated continuous waves can also be used as appropriate. In this case, the distance is calculated by measuring the frequency difference between the transmitted wave and the received wave.

FIG. 11 is a timing chart showing another example of the information acquisition process executed by the information acquisition device 1 of FIG. 1. In the example shown in FIG. 11, an arbitrary frame n and a frame n+1 following frame n are shown.

The illumination control unit 41 sequentially causes each of the light sources LS1, LS2, and LS3 to emit pulsed light within the time of each frame, and sequentially illuminates each region A, B, and C on the imaged surface. The imaging control unit 42 causes the optical sensor 23 to output a light reception signal corresponding to the light from each area after each area is illuminated, that is, at a timing synchronized with the timing at which the illumination control unit 41 illuminates each area. Further, the information acquisition unit 43 acquires a light reception signal from the optical sensor 23 regarding light received after each region is illuminated as information regarding each region, and uses the time-divided light reception signal and the emission timing of illumination light. Measure time based on.

Note that the technology related to the information acquisition device 1 provided with the optical shutter 33 and the like described above with reference to FIG. 10 is also applicable to the TOF sensor. In this case, the pulsed light illumination may be applied to the entire imaged surface or may be applied to each region.

Note that if an optical shutter 33 or the like is provided and the optical path from the imaged surface to the imaged surface 231 is restricted to differ between acquisition timings of information regarding each region, the illumination from the illumination system 3 It is also possible to make the light and the reflected light from the imaged surface the same ray path.

In this way, according to the TOF sensor to which the technology according to the present embodiment is applied, the resolution can be improved according to the number of divisions. Therefore, according to the TOF sensor, measurement can be performed with a resolution higher than that of the optical sensor 23, so that resolution performance can be improved. For example, if the image is divided into three parts, measurement can be performed with three times the resolution.

Note that, although the case where one pulse illumination is performed for each target area on the imaged surface has been described as an example, the present invention is not limited to this. During the information acquisition period, the measurement accuracy can be further improved by irradiating the target region on the imaged surface multiple times with pulses or performing modulation driving multiple times.

(Seventh embodiment)
In each of the embodiments described above, the case where the plurality of divided regions have the same size, that is, the case where the capture angle of view for each region is the same, has been described as an example, but the present invention is not limited to this. As the number of condensing elements in the condensing lens array increases, the number of boundaries between the condensing elements increases, so there is a risk that information on the boundaries may not be acquired correctly.

FIG. 12 is a diagram showing another example of the configuration of the second optical system 22 in FIG. 1. FIG. 13 is a diagram for explaining acquisition of information regarding the imaged surface when the second optical system 22 of FIG. 12 divides the intermediate image into nine parts. For example, as shown in FIG. 13, information with a higher priority may exist in the central part of the imaged area than in the peripheral part. Under these circumstances, when dividing the imaged surface into a plurality of regions, there is a risk that data regarding the boundaries may be lost. Therefore, in the information acquisition device 1 according to the present embodiment, in order to eliminate missing data in the center, the imaged surface is divided into a plurality of regions such that the center region is larger than the peripheral region. Specifically, as shown in FIG. 12, in the light condensing element array according to this embodiment, the size of the light condensing elements arranged at the center is larger than the size of the light condensing elements arranged at the periphery. Note that in order to form each of the divided intermediate images (divided intermediate images) on the imaging surface 231 of the optical sensor 23, each lens (condensing element) has a different size, but the focal length is approximately the same. We are doing so. Note that the first optical system 21 has the same configuration as the second optical system 22. According to this configuration, it is possible to reduce the frequency at which a boundary portion appears in the central portion of the imaged region where relatively important information is often present. In other words, data loss between the respective condensing elements of the condensing lens array can be eliminated. Furthermore, in general, when capturing an image of light that has passed through the periphery of the imaging lens (each condensing element), the resolution decreases due to aberrations. In other words, by reducing the frequency at which the boundaries between the condensing elements of the condensing lens array appear in the center of the condensing lens array, the small peripheral resolution of each condensing element can be compensated for. There is also.

Note that there may be cases where information with a higher priority exists in the periphery of the imaged area than in the center. In this case, the imaged surface is divided into a plurality of regions such that the peripheral region is larger than the central region. Specifically, in the light condensing element array according to this embodiment, the size of the light condensing elements arranged in the peripheral part is larger than the size of the light collecting element arranged in the central part. According to this configuration, it is possible to reduce the frequency that a boundary portion appears in the peripheral portion. In other words, data loss between lens arrays in the imaged region can be eliminated, and the small peripheral resolution of each condensing element can be compensated for.

In this way, in the condensing lens array according to this embodiment, the sizes of adjacent condensing elements are different from each other. In this way, by appropriately increasing the size of each condensing element corresponding to a region for which accuracy is desired to be increased, it is possible to reduce data loss regarding the imaged region that occurs between lens arrays.

(Eighth embodiment)
The first lens array 221 according to each of the embodiments described above has a lens center for each arrayed lens (condensing element), as shown in FIG. 2, for example. That is, the first lens array 221 has a plurality of optical axes (optical axis group) because each arrayed lens has an optical axis. On the other hand, the condensing element (imaging lens) disposed on the object side of the optical sensor 23 and on the side closer to the optical sensor 23 of the imaging system 2 is a single lens or a compound lens (a set of lenses) having a common optical axis. lens). Under these circumstances, it is preferable that the central optical axis of the optical axis group of the first lens array 221 and the optical axis of the imaging lens substantially coincide.

However, when the central optical axis of the optical axis group of the first lens array 221 and the optical axis of the imaging lens are made to substantially match, the The optical axis of the condensing element and the optical axis of the imaging lens do not coincide. Therefore, the image formed by the lenses at the center of the first lens array 221 and the image formed by the lenses at the periphery do not necessarily match in magnification error or degree of distortion. In other words, the optical aberrations generated in each of the divided images A'', B'', and C'' on the imaging surface 231 corresponding to the plurality of divided intermediate images are different between the regions.

Therefore, the memory 5 according to this embodiment stores in advance the amount of aberration occurring in each region as a table. Aberrations include aberrations that differ from region to region, such as magnification error, spherical aberration, coma aberration, and astigmatism. A table may be prepared in the memory 5 for each type of aberration. For example, regarding the amount of distortion, the information acquisition unit 43 of the processing circuit 4 performs image processing using distortion correction data (table) for each area, and divides the image into segments formed on the optical sensor 23 with different amounts of distortion in each area. Correct the distortion of the image. In other words, the information acquisition unit 43 performs the correction using image processing parameters suitable for each of the plurality of regions. The information acquisition unit 43 synthesizes the corrected information and generates one imaged information (information regarding the imaged surface).

In this way, the information processing device 1 according to the present embodiment performs image processing to correct optical aberrations occurring in each divided image on the imaging surface 231 corresponding to a plurality of divided intermediate images for each region. . Furthermore, in the image processing, the divided images are corrected using mutually different parameters for at least two of the plurality of regions corresponding to the plurality of divided intermediate images. Therefore, even if the amount of distortion is different between the lenses arranged at the periphery of the first lens array 221 and the lenses arranged at the center, they can be corrected individually, so that they are correct as the imaged object. You can get information.

(Ninth embodiment)
Note that in the first lens array 221, the boundaries between the respective condensing elements cannot be made completely zero, so data may be missing. Therefore, the missing data may be supplemented by performing machine learning on the captured image to generate imaged information. In other words, the information acquisition device 1 according to the present embodiment has learned parameters such that, in response to input of image data with missing border data, the information acquisition device 1 outputs image data in which the missing border data is supplemented. A machine learning model may be used to correct each of the plurality of segmented images. The input/output data of the machine learning model is not limited to image data of each region, but may be a light reception signal of each region. The information acquisition unit 43 inputs the time-divided light reception signal from the optical sensor 23 or the image data of each region generated based on the light reception signal to the machine learning model. The information acquisition unit 43 also acquires the output of the machine learning model. Such a machine learning model may be trained using, for example, a received light signal or image data obtained using a high-resolution optical sensor as correct data. Note that the learning data on the input side may be generated from a light reception signal or image data obtained using a high-resolution optical sensor.

Note that the machine learning model is a composite function with parameters that is a composite of multiple functions, and is defined by a combination of multiple adjustable functions and parameters. A machine learning model may be any parameterized composite function defined by a combination of multiple tunable functions and parameters. Note that the machine learning model may be a convolutional neural network (CNN) or a fully connected network. Note that it is assumed that the parameters of the learned machine learning model are stored in the memory 5, for example.

(Tenth embodiment)
FIG. 14 is a sectional view showing another example of the configuration of the imaging system 2 and illumination system 3 in FIG. 1. In the configuration of FIG. 14, the light source 32 of the illumination system 3 includes a first light source LS1 that emits light with a first wavelength λ1 and a second light source LS2 that emits light with a second wavelength λ2. Here, the light of wavelength λ1 refers to light whose wavelength spectrum has a peak wavelength of wavelength λ1. Similarly, light with wavelength λ2 refers to light whose wavelength spectrum has a peak wavelength of wavelength λ2. The first light source LS1 and the second light source LS2 illuminate the same area on the imaged surface. The first optical system 21 forms an image of light from a projection surface (imaged surface) onto an imaging surface 231 of a photosensor 23 disposed on an imaging surface that is a conjugate plane of the imaged surface. The second optical system 22 is arranged on the optical path of the first optical system 21 and deflects light from the imaged surface in a direction according to the wavelength of the light. Specifically, the second optical system 22 is arranged between the first optical system 21 and the optical sensor 23. Further, the second optical system 22 is, for example, a flat diffraction grating. The second optical system 22 as a diffraction grating has a first-order deflection angle α1 for the first wavelength λ1 and a first-order deflection angle α2 for the second wavelength λ2, It is assumed that the design is such that the intensity of the first-order deflection angle is the strongest for any wavelength. Note that as the diffraction grating, a blazed type diffraction grating that emits less zero-order light is desirable.

As shown in FIG. 14, the second optical system 22 serving as a diffraction grating separates the light with the first wavelength λ1 and the light with the second wavelength λ2 out of the light from the same area on the imaged surface. is imaged at a slightly shifted position on the imaging surface 231 of the optical sensor 23. The amount of deviation on this imaging surface 231 is assumed to be Δ. At this time, the light that is imaged on the same region on the imaging surface 231 by the first optical system 21 and the second optical system 22 is light from different regions among the regions on the imaging surface. In other words, the second optical system 22 according to the present embodiment shifts the area of the imaged surface that can be transmitted by the first optical system 21 by an amount corresponding to the amount of deviation Δ on the imaging surface 231. This is an optical system that images light from a plurality of areas onto the same area of the imaging surface 231.

The diffraction angle of the second optical system 22 as a diffraction grating is designed so that the amount of deviation Δ on the imaging surface 231 does not become a value that is an integral multiple of the pixel pitch of the optical sensor 23. As an example, the amount of deviation Δ on the imaging surface 231 is a half pixel pitch of the optical sensor 23.

The illumination control unit 41 sequentially causes the light sources LS1 and LS2 to emit light to sequentially illuminate the same area on the imaged surface. The imaging control unit 42 generates a light reception signal corresponding to the light from the imaged surface while the area is illuminated, that is, at a timing synchronized with the timing at which the illumination control unit 41 irradiates the imaged surface with light of each wavelength. is output to the optical sensor 23. The information acquisition unit 43 acquires the time series of the light reception signal from the optical sensor 23 by time-division according to the illumination timing of the light of each wavelength as information regarding the imaged surface for each wavelength of light. The information acquisition unit 43 combines information regarding the imaged surface for each wavelength of light, and acquires data about twice the number of pixels of the optical sensor 23.

According to the configuration, it is possible to read an image formed between each pixel by light of one wavelength and a light formed at each pixel by light of the other wavelength, so that the physical It is possible to increase the resolution to a higher resolution.

Note that the amount of deviation Δ on the imaging surface 231 is not limited to a half-pixel pitch, but may be a 1.5-pixel pitch, a 2.5-pixel pitch, or the like. In the case of these deviation amounts Δ, it may be corrected in processing after information acquisition. Furthermore, when correcting in post-processing, even in the case of two wavelengths, the deviation amount Δ does not need to be in units of 0.5 pixel pitch. Further, correction in post-processing may be performed in the same manner as in super-resolution.

In addition, in this embodiment, the case where two wavelengths are used is illustrated, but a plurality of wavelengths of three or more may be used. If the amount of deviation Δ on the imaging surface 231 is set to a value that is the reciprocal of the number of wavelengths used, it is possible to easily interpolate between pixels. Furthermore, since a flat diffraction grating is used as the second optical system 22, it can be placed in the optical system without changing the configuration of the first optical system 21 (imaging optical system).

(Eleventh embodiment)
FIG. 15 is a cross-sectional view showing another example of the configuration of the imaging system 2 and illumination system 3 in FIG. 14. The first optical system 21 according to this embodiment has at least two optical elements. In the example shown in FIG. 15, the first optical system 21 includes a first optical element 211 and a second optical element 212. For example, lenses can be used as the first optical element 211 and the second optical element 212. The second optical system 22 according to this embodiment is arranged between at least two optical elements of the first optical system 21. It is desirable that the second optical system 22 be placed at the aperture position of the first optical system 21. According to this configuration, since the second optical system 22 is arranged inside the first optical system 21, the requirement for placement accuracy of the second optical system 22 can be relaxed.

Further, according to the technology according to this embodiment, the first optical system 21 and the second optical system 22 can also be formed integrally. According to the integrated first optical system 21 and second optical system 22, it is possible to improve the holding accuracy of each element, and therefore the quality of imaging performance can be improved.

Note that in the tenth embodiment and the eleventh embodiment, the transmission type second optical system 22 is illustrated, but the present invention is not limited to this. The second optical system 22 may be of a transmission type or a reflection type. Further, although the most suitable diffraction grating is illustrated as the second optical system 22, the present invention is not limited thereto. The second optical system 22 may be any optical element that has wavelength dependence. As the second optical system 22, an optical element having high dispersion characteristics can also be used as appropriate.

Note that in the first to ninth embodiments, when a color filter or the like having different wavelength selectivity for each region is provided at the position where each divided intermediate image is formed, or in the tenth to eleventh embodiments, Information for each wavelength may be acquired by color-separating the light reception signal (image data) from the optical sensor 23. In this case, it is not necessary to sequentially illuminate the illumination light or to acquire the light reception signal in a time-division manner.

Note that the techniques according to each embodiment can be combined as appropriate. For example, the configurations of the imaging system 2 and illumination system 3 according to the tenth to eleventh embodiments may be applied to a TOF sensor.

The information acquisition device 1 according to each embodiment includes a control device such as a CPU, a storage device such as a ROM or a RAM, an external storage device such as an HDD or a CD drive device, a display device such as a display device, and a keyboard and a mouse. It may be equipped with an input device such as, and may have a hardware configuration using a normal computer.

The information acquisition program executed by the information acquisition device 1 according to each embodiment is an installable or executable file on a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk). It is recorded and provided on a computer-readable recording medium such as.

Further, the information acquisition program executed by the information acquisition device 1 according to each embodiment may be stored on a computer connected to a network such as the Internet, and may be provided by being downloaded via the network. . Further, the information acquisition program executed by the device of this embodiment may be configured to be provided or distributed via a network such as the Internet.

Further, the information acquisition program executed by the information acquisition device 1 according to each embodiment may be provided by being incorporated in a ROM or the like in advance.

The information acquisition program executed by the information acquisition device 1 according to each embodiment has a module configuration including the above-mentioned units (lighting control unit 41, imaging control unit 42, and information acquisition unit 43), and is based on actual hardware. When the CPU (processor) reads and executes the information acquisition program from the storage medium, each of the above units is loaded onto the main storage device, and the lighting control unit 41, imaging control unit 42, and information acquisition unit 43 are stored in the main storage device. It is now generated on top.

1 Information acquisition device 2 Imaging system 3 Illumination system 4 Processing circuit 5 Memory 21 First optical system 22 Second optical system 23 Optical sensor 31, CL1, CL2, CL3 Illumination optical system 32, LS1, LS2, LS3 Light source 33 Light Shutter 41 Illumination control section 42 Imaging control section 43 Information acquisition section 211 First optical element 212 Second optical element 221 First lens array 222 Second lens array 223 Imaging lens 231 Imaging surface

Japanese Patent Application Publication No. 2016-035398

Claims (15)

an optical sensor that acquires light delay information and outputs a light reception signal according to the intensity distribution of light incident on an imaging surface;
a first optical system that transmits light from a surface to be imaged to a conjugate surface of the surface to be imaged;
a second optical system that images light from a plurality of regions of the imaged surface that can be transmitted by the first optical system onto the same region of the imaged surface;
An information acquisition device comprising: a processing circuit that acquires information regarding the imaged surface based on the time-divided light reception signal. The first optical system forms an intermediate image on a conjugate plane of the imaged surface on the optical path from the imaged surface to the imaged surface via the first optical system,
The second optical system is disposed on the optical path from the first optical system to the imaging surface, divides the intermediate image, and forms a plurality of divided intermediate images, respectively, on the same area of the imaging surface. do,
The information acquisition device according to claim 1. The information acquisition device according to claim 2, wherein each of the plurality of divided intermediate images is formed by light from each corresponding region of the imaged surface. further comprising an illumination system capable of illuminating each region of the imaged surface corresponding to each of the plurality of divided intermediate images,
The processing circuit time-divisionally acquires the light reception signal from the optical sensor at a timing synchronized with the illumination timing at which the illumination system illuminates each region.
The information acquisition device according to claim 3. 5. The information acquisition device according to claim 4, wherein the illumination system includes an illumination light source with an emission wavelength that includes an infrared region. further comprising an optical shutter that selectively passes light from regions of the imaged surface corresponding to the plurality of divided intermediate images, respectively;
The processing circuit time-divisionally acquires the light reception signal from the optical sensor at a timing that is synchronized with the timing at which the light from each region passes through the optical shutter.
The information acquisition device according to any one of claims 3 to 5. The second optical system is
one condensing element array in which condensing elements respectively corresponding to the plurality of divided intermediate images are arranged;
Claims 2 to 6 further comprising: one or more condensing elements that respectively focus light corresponding to the plurality of divided intermediate images that have passed through the condensing element array onto the same area of the imaging surface. The information acquisition device according to any one of the above. 8. The information acquisition device according to claim 7, wherein in the condensing element array, each condensing element is arranged in a rectangular shape or a honeycomb shape. 9. The information acquisition device according to claim 8, wherein in the condensing element array, adjacent condensing elements have different sizes. 10. The information acquisition device according to claim 9, wherein in the condensing element array, the size of the condensing elements in the central part is larger than the size of the condensing elements in the peripheral part. The processing circuit corrects optical aberrations occurring in the divided images formed on the imaging surface, which correspond to the plurality of divided intermediate images, for each region corresponding to each of the plurality of divided intermediate images. The information acquisition device according to any one of claims 2 to 10, which performs image processing. The information acquisition device according to claim 11, wherein the processing circuit corrects at least two of the plurality of regions corresponding to the plurality of divided intermediate images using mutually different parameters in the image processing. . The second optical system is disposed on the optical path of the first optical system, and deflects the light from the imaged surface in a direction according to the wavelength of the light,
The time-divided light reception signal is information regarding the imaged surface for each wavelength of the light,
The information acquisition device according to claim 1. The information acquisition device according to claim 13, wherein the second optical system is a diffraction grating. The first optical system has at least two optical elements,
the second optical system is arranged between the at least two optical elements;
The information acquisition device according to claim 13 or 14.

Images