US20210132382A1 - Distance measurement device, head-mounted display device, mobile information terminal, image display device, surroundings monitoring system, and distance measurement method - Google Patents
Distance measurement device, head-mounted display device, mobile information terminal, image display device, surroundings monitoring system, and distance measurement method Download PDFInfo
- Publication number
- US20210132382A1 US20210132382A1 US16/492,094 US201716492094A US2021132382A1 US 20210132382 A1 US20210132382 A1 US 20210132382A1 US 201716492094 A US201716492094 A US 201716492094A US 2021132382 A1 US2021132382 A1 US 2021132382A1
- Authority
- US
- United States
- Prior art keywords
- polarization
- distance measurement
- polarization beam
- image
- imaging
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B5/00—Optical elements other than lenses
- G02B5/18—Diffraction gratings
- G02B5/1876—Diffractive Fresnel lenses; Zone plates; Kinoforms
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C3/00—Measuring distances in line of sight; Optical rangefinders
- G01C3/02—Details
- G01C3/06—Use of electric means to obtain final indication
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
- G01S17/894—3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
- G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/481—Constructional features, e.g. arrangements of optical elements
- G01S7/4816—Constructional features, e.g. arrangements of optical elements of receivers alone
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/499—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00 using polarisation effects
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/28—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for polarising
- G02B27/283—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for polarising used for beam splitting or combining
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/28—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for polarising
- G02B27/283—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for polarising used for beam splitting or combining
- G02B27/285—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for polarising used for beam splitting or combining comprising arrays of elements, e.g. microprisms
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/40—Optical focusing aids
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
- G02B7/28—Systems for automatic generation of focusing signals
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
- G02B7/28—Systems for automatic generation of focusing signals
- G02B7/34—Systems for automatic generation of focusing signals using different areas in a pupil plane
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
- G02B7/28—Systems for automatic generation of focusing signals
- G02B7/36—Systems for automatic generation of focusing signals using image sharpness techniques, e.g. image processing techniques for generating autofocus signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
Definitions
- the present invention relates to a distance measurement device, a head-mounted display device, a mobile information terminal, an image display device, surroundings monitoring system, and a distance measurement method.
- a passive distance measurement device typified by a lens focal method of performing distance measurement on the basis of focus blur, or a stereo method of performing distance measurement on the basis of the disparity among a plurality of imaging devices.
- a multi-focus camera of the preset invention includes a lens of an imaging optical system, a first imaging device which out of light incident from a subject through the lens, absorbs the light at a predetermined absorption rate by an organic photoelectric conversion film and photoelectrically converts it to capture an image of a subject, a second imaging device which photoelectrically convers the light transmitted through the first imaging device to capture an image of the subject, and a control section which controls an imaging operation by the first imaging device and the second imaging device.
- the control section has a distance measurement calculation section which calculates a distance from the subject to the lens on the basis of the difference in blur amount between the images of the subject captured by the first imaging device and the second imaging device (summary abstract)”.
- the multi-focus camera disclosed in Patent Literature 1 has realized simultaneous imaging of a plurality of images necessary for the lens focal method by disposing a plurality of imaging devices spaced apart from each other along the optical axis of the lens. Therefore, there is an actual situation that there are restrictions on miniaturization of the multi-focus camera due to the need to secure the distance between the first imaging device and the second imaging device.
- the present invention aims to provide a distance measurement technique which further reduces constraints on a size reduction.
- a distance measurement device measuring, based on a plurality of images, a distance to a target to be imaged captured into the corresponding images, which includes an imaging optical system, an imaging device which images incident light from the imaging optical system, and a distance measurement section which measures the distance from the imaging device to the target to be imaged on the basis of the plurality of images captured by the imaging device, and which is characterized in that the imaging optical system includes a polarization beam splitting section which splits a first polarization beam having a first polarization direction and a second polarization beam having a second polarization direction substantially orthogonal to the first polarization direction and allows the polarization beam splitting section to be incident on the imaging device, and in that the imaging device receives the first polarization beam to capture a first polarization image, and receives the second polarization beam to capture a second polarization image, and the distance measurement section measures the distance to the object to be
- FIG. 1 is a diagram showing a configuration of a distance measurement device in the present invention
- FIG. 2 is a diagram showing a configuration of an imaging optical system and an imaging device in a first embodiment
- FIG. 3 is a diagram showing a configuration of a lens in the first embodiment
- FIG. 4 is a diagram showing a configuration of a polarization beam splitting section and an imaging device in the first embodiment
- FIG. 5A is a diagram showing the manner in which a target to be imaged is captured by a first polarization beam in the distance measurement device of the present invention
- FIG. 5B is a diagram showing the manner in which a target to be imaged is captured by a second polarization beam in the distance measurement device of the present invention
- FIG. 6A is a diagram showing a point spread function of the first polarization beam captured by the imaging optical system
- FIG. 6B is a diagram showing a point spread function of the second polarization beam captured by the imaging optical system
- FIG. 7A is a diagram showing a configuration of an imaging optical system and an imaging device where a polarization control element is controlled in such a manner that a polarization beam of light transmitted through the polarization control element becomes a first polarization beam in a second embodiment;
- FIG. 7B is a diagram showing a configuration of the imaging optical system and the imaging device where the polarization control element is controlled in such a manner that the polarization beam of light transmitted through the polarization control element becomes a second polarization beam in the second embodiment;
- FIG. 8 is a diagram showing a configuration of the imaging optical system and the imaging device in the second embodiment
- FIG. 9 is a diagram showing a configuration of a lens in a third embodiment.
- FIG. 10 is a diagram showing a configuration of an imaging optical system and an imaging device in a fourth embodiment
- FIG. 11A is a diagram showing a configuration of a first polarization-dependent Fresnel zone plate in the fourth embodiment
- FIG. 11B is a diagram showing a configuration of a second polarization-dependent Fresnel zone plate in the fourth embodiment
- FIG. 12 is a diagram showing a configuration of an imaging optical system and an imaging device in a fifth embodiment
- FIG. 13 is a diagram showing a distance measurement processing flow in the first embodiment
- FIG. 14 is a diagram showing a processing flow for a PSF size adjustment in the first embodiment
- FIG. 15 is a diagram showing an example of a configuration where the distance measurement device is used in an image display device
- FIG. 16 is a diagram showing an example of a configuration where the distance measurement device is used in the image display device
- FIG. 17 is a diagram showing a processing flow for a gesture input in the image display device using the distance measurement device
- FIG. 18A is a diagram showing an image to be captured at the time of hand recognition processing by a user in the image display device using the distance measurement device;
- FIG. 18B is a diagram showing an image to be processed at the time of feature point extraction processing of a user's hand in the image display device using the distance measurement device;
- FIG. 18C is a diagram showing an image to be processed at the time of distance measurement region determination processing in the image display device using the distance measurement device;
- FIG. 19 is a diagram showing a configuration where the distance measurement device is used in a surroundings-monitoring system of a vehicle;
- FIG. 20 is a diagram showing a processing flow to determine a distance measurement range on the basis of a traveling state of the vehicle.
- FIG. 21 is a diagram showing a configuration in which two cameras are provided as distance measurement devices in a stereo method distance measurement device having a variable base length.
- a wearable terminal such as a head-mounted display or the like directed to the application to an augmented reality (Augmented Reality, AR) technology and a virtual reality (Virtual Reality, VR) technology has recently been made into a product.
- AR Augmented Reality
- VR Virtual Reality
- a gesture input to recognize the motion of a hand.
- a spatial recognition function to grasp a positional relation of an object in space in order to superimpose visual information into the space.
- a distance measurement technology of measuring depth information of the object is required for implementation of these.
- a distance measurement technology typified by a time of flight (Time-of-Flight, TOF) method has been utilized therefor.
- An active distance measurement device typified by the TOF method requires a light source irradiating an object with light.
- a distance measurement device according to the present embodiment to be described below is a so-called passive distance measurement device which does not require the light source irradiating the object with the light.
- FIG. 1 shows one example of a configuration of a distance measurement device 101 in the present invention.
- the distance measurement device 101 captures an image of a target to be imaged by an imaging device 105 through an imaging optical system 102 .
- the imaging optical system 102 has a point spread function adding section 103 which adds different point spread functions to two orthogonal polarization beams respectively, and a polarization beam splitting section 104 which splits incident light having a first polarization direction and incident light having a second polarization direction substantially perpendicular to the first polarization direction.
- substantially perpendicular referred to here is not intended to limit when a polarization direction axis of the incident light having the first polarization direction, and a polarization direction axis of the incident light having the second polarization direction are exactly 90°, and includes such a case that the angle formed by the polarization direction axis of the incident light having the first polarization direction and the polarization direction axis of the incident light having the second polarization direction is included in an allowable accuracy range in which they are regarded as being orthogonal to each other centering around 90 degrees.
- point spread function is a function indicating a response to a point light source of the optical system.
- images each obtained when a light beam generated from one point of a target to be imaged is captured by the imaging device there occurs a difference in the degree of blurring between the captured images where the point spread functions of the imaging optical system differ.
- the distance is measured by using the difference in the image blur degree.
- Image data captured by the imaging device 105 is transferred to an image processing section 106 .
- a distance measurement section 107 generates distance information on the basis of the image, and a display image forming section 108 generates an image for display on which the distance information is reflected.
- the distance information and display image generated in the image processing section 106 are output to a control section 109 and may be used for usage applications other than the generation of the display image, e.g., an operation input to be described later.
- the input side of light in the imaging optical system 102 further includes an aperture stop 202 configured to include, e.g., a plurality of movable vanes, and a aperture stop driving section 206 for driving the aperture stop 202 to change an aperture size.
- the control section 109 sends a drive control signal to the aperture stop driving section 206 to drive the aperture stop 202 and thereby change the aperture size of the imaging optical system 102 to perform an optical adjustment. Further, the control section 109 performs control of execution timing for each processing in the imaging device 105 and the image processing section 106 , and communications with an external device such as an image display device.
- a CMOS image sensor, a CCD image sensor and the like can be used as the imaging device 105 .
- FIG. 2 shows one example of a configuration of the imaging optical system 102 and the imaging device 105 in the present embodiment.
- Incident light 201 incident on the distance measurement device 101 according to the present embodiment enters the point spread function adding section 103 in the imaging optical system 102 and transmits an aperture part 202 a formed by the aperture stop 202 in the point spread function adding section 103 , and a lens 203 .
- a first polarization beam 204 and a second polarization beam 205 perpendicular to each other respectively transmit the point spread function adding section 103 in a state in which they are added with different point spread functions.
- the first polarization beam 204 and the second polarization beam 205 are received by the imaging device 105 through the polarization beam splitting section 104 , so that an image of the target to be imaged is recorded.
- the polarization beam splitting section 104 and the imaging device 105 have been described away from each other for convenience of explanation in FIG. 2 , they preferably approximate each other, and both may be configured by laminating them.
- FIG. 3 shows one example of a configuration of the lens 203 in the present embodiment.
- a traveling direction of the incident light 201 is assumed to be a direction perpendicular to the paper
- a polarization direction of the first polarization beam 204 is assumed to be parallel to an axis 301
- a polarization direction of the second polarization beam 205 is assumed to be parallel to an axis 302 .
- the lens 203 is made of a birefringent material.
- a refractive index to the first polarization beam 204 parallel to the axis 301 is n 1
- a refractive index to the second polarization beam 205 parallel to the axis 302 is n 2 . Therefore, a focal length of the lens 203 becomes lengths respectively different from each other with respect to the first polarization beam 204 and the second polarization beam 205 .
- the lens 203 is a spherical plane-convex lens having a curvature radius R
- a focal length f 1 to the first polarization beam 204 and a focal length f 2 to the second polarization beam 205 can be expressed by equations 1 and 2 respectively. Therefore, for example, when n 1 is larger than n 2 , f 1 becomes a value smaller than f 2 .
- FIG. 4 shows one example of a configuration of the polarization beam splitting section 104 and the imaging device 105 in the present embodiment.
- the polarization beam splitting section 104 is a polarizer array in which first polarizers 401 , 403 , 405 , 407 , and 409 which transmit the first polarization beam 204 and shield the second polarization beam 205 , and second polarizers 402 , 404 , 406 , and 408 which shield the first polarization beam 204 and transmit the second polarization beam 205 are arranged.
- the polarization beam splitting section 104 and the imaging device 105 are disposed in such a manner that the beams transmitted through the first polarizers or second polarizers 401 through 409 are respectively imaged by pixels 410 through 418 of the imaging device 105 . Therefore, the pixels 410 , 412 , 414 , 416 , and 418 capture only the first polarization beam 204 , and the pixels 411 , 413 , 415 , and 417 capture only the second polarization beam 205 .
- FIG. 4 describes a part of the pixels which constitute the imaging device 105 .
- the number of pixels is not limited to that shown in FIG. 4 .
- the pixels 410 through 418 may be single monochrome image sensors respectively.
- the respective pixels of the pixels 410 through 418 may be Bayer filter-type image sensors respectively having one or more pixels corresponding to RGB.
- the first polarization beam 204 and the second polarization beam 205 can be incident on the imaging device 105 at different focal lengths respectively and individually capture a first polarization image obtained by capturing the first polarization beam 204 by the imaging device 105 and a second polarization image obtained by capturing the second polarization beam 205 respectively by the imaging device 105 . It is therefore possible to simultaneously capture the first polarization image and the second polarization image being two images different in blur amount and execute a distance measurement by a Depth from defocus (DFD) method being one of lens focal methods.
- DMD Depth from defocus
- the lens focal method is accompanied by a problem that since it needs at least two or more images different in blur amount, imaging is performed plural times while changing the distance between the lens and the imaging device 105 in the conventional DFD method, and the measurement time is long as compared with other distance measurement methods. Further, the configuration described in Patent Literature 1 has a problem in that images different in blur amount can be simultaneously captured by using the two or more imaging devices 105 , but power consumption is increased because the number of imaging devices 105 increases.
- a small-sized/low power consumption distance measurement device which executes the distance measurement by the DFD method at high speed because the two first polarization image and second polarization image different in blur amount can simultaneously be captured by using one imaging device 105 .
- the number of polarizers that constitute the polarization beam splitting section 104 and how to arrange them are not limited to those shown in FIG. 4 .
- the first polarizers 401 , 403 , 405 , 407 , and 409 , and the second polarizers 402 , 404 , 406 , and 408 may be those arranged in a stripe form.
- a polarizer array using a photonic crystal may be used for the polarization beam splitting section 104 .
- the number of pixels that constitute the imaging device 105 is not limited to that shown in FIG. 4 .
- the first polarization image and the second polarization image can be generated by performing interpolation processing on the basis of a light intensity signal received by the adjacent pixels.
- the second polarization image by way of example, the pixel 414 of FIG.
- the second polarization image can be generated.
- the first polarization image can be generated by performing similar interpolation processing even on the image of the first polarization beam 204 .
- the lens 203 in the present embodiment may be a lens using a birefringent crystal such as a crystal, calcite, magnesium fluoride, barium metaborate, or the like or may be a plastic lens indicative of birefringence.
- a birefringent crystal such as a crystal, calcite, magnesium fluoride, barium metaborate, or the like
- plastic lens indicative of birefringence may be a lens using a birefringent crystal such as a crystal, calcite, magnesium fluoride, barium metaborate, or the like.
- FIG. 5 is a diagram describing the manner in which an object 501 to be imaged is captured in the distance measurement device of the present embodiment.
- the distance between the object 501 and the lens 203 is assumed to be u, and the distance between the lens 203 and the imaging device 105 is assumed to be s. Further, the diameter of the aperture part 202 a is assumed to be D 0 .
- FIG. 5A shows the manner in which the incident light of the first polarization beam 204 is captured by the imaging device 105 . Light of the first polarization beam 204 resulting from one point of the object 501 is transmitted through the lens 203 and then imaged on an imaging point 502 focused with the lens having the focal length f 1 .
- FIG. 5B shows the manner in which the incident light of the second polarization beam 205 is imaged by the imaging device 105 .
- Light of the second polarization beam 205 resulting from one point of the object 501 is transmitted through the lens 203 and then imaged on an imaging point 503 focused with the lens having the focal length f 2 .
- the image of the second polarization beam 205 becomes an image stronger in blur than that of the first polarization beam 204 .
- FIG. 6 illustrates a point spread function captured by the imaging optical system of the present embodiment.
- An image 601 is an example of an image when imaging a point spread function 602 of the first polarization beam 204 .
- An image 603 is an example of an image when imaging a point spread function 604 of the second polarization beam 205 .
- a two-dimensional Fourier transform may be utilized as a method of extracting the point spread function from the two convoluted images.
- the point spread function may be calculated by using a Spatial Domain Convolution/Deconvolution Transform described in the following Literature.
- the imaging device 105 may be disposed in such a manner that either the first polarization image or the second polarization image is formed on the imaging device 105 .
- one of the two images captured for distance measurement is displayed as it is and thereby usable as a display image for a user in AR systems or the like.
- the PSF is calculated from the two polarization images and thereafter blurring of the captured image is removed based on information of the PSF, whereby a display image for the user may be generated.
- a processing S 1301 performs initialization of a part related to the distance measurement such as an adjustment of the PSF size.
- the first polarization beam 204 is captured to generate a first raw (raw) polarization image
- the second polarization beam 205 is captured to generate a second raw polarization image.
- interpolation processing is performed on the first raw polarization image and the second raw polarization image captured in the processing S 1302 respectively to generate a first polarization image and a second polarization image.
- a PSF is extracted from the first polarization image and the second polarization image generated in the processing S 1303 .
- a processing S 1305 the distance to a target object is calculated based on the PSF extracted in the processing S 1304 .
- Whether the size of the PSF is appropriate can be determined from the change in the PSF extracted before and after the change of the diameter of the aperture part 202 a , for example.
- the diameter of the aperture part 202 a is reduced, the PSF of an image captured by the imaging device 105 becomes small.
- the diameter of the aperture part 202 a is made small where the size of the PSF becomes a predetermined size or more than the imaging device 105 , the PSF size extracted from the image remains unchanged or increases. This is because when the PSF is a predetermined size or more, at least one of the two images used for DFD does not cause a distinct change in terms of how to blur even if the diameter of the aperture part 202 a is made small.
- a processing S 1401 the diameter of the aperture part 202 a is set.
- a first polarization image and a second polarization image are generated to extract a PSF.
- the term “the generation of the first polarization image” may be processing of capturing the first raw polarization image and performing interpolation processing thereto to generate the first polarization image, or may be processing in which the first polarization image is obtained in a state of having no defects in pixels of the imaging device 105 upon imaging the first polarization beam 204 as in a second embodiment to be described later. The same applies even to the second polarization image.
- a processing S 1405 the PSF extracted in the processing S 1402 and the PSF extracted in the processing S 1404 are compared with each other to determine whether the current PSF is appropriate. For example, when the diameter of the aperture part 202 a at the capturing in the processing S 1402 is larger than the diameter of the aperture part 202 a at the capturing in the processing S 1404 , the PSF extracted in the processing S 1402 is determined to be appropriate if being larger than the PSF extracted in the processing S 1404 (S 1405 /Yes), and the processing is terminated.
- the PSF extracted in the first processing S 1404 and the PSF extracted in the second processing S 1404 are compared with each other to re-determine whether the PSF size is appropriate.
- the processing of FIG. 14 may be done only once within the processing S 1301 of FIG. 13 .
- the processing may be done every time when shifting from the processing S 1307 to the processing S 1302 or may be done every several times. Further, when the processing of FIG. 14 is done upon shifting from the processing S 1307 of FIG. 13 to the processing S 1302 , the distance information calculation of the processing S 1405 is performed using the PSF sizes extracted in the processing S 1402 and the processing S 1404 , so that the processing S 1302 to the processing S 1304 may be omitted.
- FIG. 21 shows an example of a configuration in which two cameras are provided as the distance measurement devices 101 of the present embodiment in a stereo method distance measurement device having a variable base length.
- the two distance measurement devices 101 are fixed to a camera holding section 2101 .
- the camera holding section 2101 has a telescopic function and changes the base length according to the required distance measurement accuracy.
- FIG. 15 shows an example of an external appearance in the case of using the distance measurement device 101 of the present embodiment in the image display device 1501 .
- FIG. 16 shows an example of a configuration in the case of using the distance measurement device 101 of the present embodiment in the image display device 1501 .
- the image display device 1501 is configured to include an image display section 1502 , an audio input section 1503 , an input switch 1504 , a personal computer held by a server on a network or a user, a communication section 1604 which communicates with a mobile information terminal or the like, an information storage section 1605 storing information therein, and a control section 1602 .
- the distance measurement device 101 is arranged so as to be able to measure the forward when the user wears the image display device 1501 as an example.
- the control section 1602 controls the operations of the image display section 1502 , the distance measurement device 101 , the communication section 1604 , and the information storage section 1605 on the basis of inputs from the distance measurement device 101 , the audio input section 1503 , the input switch 1504 , the communication section 1604 , and the information storage section 1605 .
- FIG. 17 shows a processing flow for a gesture input in the image display device 1501 using the distance measurement device 101 .
- the image display device 1501 starts gesture recognition processing in the distance measurement device 101 in accordance with an instruction from the control section 1602 .
- Start timing of the gesture recognition processing may be the immediately following the startup of the image display device 1501 , or may always be done in accordance with the inputs from the audio input section 1503 , the input switch 1504 , the mobile information terminal connected to the communication section 1604 , etc.
- a part related to the distance measurement is initialized.
- a processing S 1702 the hand of the user is recognized from an image captured by the imaging device 105 to extract feature points.
- a distance measurement region is determined based on the feature points extracted in the processing S 1702 .
- a processing S 1704 the distance measurement of the distance measurement region determined in the processing S 1703 is performed to acquire the three-dimensional coordinates of the respective feature points.
- the feature point extracting processing and the distance measurement region determining processing are not performed each time the feature point coordinate acquiring processing is executed, and may be executed once in several times. Further, a waiting time may be deliberately provided between the processing S 1704 and the processing S 1707 .
- FIG. 18 shows an example of an image to be captured and processed by the distance measurement device 101 at the time of execution of the processing S 1702 and the processing S 1703 .
- a feature point 1803 is extracted in the processing S 1702 as shown in FIG. 18B .
- the recognition of the user's hand 1802 is performed based on the color or shape, for example.
- a user registration function is provided to the image display device 1501 to thereby register the color and size of the user's hand 1802 , and a region with high frequency of use to the gesture input on the imaging screen.
- the recognition may be performed based on these registration information to enhance the recognition accuracy of the user's hand 1802 .
- the feature point 1803 for example, the joints of the user's hand 1802 , etc. are used.
- a distance measurement region 1804 is determined in such a manner that depth information of the feature point 1803 can be acquired ( FIG. 18C ).
- the load of calculation processing can be reduced because it is not necessary to perform the distance measurement processing for all pixels of the imaging device 105 .
- an image obtained by superimposing the feature points and the distance measurement region may be displayed to the user at the image display section 1502 , or the distance information to the respective feature points may be superimposed and displayed.
- the distance for example, numeric values may be superimposed as they are, or the distance may be displayed as isolines or isolines stepwise colored according to the distance.
- the head-mounted display is used as the image display device, and the input operation to the head-mounted display is performed with the gesture of the user's hand to calculate the amount of movement of the feature point of the hand by the distance measurement device according to the present embodiment. Then, an operation signal indicating the amount of its movement is output to the control section to update the display image reflecting the movement amount of the feature point.
- the operation to switch the communication destination where the head-mounted display has a function different from the image display device e.g., the head-mounted display has a function to communicate with the external device, or the input operation to perform a volume adjustment where it has an audio output function is performed by the gesture, and the input operation is performed by the control section.
- Other application destinations of the distance measurement device 101 of the present embodiment include, for example, an automobile, an unmanned aircraft such as a drone, a robot, a surgical operation assisting system, a precise shape measuring device, etc.
- the distance measurement device 101 may be mounted in a mobile mechanical device such as an automobile, a drone, a disaster survey robot and an industrial robot, and utilized in collision avoidance by surrounding monitoring.
- Position information and distance measurement information may be uploaded to the server on the network and presented to a remote operator.
- the position information and the distance measurement information uploaded to the server on the network may be shared with another mobile mechanical device and utilized in cooperative position control of the mobile mechanical device in the automatic operation and autonomous navigation.
- FIG. 19 is a diagram in which in an example of a configuration where the distance measurement device 101 is used in a surroundings-monitoring system of an automobile 1901 , a part interlocked with the distance measurement device 101 is extracted.
- the distance measurement device 101 obtains the current traveling state from a speed measurement section 1904 , an electronic charge payment terminal 1905 , a GPS receiver 1906 , etc. through an automobile control section 1902 to determine a distance measurement range.
- FIG. 20 shows an example of a processing flow for determining the distance measurement range, based on the traveling state of the automobile.
- An automobile traveling mode is determined in a processing S 2001 after the engine of the automobile has started.
- the automobile traveling mode may be determined by the speed measurement section 1904 , for example.
- a determination may be made by the GPS receiver 1906 as to whether a traveling speed and a road on which the automobile is traveling at present are subject to a highway or a general road. Further, on the basis of communication records of the electronic charge payment terminal 1905 , a determination may be made as to whether the road on which the automobile is traveling at present is a toll highway or a general road.
- a distance measurement mode is decided based on the determination result in a processing S 2002 .
- the diameter of the aperture part 202 a is set to increase the PSF in order to perform the remote distance measurement with high accuracy during high-speed traveling, and the diameter of the aperture part 202 a is set to decrease the PSF in order to perform the distance measurement at a close range with high accuracy during low-speed traveling.
- the PSF size optimizing processing by the diameter control of the aperture part 202 a described in FIG. 14 is also performed.
- the pixel region used for the distance measurement, of the pixels in the imaging device 105 is changed based on the determination result of the processing S 2001 , for example, the distance measurement may be performed at all pixels during the general road traveling, and the distance measurement may be performed only by some pixels in the imaging device during traveling on the highway. Even in addition to the above, the pixel region used for the distance measurement may be changed in accordance with the shape of the road like intersections, curves, straight lines, etc.
- the distance measurement processing described in FIG. 13 and FIG. 14 is performed in a processing S 2003 , and the distance information to an object around the vehicle is used for collision avoidance control or automatic driving control.
- a processing S 2004 the automobile traveling mode is determined again.
- a processing S 2005 a determination is made as to whether the automobile traveling mode is changed.
- the processing flow returns to the processing S 2003 and the distance measurement is continued.
- the traveling mode is changed (S 2005 /Yes)
- the processing is terminated.
- the processing flow returns to the processing S 2002 , where the distance measurement mode suitable for the traveling state is re-selected.
- the distance information around the vehicles obtained by the distance measurement device 101 of the present embodiment may be displayed on an image display device such as a head-up display to display an image on the automobile and its windshield, an image display screen of a car navigation system, or the like.
- a marker and distance information may be displayed with being superimposed on the forward vehicle and pedestrian.
- a marker superimposed on an object high in the degree of risk of collision may be displayed in color different from the colors of other markers.
- the moving speed of each object may be estimated from a change in the distance between the own vehicle and the object and displayed on the image display section.
- the distance information measured by the respective vehicles are transmitted to the center of the car navigation system or the like and integrated to generate highly accurate positional information of the vehicle/pedestrian on the road. Then, the generated information may be transmitted to each vehicle and displayed on the image display section of the car navigation system or the head-up display.
- the present embodiment is different from the first embodiment in that a polarization beam splitting section is configured of a polarizer and a polarization control element.
- a polarization beam splitting section is configured of a polarizer and a polarization control element.
- FIG. 7 shows an example of a configuration of an imaging optical system 102 and the imaging device 105 in the present embodiment.
- Incident light 201 incident on the imaging optical system 102 enters into a polarization beam splitting section 104 and is transmitted through a polarization control element 702 and a polarizer 701 and then transmitted through the polarization beam splitting section 104 .
- the incident light After passing through the polarization beam splitting section 104 , the incident light is transmitted through a point spread function adding section 103 and imaged by the imaging device 105 .
- the polarizer 701 is an element to take out a specific polarization component of the incident light 201 polarized at random.
- the polarization control element 702 is constituted of, for example, a liquid crystal element and capable of rotating the polarization beam of light to be transmitted through the polarization control element 702 in any direction by voltage control. Therefore, the light transmitted through the polarizer 701 is transmitted through the polarization control element 702 to perform voltage control thereof, whereby the polarization beam entering into the point spread function adding section 103 can be switched to a first polarization beam 204 and a second polarization beam 205 different in polarization direction.
- FIG. 7A shows a configuration of the imaging optical system 102 and the imaging device 105 where the polarization control element 702 is controlled such that the polarization beam of light transmitted through the polarization control element 702 becomes the first polarization beam 204 .
- a lens 203 functions as a lens having a focal length f 1 with respect to the first polarization beam 204 , an image to be captured by the imaging device 105 becomes an image captured by the lens having the focal length f 1 .
- FIG. 7B shows a configuration of the imaging optical system 102 and the imaging device 105 where the polarization control element 702 is controlled in such a manner that the polarization beam of light transmitted through the polarization control element 702 becomes the second polarization beam 205 .
- the lens 203 functions as a lens having a focal length f 2 (>f 1 ) with respect to the second polarization beam 205 , an image to be captured by the imaging device 105 becomes an image captured by the lens having the focal length f 2 .
- the two images (corresponding to the first polarization image and the second polarization image) different in blur amount can be captured, and the distance measurement of the target object by the DFD method can be performed. Further, since the first polarization image and the second polarization image are captured separately twice, the number of pixels of the imaging device 105 usable when capturing each image is increased twice in comparison with the first embodiment, and hence high resolution can be attained.
- the configuration of the imaging optical system and the imaging device 105 in the present embodiment is not limited to the above configuration.
- the configuration of the imaging optical system 102 and the imaging device 105 which is shown in FIG. 8 , may be adopted.
- the polarization beam splitting section 104 is located in the rear of the point spread function adding section 103 . The light incident on the polarization beam splitting section 104 first enters the polarization control element 702 and then enters into the polarizer 701 , followed by being captured by the imaging device 105 .
- the imaging device 105 By controlling the polarization control element 702 so as to allow the polarizer 701 to transmit only the polarization beam condensed at the focal length f 1 , the imaging device 105 is capable of capturing an image captured by the lens having the focal length f 1 . By controlling the polarization control element 702 so as to allow the polarizer 701 to transmit only the polarization beam condensed at the focal length f 2 , the imaging device 105 is capable of capturing an image captured by the lens having the focal length f 2 .
- the polarization beam splitting section 104 is not limited to the combination of the polarizer and the polarization control element.
- the polarization beam splitting section 104 may be, for example, a rotary polarizer capable of controlling the direction of the polarization beam to be transmitted, or the like.
- the present embodiment is different from the first and second embodiments in that a lens 203 is configured of a material with chirality. With this configuration, it is possible to relax lens mounting accuracy at the time of manufacture of the distance measurement device and to improve resistance to the rotation of the lens due to loosening of a lens holding section in use. Further, it is possible to cope even with a zoom function with the rotation of the lens.
- FIG. 9 is a diagram showing an example of a configuration of the lens 203 in the present embodiment.
- the traveling direction of the incident light 201 is assumed to be a direction perpendicular to the paper surface.
- the material constituting the lens 203 is a material with chirality and indicates a different refractive index between right circular polarization and left circular polarization.
- the focal length with respect to the right circular polarization 901 is n 1
- the refractive index with respect to the left circular polarization 902 is n 2
- the radius of curvature of the lens 203 is R
- the polarization beam splitting section 104 may be, for example, a polarizer array in which a polarizer for transmitting only the right circular polarization and a polarizer for transmitting only the left circular polarization are arranged. There may be adopted a configuration in which a 1 ⁇ 4-wavelength plate is disposed in front of the polarizer array described in the first embodiment, or there may be one arranged by combining the liquid crystal element and the polarizer described in the second embodiment.
- the lens 203 of the present embodiment utilizes the birefringence with respect to the right and left circular polarizations, the image having the focal length f 1 and the image having the focal length f 2 are not captured with being mixed even when the lens 203 is rotated with the traveling direction of the incident light 201 as an axis. Therefore, the distance measurement accuracy is not reduced, the lens mounting accuracy at the time of manufacture of the distance measurement device can be relaxed, and the resistance to the rotation of the lens due to loosening of the lens holding section in use can be improved. Further, it is possible to cope even with a zoom function with the rotation of the lens.
- the present embodiment is different from the first and second embodiments in that instead of using the birefringent lens in the point spread function adding section 103 , polarization-dependent Fresnel zone plates 1001 and 1002 are used.
- polarization-dependent Fresnel zone plates 1001 and 1002 are used.
- the combination of the focal lengths of the two orthogonal polarized lights can arbitrarily be selected, and the difference between the blur amounts to be added can be increased, thereby making it possible to improve the accuracy of the distance measurement.
- FIG. 10 shows an example of a configuration of the imaging optical system 102 and the imaging device 105 in the present embodiment.
- the point spread function adding section 103 is configured to include an aperture stop 202 , the polarization-dependent Fresnel zone plate 1001 , and the polarization-dependent Fresnel zone plate 1002 .
- the Fresnel zone plate is a pattern in which a transparent ring belt and an opaque ring belt are alternately arranged concentrically, and serves to focus light.
- the radius r n of the nth ring belt of the Fresnel zone plate to focus the light of a wavelength ⁇ at a focal length f is expressed by an equation 5. Therefore, a Fresnel zone plate of any focal length can be created depending on the design of spacing of each ring belt.
- FIG. 11 shows an example of a configuration of the polarization-dependent Fresnel zone plate in the present embodiment.
- the polarization-dependent Fresnel zone plate 1001 of FIG. 11A is one in which an opaque ring belt 1101 which shields a first polarization beam 204 and transmits a second polarization beam 205 , and a transparent ring belt 1102 which transmits both the first polarization beam 204 and the second polarization beam 205 are arranged concentrically. Since the polarization-dependent Fresnel zone plate 1001 is transparent with respect to the second polarization beam 205 , it condenses only the first polarization beam 204 .
- the polarization-dependent Fresnel zone plate 1002 of FIG. 11B is one in which an opaque ring belt 1103 which transmits the first polarization beam 204 and shields the second polarization beam 205 , and a transparent ring belt 1102 which transmits both of the first polarization beam 204 and the second polarization beam 205 are arranged concentrically. Since the polarization-dependent Fresnel zone plate 1002 is transparent with respect to the first polarization beam 204 , it condenses only the second polarization beam 205 .
- the opaque ring belts 1101 and 1103 which respectively constitute the polarization-dependent Fresnel zone plates 1001 and 1002 utilize, for example, polarizers, photonic crystals, etc.
- the radius of the ring belt corresponding to the nth of the ring belts in the polarization-dependent Fresnel zone plate 1001 is set such that the focal length of the polarization-dependent Fresnel zone plate 1001 becomes f 1
- the radius of the ring belt corresponding to the nth of the ring belts in the polarization-dependent Fresnel zone plate 1002 is set such that the focal length f 2 of the polarization-dependent Fresnel zone plate 1002 is taken, and the polarization-dependent Fresnel zone plate 1001 and the polarization-dependent Fresnel zone plate 1002 are bonded to each other and placed in the point spread function adding section 103 , as shown in FIG.
- the first polarization beam 204 is imaged at the focal length f 1
- the second polarization beam 205 is imaged at the focal length f 2 . Therefore, two images different in blur amount can be captured by imaging the first polarization beam 204 and the second polarization beam 205 by the imaging device 105 after they are split by the polarization beam splitting section 104 , and hence the distance measurement to the target object using the DFD method can be performed.
- the difference between the blur amounts addable to the two orthogonal polarized lights has been limited by the difference in refractive index between the birefringent materials to be used in the first through third embodiments.
- the combination of the focal lengths of the two orthogonal polarized lights can arbitrarily be selected, and the difference between the blur amounts to be added can be increased, thereby making it possible to improve the accuracy of the distance measurement.
- a phase-type polarization-dependent zone plate may be used in which instead of the opaque ring belts, birefringent ring belts to add a phase difference with a light beam transmitted through the transparent ring belts to only either the first polarization beam 204 and the second polarization beam 205 are arranged.
- birefringent ring belts to add a phase difference with a light beam transmitted through the transparent ring belts to only either the first polarization beam 204 and the second polarization beam 205 are arranged.
- the polarization beam splitting method for the first polarization beam 204 and the second polarization beam 205 in the polarization beam splitting section 104 may be used, or the method described in the second embodiment may be used. Further, in the case where the method described in the second embodiment is used, the polarization beam splitting section 104 may be disposed in front of the point spread function adding section 103 . Moreover, instead of using the two polarization-dependent Fresnel zone plates, one of them may be used as a normal imaging lens.
- the present embodiment is different from the first through fourth embodiments in that using a polarization-dependent aperture; two orthogonal polarized lights are narrowed by apertures different in size to thereby add different blur amounts to the two orthogonal polarized lights.
- This configuration leads to reductions in thickness, weight and cost of a point spread function adding section 103 .
- simply by changing the diameters of an aperture part 202 a and a polarization-dependent aperture part 1201 a it is possible to arbitrarily polarize the difference in blur amount between images of a first polarization beam 204 and a second polarization beam 205 , and it is possible to control the distance measurement accuracy according to purposes of use.
- FIG. 12 shows an example of a configuration of an imaging optical system 102 and an imaging device 105 in the present embodiment.
- the point spread function adding section 103 is configured to include an aperture stop 202 , a polarization-dependent aperture stop 1201 , and a lens 203 .
- the polarization-dependent aperture stop 1201 transmits both the first polarization beam 204 and the second polarization beam 205 within a range of a diameter D 0 ′ from the center, for example, but in terms of the incident light thereoutside, the circular polarization-dependent aperture part 1201 a which shields the first polarization beam 204 and transmits the second polarization beam 205 is formed.
- an aperture diameter D 0 ′ of the polarization-dependent aperture part 1201 a is assumed to be smaller than an aperture diameter D 0 of the aperture part 202 a.
- the lens 203 does not require the use of the birefringent material, and functions as a lens having a focal distance f with respect to both of the first polarization beam 204 and the second polarization beam 205 .
- the first polarization beam 204 is narrowed down to the aperture diameter D 0 at the aperture part 202 a and then narrowed down to the aperture diameter D 0 ′ by the polarization-dependent aperture part 1201 a . Thereafter, the first polarization beam 204 is condensed by the lens 203 and transmitted through the polarization beam splitting section 104 , followed by being captured by the imaging device 105 .
- a diameter D 1 of a point spread function of the first polarization beam 204 can be represented by an equation 6 when using D 0 ′, a distance u between the lens 203 and a target object, a distance s between the lens 203 and the imaging device 105 , and f.
- the incident light 201 which is incident on the imaging optical system 102
- light of the second polarization beam 205 is narrowed down to the aperture diameter D 0 at the aperture part 202 a and then transmitted through the polarization-dependent aperture part 1201 a , but the aperture diameter does not change.
- the second polarization beam 205 is condensed by the lens 203 and transmitted through the polarization beam splitting section 104 , followed by being captured by the imaging device 105 .
- a diameter D 2 of a point spread function of the second polarization beam 205 can be represented by an equation 7 when using D 0 , the distance u between the lens 203 and the target object, the distance s between the lens 203 and the imaging device 105 , and the focal length f of the lens 203 .
- the two images different in blur amount can be captured by imaging the first polarization beam 204 and the second polarization beam 205 by the imaging device 105 after splitting the first polarization beam 204 and the second polarization beam 205 by the polarization beam splitting section 104 . It is therefore possible to perform the distance measurement to the target object using the DFD method.
- the polarization-dependent aperture stop 1201 using the circular aperture by the polarizer is used instead of the birefringent lens used in the first embodiment to 3, the point spread function adding section 103 can be made thin and light.
- the polarization-dependent aperture stop 1201 of the present embodiment is a simpler structure than the polarization-dependent Fresnel zone plate of the fourth embodiment and is hence capable of being reduced in manufacturing cost.
- the aperture stop 202 the so-called mechanical aperture stop to open and close the movable blade by using the already-described aperture stop driving section 206 is used, and the aperture part 202 a is formed as a variable aperture, whereby the difference in blur amount between the images of the first polarization beam 204 and the second polarization beam 205 can arbitrarily be polarized simply by changing the size of the aperture diameter D 0 , and the distance measurement accuracy can be controlled according to the purposes of use.
- the wavelength of the imaging light is not limited to visible light. It may be an electromagnetic wave longer in wavelength than infrared, or may be an electromagnetic wave shorter in wavelength than ultraviolet light.
- the combination of the point spread function adding section and the polarization beam splitting section is not limited to the above either.
- the processing order described in the above-described flowcharts is an example, and embodiments in which processing capable of obtaining the same operative effect even if the processing order is exchanged is interchanged before and after the processing are also included in the present invention.
- the application examples of the present invention described with reference to FIGS. 15 through 21 can be applied even to not only the distance measurement device according to the first embodiment, but also the distance measurement devices according to the second to fifth embodiments.
- polarization-dependent Fresnel zone plate 1201 . . . polarization-dependent aperture, 1501 . . . image display device, 1502 . . . image display section, 1503 . . . audio input section, 1504 . . . input switch, 2101 . . . camera holding section.
Abstract
Description
- The present invention relates to a distance measurement device, a head-mounted display device, a mobile information terminal, an image display device, surroundings monitoring system, and a distance measurement method.
- As a distance measurement technique which needs no light source, there has been known a passive distance measurement device typified by a lens focal method of performing distance measurement on the basis of focus blur, or a stereo method of performing distance measurement on the basis of the disparity among a plurality of imaging devices. As an example thereof, there has been described in
Patent Literature 1 that “a multi-focus camera of the preset invention includes a lens of an imaging optical system, a first imaging device which out of light incident from a subject through the lens, absorbs the light at a predetermined absorption rate by an organic photoelectric conversion film and photoelectrically converts it to capture an image of a subject, a second imaging device which photoelectrically convers the light transmitted through the first imaging device to capture an image of the subject, and a control section which controls an imaging operation by the first imaging device and the second imaging device. The control section has a distance measurement calculation section which calculates a distance from the subject to the lens on the basis of the difference in blur amount between the images of the subject captured by the first imaging device and the second imaging device (summary abstract)”. - PTL 1: Japanese Patent Application Laid-Open No. 2013-205516
- The multi-focus camera disclosed in
Patent Literature 1 has realized simultaneous imaging of a plurality of images necessary for the lens focal method by disposing a plurality of imaging devices spaced apart from each other along the optical axis of the lens. Therefore, there is an actual situation that there are restrictions on miniaturization of the multi-focus camera due to the need to secure the distance between the first imaging device and the second imaging device. - Therefore, the present invention aims to provide a distance measurement technique which further reduces constraints on a size reduction.
- The above problem is solved by the invention described in the scope of claims, for example. As an example thereof, there is provided a distance measurement device measuring, based on a plurality of images, a distance to a target to be imaged captured into the corresponding images, which includes an imaging optical system, an imaging device which images incident light from the imaging optical system, and a distance measurement section which measures the distance from the imaging device to the target to be imaged on the basis of the plurality of images captured by the imaging device, and which is characterized in that the imaging optical system includes a polarization beam splitting section which splits a first polarization beam having a first polarization direction and a second polarization beam having a second polarization direction substantially orthogonal to the first polarization direction and allows the polarization beam splitting section to be incident on the imaging device, and in that the imaging device receives the first polarization beam to capture a first polarization image, and receives the second polarization beam to capture a second polarization image, and the distance measurement section measures the distance to the object to be imaged on the basis of the first polarization image and the second polarization image.
- According to the present invention, it is possible to provide a distance measurement technique which further reduces constraints on a size reduction. Incidentally, objects, configurations and effects other than the above are demonstrated in the following embodiments.
-
FIG. 1 is a diagram showing a configuration of a distance measurement device in the present invention; -
FIG. 2 is a diagram showing a configuration of an imaging optical system and an imaging device in a first embodiment; -
FIG. 3 is a diagram showing a configuration of a lens in the first embodiment; -
FIG. 4 is a diagram showing a configuration of a polarization beam splitting section and an imaging device in the first embodiment; -
FIG. 5A is a diagram showing the manner in which a target to be imaged is captured by a first polarization beam in the distance measurement device of the present invention; -
FIG. 5B is a diagram showing the manner in which a target to be imaged is captured by a second polarization beam in the distance measurement device of the present invention; -
FIG. 6A is a diagram showing a point spread function of the first polarization beam captured by the imaging optical system; -
FIG. 6B is a diagram showing a point spread function of the second polarization beam captured by the imaging optical system; -
FIG. 7A is a diagram showing a configuration of an imaging optical system and an imaging device where a polarization control element is controlled in such a manner that a polarization beam of light transmitted through the polarization control element becomes a first polarization beam in a second embodiment; -
FIG. 7B is a diagram showing a configuration of the imaging optical system and the imaging device where the polarization control element is controlled in such a manner that the polarization beam of light transmitted through the polarization control element becomes a second polarization beam in the second embodiment; -
FIG. 8 is a diagram showing a configuration of the imaging optical system and the imaging device in the second embodiment; -
FIG. 9 is a diagram showing a configuration of a lens in a third embodiment; -
FIG. 10 is a diagram showing a configuration of an imaging optical system and an imaging device in a fourth embodiment; -
FIG. 11A is a diagram showing a configuration of a first polarization-dependent Fresnel zone plate in the fourth embodiment; -
FIG. 11B is a diagram showing a configuration of a second polarization-dependent Fresnel zone plate in the fourth embodiment; -
FIG. 12 is a diagram showing a configuration of an imaging optical system and an imaging device in a fifth embodiment; -
FIG. 13 is a diagram showing a distance measurement processing flow in the first embodiment; -
FIG. 14 is a diagram showing a processing flow for a PSF size adjustment in the first embodiment; -
FIG. 15 is a diagram showing an example of a configuration where the distance measurement device is used in an image display device; -
FIG. 16 is a diagram showing an example of a configuration where the distance measurement device is used in the image display device; -
FIG. 17 is a diagram showing a processing flow for a gesture input in the image display device using the distance measurement device; -
FIG. 18A is a diagram showing an image to be captured at the time of hand recognition processing by a user in the image display device using the distance measurement device; -
FIG. 18B is a diagram showing an image to be processed at the time of feature point extraction processing of a user's hand in the image display device using the distance measurement device; -
FIG. 18C is a diagram showing an image to be processed at the time of distance measurement region determination processing in the image display device using the distance measurement device; -
FIG. 19 is a diagram showing a configuration where the distance measurement device is used in a surroundings-monitoring system of a vehicle; -
FIG. 20 is a diagram showing a processing flow to determine a distance measurement range on the basis of a traveling state of the vehicle; and -
FIG. 21 is a diagram showing a configuration in which two cameras are provided as distance measurement devices in a stereo method distance measurement device having a variable base length. - Embodiments of the present invention will hereinafter be described with reference to the accompanying drawings. Like reference numerals designate identical components throughout all the drawings, and their dual description will be omitted.
- A wearable terminal such as a head-mounted display or the like directed to the application to an augmented reality (Augmented Reality, AR) technology and a virtual reality (Virtual Reality, VR) technology has recently been made into a product. As one input method by a user in the AR⋅VR technology, there is known a gesture input to recognize the motion of a hand. Further, in the AR technology, there is a need for a spatial recognition function to grasp a positional relation of an object in space in order to superimpose visual information into the space. A distance measurement technology of measuring depth information of the object is required for implementation of these. A distance measurement technology typified by a time of flight (Time-of-Flight, TOF) method has been utilized therefor.
- An active distance measurement device typified by the TOF method requires a light source irradiating an object with light. On the other hand, a distance measurement device according to the present embodiment to be described below is a so-called passive distance measurement device which does not require the light source irradiating the object with the light.
- A first embodiment in the present invention will be described in accordance with the accompanying drawings.
FIG. 1 shows one example of a configuration of adistance measurement device 101 in the present invention. Thedistance measurement device 101 captures an image of a target to be imaged by animaging device 105 through an imagingoptical system 102. The imagingoptical system 102 has a point spreadfunction adding section 103 which adds different point spread functions to two orthogonal polarization beams respectively, and a polarizationbeam splitting section 104 which splits incident light having a first polarization direction and incident light having a second polarization direction substantially perpendicular to the first polarization direction. The term substantially perpendicular referred to here is not intended to limit when a polarization direction axis of the incident light having the first polarization direction, and a polarization direction axis of the incident light having the second polarization direction are exactly 90°, and includes such a case that the angle formed by the polarization direction axis of the incident light having the first polarization direction and the polarization direction axis of the incident light having the second polarization direction is included in an allowable accuracy range in which they are regarded as being orthogonal to each other centering around 90 degrees. - Further, the term “point spread function” is a function indicating a response to a point light source of the optical system. In images each obtained when a light beam generated from one point of a target to be imaged is captured by the imaging device, there occurs a difference in the degree of blurring between the captured images where the point spread functions of the imaging optical system differ. In the present embodiment, the distance is measured by using the difference in the image blur degree.
- Image data captured by the
imaging device 105 is transferred to animage processing section 106. Adistance measurement section 107 generates distance information on the basis of the image, and a displayimage forming section 108 generates an image for display on which the distance information is reflected. The distance information and display image generated in theimage processing section 106 are output to acontrol section 109 and may be used for usage applications other than the generation of the display image, e.g., an operation input to be described later. - The input side of light in the imaging
optical system 102 further includes anaperture stop 202 configured to include, e.g., a plurality of movable vanes, and a aperturestop driving section 206 for driving theaperture stop 202 to change an aperture size. Thecontrol section 109 sends a drive control signal to the aperture stop drivingsection 206 to drive theaperture stop 202 and thereby change the aperture size of the imagingoptical system 102 to perform an optical adjustment. Further, thecontrol section 109 performs control of execution timing for each processing in theimaging device 105 and theimage processing section 106, and communications with an external device such as an image display device. A CMOS image sensor, a CCD image sensor and the like can be used as theimaging device 105. -
FIG. 2 shows one example of a configuration of the imagingoptical system 102 and theimaging device 105 in the present embodiment.Incident light 201 incident on thedistance measurement device 101 according to the present embodiment enters the point spreadfunction adding section 103 in the imagingoptical system 102 and transmits anaperture part 202 a formed by theaperture stop 202 in the point spreadfunction adding section 103, and alens 203. Of the light transmitted through the point spreadfunction adding section 103, afirst polarization beam 204 and asecond polarization beam 205 perpendicular to each other respectively transmit the point spreadfunction adding section 103 in a state in which they are added with different point spread functions. Afterwards, thefirst polarization beam 204 and thesecond polarization beam 205 are received by theimaging device 105 through the polarizationbeam splitting section 104, so that an image of the target to be imaged is recorded. Although the polarizationbeam splitting section 104 and theimaging device 105 have been described away from each other for convenience of explanation inFIG. 2 , they preferably approximate each other, and both may be configured by laminating them. -
FIG. 3 shows one example of a configuration of thelens 203 in the present embodiment. Incidentally, a traveling direction of theincident light 201 is assumed to be a direction perpendicular to the paper, a polarization direction of thefirst polarization beam 204 is assumed to be parallel to anaxis 301, and a polarization direction of thesecond polarization beam 205 is assumed to be parallel to anaxis 302. - The
lens 203 is made of a birefringent material. A refractive index to thefirst polarization beam 204 parallel to theaxis 301 is n1, and a refractive index to thesecond polarization beam 205 parallel to theaxis 302 is n2. Therefore, a focal length of thelens 203 becomes lengths respectively different from each other with respect to thefirst polarization beam 204 and thesecond polarization beam 205. For example, when thelens 203 is a spherical plane-convex lens having a curvature radius R, a focal length f1 to thefirst polarization beam 204 and a focal length f2 to thesecond polarization beam 205 can be expressed byequations 1 and 2 respectively. Therefore, for example, when n1 is larger than n2, f1 becomes a value smaller than f2. -
(Equation 1) f 1 =R/(n 1−1) (1) -
(Equation 2) f 2 =R/(n 2−1) (2) -
FIG. 4 shows one example of a configuration of the polarizationbeam splitting section 104 and theimaging device 105 in the present embodiment. The polarizationbeam splitting section 104 is a polarizer array in whichfirst polarizers first polarization beam 204 and shield thesecond polarization beam 205, andsecond polarizers first polarization beam 204 and transmit thesecond polarization beam 205 are arranged. The polarizationbeam splitting section 104 and theimaging device 105 are disposed in such a manner that the beams transmitted through the first polarizers orsecond polarizers 401 through 409 are respectively imaged bypixels 410 through 418 of theimaging device 105. Therefore, thepixels first polarization beam 204, and thepixels second polarization beam 205. -
FIG. 4 describes a part of the pixels which constitute theimaging device 105. The number of pixels is not limited to that shown inFIG. 4 . Further, thepixels 410 through 418 may be single monochrome image sensors respectively. Alternatively, the respective pixels of thepixels 410 through 418 may be Bayer filter-type image sensors respectively having one or more pixels corresponding to RGB. - With the configuration as described above, it is possible to allow the
first polarization beam 204 and thesecond polarization beam 205 to be incident on theimaging device 105 at different focal lengths respectively and individually capture a first polarization image obtained by capturing thefirst polarization beam 204 by theimaging device 105 and a second polarization image obtained by capturing thesecond polarization beam 205 respectively by theimaging device 105. It is therefore possible to simultaneously capture the first polarization image and the second polarization image being two images different in blur amount and execute a distance measurement by a Depth from defocus (DFD) method being one of lens focal methods. - The lens focal method is accompanied by a problem that since it needs at least two or more images different in blur amount, imaging is performed plural times while changing the distance between the lens and the
imaging device 105 in the conventional DFD method, and the measurement time is long as compared with other distance measurement methods. Further, the configuration described inPatent Literature 1 has a problem in that images different in blur amount can be simultaneously captured by using the two ormore imaging devices 105, but power consumption is increased because the number ofimaging devices 105 increases. On the other hand, according to the configuration of the present embodiment, there can be provided a small-sized/low power consumption distance measurement device which executes the distance measurement by the DFD method at high speed because the two first polarization image and second polarization image different in blur amount can simultaneously be captured by using oneimaging device 105. - Incidentally, the number of polarizers that constitute the polarization
beam splitting section 104 and how to arrange them are not limited to those shown inFIG. 4 . For example, thefirst polarizers second polarizers beam splitting section 104. Further, the number of pixels that constitute theimaging device 105 is not limited to that shown inFIG. 4 . - Further, in the configuration of the polarization
beam splitting section 104 and theimaging device 105 in the present embodiment, since the half of the pixels of theimaging device 105 is used for the imaging of the first polarization image, and the other half of the pixels is used for the imaging of the second polarization image, the number of pixels of the image obtainable in theimaging device 105 is reduced by half, but for the uncapturable pixels, the first polarization image and the second polarization image can be generated by performing interpolation processing on the basis of a light intensity signal received by the adjacent pixels. Considering the second polarization image by way of example, thepixel 414 ofFIG. 4 is not capable of measuring a light intensity signal of thesecond polarization beam 205, and hence the light intensity signal for the corresponding pixel lacks. However, the average value of light intensity signals of thesecond polarization beam 205, which are measured by the surroundingpixels second polarization beam 205 of thepixel 414 to interpolate the light intensity signal of thepixel 414, whereby the second polarization image can be generated. The first polarization image can be generated by performing similar interpolation processing even on the image of thefirst polarization beam 204. - Further, the
lens 203 in the present embodiment may be a lens using a birefringent crystal such as a crystal, calcite, magnesium fluoride, barium metaborate, or the like or may be a plastic lens indicative of birefringence. - Here, the distance measurement method according to the present embodiment will be described using
FIGS. 5 and 6 .FIG. 5 is a diagram describing the manner in which anobject 501 to be imaged is captured in the distance measurement device of the present embodiment. The distance between theobject 501 and thelens 203 is assumed to be u, and the distance between thelens 203 and theimaging device 105 is assumed to be s. Further, the diameter of theaperture part 202 a is assumed to be D0.FIG. 5A shows the manner in which the incident light of thefirst polarization beam 204 is captured by theimaging device 105. Light of thefirst polarization beam 204 resulting from one point of theobject 501 is transmitted through thelens 203 and then imaged on animaging point 502 focused with the lens having the focal length f1. -
FIG. 5B shows the manner in which the incident light of thesecond polarization beam 205 is imaged by theimaging device 105. Light of thesecond polarization beam 205 resulting from one point of theobject 501 is transmitted through thelens 203 and then imaged on animaging point 503 focused with the lens having the focal length f2. In the case ofFIG. 5 , since the distance between theimaging point 502 and theimaging device 105 is shorter than the distance between theimaging point 503 and theimaging device 105, the image of thesecond polarization beam 205 becomes an image stronger in blur than that of thefirst polarization beam 204. - A description will be made as to the difference in blur amount between the captured images of the
first polarization beam 204 and thesecond polarization beam 205 with reference toFIG. 6 . An image in the case where rays of light resulting from one point of theobject 501 are captured by theimaging device 105 is called a point spread function (Point Spread Function, PSF).FIG. 6 illustrates a point spread function captured by the imaging optical system of the present embodiment. Animage 601 is an example of an image when imaging apoint spread function 602 of thefirst polarization beam 204. Animage 603 is an example of an image when imaging apoint spread function 604 of thesecond polarization beam 205. The diameter D1 of thepoint spread function 602 can be represented by an equation 3 when using D0, u, s and f1. The diameter D2 of thepoint spread function 604 can be represented by an equation 4 when using D0, u, s and f2. -
(Equation 3) D 1 =sD 0(1/f 1−1/u−1/s) (3) -
(Equation 4) D 2 =sD 0(1/f 2−1/u−1/s) (4) - When capturing the
object 501 with thefirst polarization beam 204 or thesecond polarization beam 205, an image corresponding to the convolution integral of the point spread functions respectively shown in the equations 3 and 4 with the image of noblur object 501 is captured. Therefore, by extracting the size of thepoint spread function first polarization beam 204 or thesecond polarization beam 205, the distance between theobject 501 and thelens 203 can be measured. - As a method of extracting the point spread function from the two convoluted images, for example, a two-dimensional Fourier transform may be utilized. The point spread function may be calculated by using a Spatial Domain Convolution/Deconvolution Transform described in the following Literature.
-
- Murali Subbarao, Gopal Surya, “Depth from defocus: A spatial domain approach,” International Journal of Computer Vision, (US) Springer, 1994, December, Volume 13, No. 3, p. 271-294.
- Incidentally, the
imaging device 105 may be disposed in such a manner that either the first polarization image or the second polarization image is formed on theimaging device 105. With such an arrangement, one of the two images captured for distance measurement is displayed as it is and thereby usable as a display image for a user in AR systems or the like. Further, even when neither of the first polarization image and the second polarization image is formed on theimaging device 105, the PSF is calculated from the two polarization images and thereafter blurring of the captured image is removed based on information of the PSF, whereby a display image for the user may be generated. - Next, an example of a distance measurement processing flow using the
distance measurement device 101 of the present embodiment will be described usingFIG. 13 . A processing S1301 performs initialization of a part related to the distance measurement such as an adjustment of the PSF size. - In a processing S1302, the
first polarization beam 204 is captured to generate a first raw (raw) polarization image, and thesecond polarization beam 205 is captured to generate a second raw polarization image. - In a processing S1303, interpolation processing is performed on the first raw polarization image and the second raw polarization image captured in the processing S1302 respectively to generate a first polarization image and a second polarization image.
- In a processing S1304, a PSF is extracted from the first polarization image and the second polarization image generated in the processing S1303.
- In a processing S1305, the distance to a target object is calculated based on the PSF extracted in the processing S1304.
- In a processing S1306, information about the distance measured in the processing S1305 is output in a form depending on the application. For example, in the case of a usage method of providing the user with the distance information to an object as a display image, an image obtained by superimposing the distance information measured in the processing S1305 on the image captured in the processing S1302 is generated and displayed on a displaying display or the like. Further, when used for the recognition of a gesture input in the AR system, distance measurement information is recorded, and the difference between the distance measurement information and distance measurement information measured in the processing S1305 at the next distance measurement is calculated to determine the input contents, which is then output to the input of the AR system.
- In a processing S1307, a determination is made as to the continuation and end of the measurement. In the case of the continuation thereof (S1307/Yes), the distance measurement processing proceeds to the processing S1302. In the case of the end thereof (S1307/No), the distance measurement processing is terminated.
- Next, a method for adjusting the PSF size will be described. Generally, when the focal length of the
lens 203, the diameter of theaperture part 202 a, the distance between theimaging device 105 and thelens 203, etc. are the same conditions, an object lying in the vicinity than a distant object becomes large in PSF. When the PSF becomes a predetermined size or more as compared with theimaging device 105, accurate distance measurement can not be performed. Therefore, an operation to control the diameter of theaperture part 202 a and the distance between theimaging device 105 and thelens 203 and reduce the size of the PSF is required. - Whether the size of the PSF is appropriate can be determined from the change in the PSF extracted before and after the change of the diameter of the
aperture part 202 a, for example. Originally, when the diameter of theaperture part 202 a is reduced, the PSF of an image captured by theimaging device 105 becomes small. When, however, the diameter of theaperture part 202 a is made small where the size of the PSF becomes a predetermined size or more than theimaging device 105, the PSF size extracted from the image remains unchanged or increases. This is because when the PSF is a predetermined size or more, at least one of the two images used for DFD does not cause a distinct change in terms of how to blur even if the diameter of theaperture part 202 a is made small. In this situation, when another image is reduced in blur amount under the influence of the diameter of theaperture part 202 a being reduced, the difference in blur amount between the two images increases. In the present measuring method, an increase in the difference in the blur amount means an increase in the PSF. Further, when another image does not cause a distinct change either where the diameter of theaperture part 202 a is reduced, the difference in the amount of blurring is not generated between the two images even if the diameter of theaperture part 202 a is changed, thereby leading to the calculation result that the PSF is constant. It is determined using this characteristic whether the PSF size is appropriate. When it is not appropriate, the PSF size is reduced. - An example of a processing flow for the PSF size adjustment will be described using
FIG. 14 . - In a processing S1401, the diameter of the
aperture part 202 a is set. - In a processing S1402, a first polarization image and a second polarization image are generated to extract a PSF. The term “the generation of the first polarization image” may be processing of capturing the first raw polarization image and performing interpolation processing thereto to generate the first polarization image, or may be processing in which the first polarization image is obtained in a state of having no defects in pixels of the
imaging device 105 upon imaging thefirst polarization beam 204 as in a second embodiment to be described later. The same applies even to the second polarization image. - In a processing S1403, the diameter of the
aperture part 202 a is changed. - In a processing S1404, the first polarization image and the second image are captured again to extract the PSF.
- In a processing S1405, the PSF extracted in the processing S1402 and the PSF extracted in the processing S1404 are compared with each other to determine whether the current PSF is appropriate. For example, when the diameter of the
aperture part 202 a at the capturing in the processing S1402 is larger than the diameter of theaperture part 202 a at the capturing in the processing S1404, the PSF extracted in the processing S1402 is determined to be appropriate if being larger than the PSF extracted in the processing S1404 (S1405/Yes), and the processing is terminated. - On the other hand, when it is determined that the PSF extracted in the processing S1402 is smaller than the PSF extracted in the processing S1404 or remains unchanged, it is determined that the current PSF exceeds the proper size (S1405/No). The processing proceeds to the processing S1403, and a PSF is extracted in the processing S1404 after changing the diameter of the
aperture part 202 a again. - In the processing S1405, the PSF extracted in the first processing S1404 and the PSF extracted in the second processing S1404 are compared with each other to re-determine whether the PSF size is appropriate.
- Incidentally, the processing of
FIG. 14 may be done only once within the processing S1301 ofFIG. 13 . The processing may be done every time when shifting from the processing S1307 to the processing S1302 or may be done every several times. Further, when the processing ofFIG. 14 is done upon shifting from the processing S1307 ofFIG. 13 to the processing S1302, the distance information calculation of the processing S1405 is performed using the PSF sizes extracted in the processing S1402 and the processing S1404, so that the processing S1302 to the processing S1304 may be omitted. - Incidentally, the distance measurement performance may be improved by using a plurality of the
distance measurement devices 101 of the present embodiment in combination.FIG. 21 shows an example of a configuration in which two cameras are provided as thedistance measurement devices 101 of the present embodiment in a stereo method distance measurement device having a variable base length. The twodistance measurement devices 101 are fixed to acamera holding section 2101. Thecamera holding section 2101 has a telescopic function and changes the base length according to the required distance measurement accuracy. With the above configuration, for example, distance information measured by thedistance measurement device 101 and distance information measured by the stereo method are compared with each other to thereby make it possible to calibrate the change in the base length when expansion and contraction of thecamera holding section 2101 or due to a change in temperature or stress-time degradation. - An application destination of the
distance measurement device 101 of the present embodiment includes, for example, an AR corresponding type image display device such as a head-mounted display, or a mobile information terminal such as a smart phone. An example of using thedistance measurement device 101 in the image display device such as the head-mounted display will be described usingFIGS. 15 to 18 . -
FIG. 15 shows an example of an external appearance in the case of using thedistance measurement device 101 of the present embodiment in theimage display device 1501.FIG. 16 shows an example of a configuration in the case of using thedistance measurement device 101 of the present embodiment in theimage display device 1501. Theimage display device 1501 is configured to include animage display section 1502, anaudio input section 1503, aninput switch 1504, a personal computer held by a server on a network or a user, acommunication section 1604 which communicates with a mobile information terminal or the like, aninformation storage section 1605 storing information therein, and acontrol section 1602. - The
distance measurement device 101 is arranged so as to be able to measure the forward when the user wears theimage display device 1501 as an example. Thecontrol section 1602 controls the operations of theimage display section 1502, thedistance measurement device 101, thecommunication section 1604, and theinformation storage section 1605 on the basis of inputs from thedistance measurement device 101, theaudio input section 1503, theinput switch 1504, thecommunication section 1604, and theinformation storage section 1605. -
FIG. 17 shows a processing flow for a gesture input in theimage display device 1501 using thedistance measurement device 101. Theimage display device 1501 starts gesture recognition processing in thedistance measurement device 101 in accordance with an instruction from thecontrol section 1602. Start timing of the gesture recognition processing may be the immediately following the startup of theimage display device 1501, or may always be done in accordance with the inputs from theaudio input section 1503, theinput switch 1504, the mobile information terminal connected to thecommunication section 1604, etc. - In a processing S1701, a part related to the distance measurement is initialized.
- In a processing S1702, the hand of the user is recognized from an image captured by the
imaging device 105 to extract feature points. - In a processing S1703, a distance measurement region is determined based on the feature points extracted in the processing S1702.
- In a processing S1704, the distance measurement of the distance measurement region determined in the processing S1703 is performed to acquire the three-dimensional coordinates of the respective feature points.
- Then, feature points of the user's hand are extracted again in a processing S1705, and a distance measurement region is determined in a processing S1706.
- In a processing S1707, the distance measurement of the distance measurement region determined in the processing S1706 is performed to acquire the three-dimensional coordinates of the respective feature points again.
- In a processing S1708, a movement amount of the feature point is calculated from the difference between the respective feature points acquired in the processing S1704 and the processing S1707, and the display image on which the movement amount of the feature point is reflected is updated in a processing S1709.
- In a processing S1710, a determination is made as to the end and continuation of the input. In the case of the end thereof (S1710/Yes), the input is terminated. In the case of the continuation thereof (S1710/No), the processing flow is returned to the processing S1702. Incidentally, in the scene or the like where a change in the position of the user's hand is small, the feature point extracting processing and the distance measurement region determining processing are not performed each time the feature point coordinate acquiring processing is executed, and may be executed once in several times. Further, a waiting time may be deliberately provided between the processing S1704 and the processing S1707.
-
FIG. 18 shows an example of an image to be captured and processed by thedistance measurement device 101 at the time of execution of the processing S1702 and the processing S1703. When it is recognized that a user'shand 1802 is reflected onto an imaging screen 1801 (FIG. 18A ), afeature point 1803 is extracted in the processing S1702 as shown inFIG. 18B . The recognition of the user'shand 1802 is performed based on the color or shape, for example. Further, a user registration function is provided to theimage display device 1501 to thereby register the color and size of the user'shand 1802, and a region with high frequency of use to the gesture input on the imaging screen. The recognition may be performed based on these registration information to enhance the recognition accuracy of the user'shand 1802. Furthermore, as thefeature point 1803, for example, the joints of the user'shand 1802, etc. are used. - Next, in the processing S1703, a
distance measurement region 1804 is determined in such a manner that depth information of thefeature point 1803 can be acquired (FIG. 18C ). When the above processing is performed, the load of calculation processing can be reduced because it is not necessary to perform the distance measurement processing for all pixels of theimaging device 105. - Further, for example, as shown in
FIG. 18B andFIG. 18C , an image obtained by superimposing the feature points and the distance measurement region may be displayed to the user at theimage display section 1502, or the distance information to the respective feature points may be superimposed and displayed. As a method of displaying the distance, for example, numeric values may be superimposed as they are, or the distance may be displayed as isolines or isolines stepwise colored according to the distance. - In the above example, the head-mounted display is used as the image display device, and the input operation to the head-mounted display is performed with the gesture of the user's hand to calculate the amount of movement of the feature point of the hand by the distance measurement device according to the present embodiment. Then, an operation signal indicating the amount of its movement is output to the control section to update the display image reflecting the movement amount of the feature point. However, there may be configured such that the operation to switch the communication destination where the head-mounted display has a function different from the image display device, e.g., the head-mounted display has a function to communicate with the external device, or the input operation to perform a volume adjustment where it has an audio output function is performed by the gesture, and the input operation is performed by the control section.
- Other application destinations of the
distance measurement device 101 of the present embodiment include, for example, an automobile, an unmanned aircraft such as a drone, a robot, a surgical operation assisting system, a precise shape measuring device, etc. Thedistance measurement device 101 may be mounted in a mobile mechanical device such as an automobile, a drone, a disaster survey robot and an industrial robot, and utilized in collision avoidance by surrounding monitoring. Position information and distance measurement information may be uploaded to the server on the network and presented to a remote operator. The position information and the distance measurement information uploaded to the server on the network may be shared with another mobile mechanical device and utilized in cooperative position control of the mobile mechanical device in the automatic operation and autonomous navigation. -
FIG. 19 is a diagram in which in an example of a configuration where thedistance measurement device 101 is used in a surroundings-monitoring system of anautomobile 1901, a part interlocked with thedistance measurement device 101 is extracted. Thedistance measurement device 101 obtains the current traveling state from aspeed measurement section 1904, an electroniccharge payment terminal 1905, aGPS receiver 1906, etc. through anautomobile control section 1902 to determine a distance measurement range. -
FIG. 20 shows an example of a processing flow for determining the distance measurement range, based on the traveling state of the automobile. An automobile traveling mode is determined in a processing S2001 after the engine of the automobile has started. The automobile traveling mode may be determined by thespeed measurement section 1904, for example. A determination may be made by theGPS receiver 1906 as to whether a traveling speed and a road on which the automobile is traveling at present are subject to a highway or a general road. Further, on the basis of communication records of the electroniccharge payment terminal 1905, a determination may be made as to whether the road on which the automobile is traveling at present is a toll highway or a general road. - After determining the current traveling mode in the processing S2001, a distance measurement mode is decided based on the determination result in a processing S2002. In the distance measurement mode, for example, the diameter of the
aperture part 202 a is set to increase the PSF in order to perform the remote distance measurement with high accuracy during high-speed traveling, and the diameter of theaperture part 202 a is set to decrease the PSF in order to perform the distance measurement at a close range with high accuracy during low-speed traveling. Also, since it is necessary to cope even with sudden interruption to the front, of pedestrians and light vehicles during traveling on a general road, the PSF size optimizing processing by the diameter control of theaperture part 202 a described inFIG. 14 is also performed. - Further, in the processing S2002, the pixel region used for the distance measurement, of the pixels in the
imaging device 105 is changed based on the determination result of the processing S2001, for example, the distance measurement may be performed at all pixels during the general road traveling, and the distance measurement may be performed only by some pixels in the imaging device during traveling on the highway. Even in addition to the above, the pixel region used for the distance measurement may be changed in accordance with the shape of the road like intersections, curves, straight lines, etc. - After deciding the distance measurement mode in the processing S2002, the distance measurement processing described in
FIG. 13 andFIG. 14 is performed in a processing S2003, and the distance information to an object around the vehicle is used for collision avoidance control or automatic driving control. - In a processing S2004, the automobile traveling mode is determined again. In a processing S2005, a determination is made as to whether the automobile traveling mode is changed. When the automobile traveling mode remains unchanged (S2005/No), the processing flow returns to the processing S2003 and the distance measurement is continued. When the traveling mode is changed (S2005/Yes), it is determined in a processing S2006 whether the engine is stopped. When the engine is stopped (S2006/Yes), the processing is terminated. When the engine is not stopped (S2006/No), the processing flow returns to the processing S2002, where the distance measurement mode suitable for the traveling state is re-selected.
- Also, the distance information around the vehicles obtained by the
distance measurement device 101 of the present embodiment may be displayed on an image display device such as a head-up display to display an image on the automobile and its windshield, an image display screen of a car navigation system, or the like. For example, when displayed on the head-up display, a marker and distance information may be displayed with being superimposed on the forward vehicle and pedestrian. A marker superimposed on an object high in the degree of risk of collision may be displayed in color different from the colors of other markers. Further, the moving speed of each object may be estimated from a change in the distance between the own vehicle and the object and displayed on the image display section. - Further, the distance information measured by the respective vehicles are transmitted to the center of the car navigation system or the like and integrated to generate highly accurate positional information of the vehicle/pedestrian on the road. Then, the generated information may be transmitted to each vehicle and displayed on the image display section of the car navigation system or the head-up display.
- The present embodiment is different from the first embodiment in that a polarization beam splitting section is configured of a polarizer and a polarization control element. With this configuration, an image of a
first polarization beam 204 and an image of asecond polarization beam 205 can respectively be captured by using all pixels of animaging device 105. The interpolation processing becomes unnecessary as compared with the configuration of the first embodiment, and high resolution can be attained. - A method of capturing a first polarization image and a second image in the present embodiment will be described using
FIG. 7 .FIG. 7 shows an example of a configuration of an imagingoptical system 102 and theimaging device 105 in the present embodiment.Incident light 201 incident on the imagingoptical system 102 enters into a polarizationbeam splitting section 104 and is transmitted through apolarization control element 702 and apolarizer 701 and then transmitted through the polarizationbeam splitting section 104. After passing through the polarizationbeam splitting section 104, the incident light is transmitted through a point spreadfunction adding section 103 and imaged by theimaging device 105. - The
polarizer 701 is an element to take out a specific polarization component of the incident light 201 polarized at random. Thepolarization control element 702 is constituted of, for example, a liquid crystal element and capable of rotating the polarization beam of light to be transmitted through thepolarization control element 702 in any direction by voltage control. Therefore, the light transmitted through thepolarizer 701 is transmitted through thepolarization control element 702 to perform voltage control thereof, whereby the polarization beam entering into the point spreadfunction adding section 103 can be switched to afirst polarization beam 204 and asecond polarization beam 205 different in polarization direction. -
FIG. 7A shows a configuration of the imagingoptical system 102 and theimaging device 105 where thepolarization control element 702 is controlled such that the polarization beam of light transmitted through thepolarization control element 702 becomes thefirst polarization beam 204. Since alens 203 functions as a lens having a focal length f1 with respect to thefirst polarization beam 204, an image to be captured by theimaging device 105 becomes an image captured by the lens having the focal length f1. -
FIG. 7B shows a configuration of the imagingoptical system 102 and theimaging device 105 where thepolarization control element 702 is controlled in such a manner that the polarization beam of light transmitted through thepolarization control element 702 becomes thesecond polarization beam 205. Since thelens 203 functions as a lens having a focal length f2(>f1) with respect to thesecond polarization beam 205, an image to be captured by theimaging device 105 becomes an image captured by the lens having the focal length f2. - With the above configuration, the two images (corresponding to the first polarization image and the second polarization image) different in blur amount can be captured, and the distance measurement of the target object by the DFD method can be performed. Further, since the first polarization image and the second polarization image are captured separately twice, the number of pixels of the
imaging device 105 usable when capturing each image is increased twice in comparison with the first embodiment, and hence high resolution can be attained. - Incidentally, the configuration of the imaging optical system and the
imaging device 105 in the present embodiment is not limited to the above configuration. For example, the configuration of the imagingoptical system 102 and theimaging device 105, which is shown inFIG. 8 , may be adopted. In the configuration ofFIG. 8 , the polarizationbeam splitting section 104 is located in the rear of the point spreadfunction adding section 103. The light incident on the polarizationbeam splitting section 104 first enters thepolarization control element 702 and then enters into thepolarizer 701, followed by being captured by theimaging device 105. By controlling thepolarization control element 702 so as to allow thepolarizer 701 to transmit only the polarization beam condensed at the focal length f1, theimaging device 105 is capable of capturing an image captured by the lens having the focal length f1. By controlling thepolarization control element 702 so as to allow thepolarizer 701 to transmit only the polarization beam condensed at the focal length f2, theimaging device 105 is capable of capturing an image captured by the lens having the focal length f2. - Further, the polarization
beam splitting section 104 is not limited to the combination of the polarizer and the polarization control element. The polarizationbeam splitting section 104 may be, for example, a rotary polarizer capable of controlling the direction of the polarization beam to be transmitted, or the like. - The present embodiment is different from the first and second embodiments in that a
lens 203 is configured of a material with chirality. With this configuration, it is possible to relax lens mounting accuracy at the time of manufacture of the distance measurement device and to improve resistance to the rotation of the lens due to loosening of a lens holding section in use. Further, it is possible to cope even with a zoom function with the rotation of the lens. -
FIG. 9 is a diagram showing an example of a configuration of thelens 203 in the present embodiment. Incidentally, the traveling direction of theincident light 201 is assumed to be a direction perpendicular to the paper surface. The material constituting thelens 203 is a material with chirality and indicates a different refractive index between right circular polarization and left circular polarization. For example, assuming that the refractive index with respect to the rightcircular polarization 901 is n1, the refractive index with respect to the leftcircular polarization 902 is n2, and the radius of curvature of thelens 203 is R, the focal length with respect to the right circular polarization becomes f1 of theequation 1, and the focal length with respect to the left circular polarization becomes f2 of the equation 2. - Therefore, an image (first polarization image) by the right
circular polarization 901 and an image (second polarization image) by the leftcircular polarization 902 are split by the polarizationbeam splitting section 104 and then captured by theimaging device 105. It is thus possible to capture the two images different in blur amount in a manner similar to the first embodiment and 2. The polarizationbeam splitting section 104 may be, for example, a polarizer array in which a polarizer for transmitting only the right circular polarization and a polarizer for transmitting only the left circular polarization are arranged. There may be adopted a configuration in which a ¼-wavelength plate is disposed in front of the polarizer array described in the first embodiment, or there may be one arranged by combining the liquid crystal element and the polarizer described in the second embodiment. - With the above configuration, it is possible to measure the distance to the target object from the two images different in blur amount in a manner similar to the first embodiment and 2. Further, since the
lens 203 of the present embodiment utilizes the birefringence with respect to the right and left circular polarizations, the image having the focal length f1 and the image having the focal length f2 are not captured with being mixed even when thelens 203 is rotated with the traveling direction of the incident light 201 as an axis. Therefore, the distance measurement accuracy is not reduced, the lens mounting accuracy at the time of manufacture of the distance measurement device can be relaxed, and the resistance to the rotation of the lens due to loosening of the lens holding section in use can be improved. Further, it is possible to cope even with a zoom function with the rotation of the lens. - The present embodiment is different from the first and second embodiments in that instead of using the birefringent lens in the point spread
function adding section 103, polarization-dependentFresnel zone plates -
FIG. 10 shows an example of a configuration of the imagingoptical system 102 and theimaging device 105 in the present embodiment. The point spreadfunction adding section 103 is configured to include anaperture stop 202, the polarization-dependentFresnel zone plate 1001, and the polarization-dependentFresnel zone plate 1002. - The Fresnel zone plate is a pattern in which a transparent ring belt and an opaque ring belt are alternately arranged concentrically, and serves to focus light. The radius rn of the nth ring belt of the Fresnel zone plate to focus the light of a wavelength λ at a focal length f is expressed by an equation 5. Therefore, a Fresnel zone plate of any focal length can be created depending on the design of spacing of each ring belt.
-
(Equation 5) r n=(nλf){circumflex over ( )}½ (5) -
FIG. 11 shows an example of a configuration of the polarization-dependent Fresnel zone plate in the present embodiment. The polarization-dependentFresnel zone plate 1001 ofFIG. 11A is one in which an opaque ring belt 1101 which shields afirst polarization beam 204 and transmits asecond polarization beam 205, and a transparent ring belt 1102 which transmits both thefirst polarization beam 204 and thesecond polarization beam 205 are arranged concentrically. Since the polarization-dependentFresnel zone plate 1001 is transparent with respect to thesecond polarization beam 205, it condenses only thefirst polarization beam 204. - On the other hand, the polarization-dependent
Fresnel zone plate 1002 ofFIG. 11B is one in which an opaque ring belt 1103 which transmits thefirst polarization beam 204 and shields thesecond polarization beam 205, and a transparent ring belt 1102 which transmits both of thefirst polarization beam 204 and thesecond polarization beam 205 are arranged concentrically. Since the polarization-dependentFresnel zone plate 1002 is transparent with respect to thefirst polarization beam 204, it condenses only thesecond polarization beam 205. The opaque ring belts 1101 and 1103 which respectively constitute the polarization-dependentFresnel zone plates - When the radius of the ring belt corresponding to the nth of the ring belts in the polarization-dependent
Fresnel zone plate 1001 is set such that the focal length of the polarization-dependentFresnel zone plate 1001 becomes f1, the radius of the ring belt corresponding to the nth of the ring belts in the polarization-dependentFresnel zone plate 1002 is set such that the focal length f2 of the polarization-dependentFresnel zone plate 1002 is taken, and the polarization-dependentFresnel zone plate 1001 and the polarization-dependentFresnel zone plate 1002 are bonded to each other and placed in the point spreadfunction adding section 103, as shown inFIG. 10 , thefirst polarization beam 204 is imaged at the focal length f1, and thesecond polarization beam 205 is imaged at the focal length f2. Therefore, two images different in blur amount can be captured by imaging thefirst polarization beam 204 and thesecond polarization beam 205 by theimaging device 105 after they are split by the polarizationbeam splitting section 104, and hence the distance measurement to the target object using the DFD method can be performed. - With the above configuration, since it is possible to add different blur amounts to two orthogonal polarized lights by using a thin and light Fresnel zone plate instead of the birefringent lens, a weight reduction of the device is made possible. Further, the difference between the blur amounts addable to the two orthogonal polarized lights has been limited by the difference in refractive index between the birefringent materials to be used in the first through third embodiments. However, with the configuration which uses the Fresnel zone plate, the combination of the focal lengths of the two orthogonal polarized lights can arbitrarily be selected, and the difference between the blur amounts to be added can be increased, thereby making it possible to improve the accuracy of the distance measurement.
- Further, as for the polarization-dependent
Fresnel zone plates first polarization beam 204 and thesecond polarization beam 205 are arranged. With such a configuration, it is possible to suppress the loss of light intensity which occurs in the opaque ring belts of the polarization-dependent Fresnel zone plate. - Incidentally, in the present embodiment, as the polarization beam splitting method for the
first polarization beam 204 and thesecond polarization beam 205 in the polarizationbeam splitting section 104, the method described in the first embodiment may be used, or the method described in the second embodiment may be used. Further, in the case where the method described in the second embodiment is used, the polarizationbeam splitting section 104 may be disposed in front of the point spreadfunction adding section 103. Moreover, instead of using the two polarization-dependent Fresnel zone plates, one of them may be used as a normal imaging lens. - The present embodiment is different from the first through fourth embodiments in that using a polarization-dependent aperture; two orthogonal polarized lights are narrowed by apertures different in size to thereby add different blur amounts to the two orthogonal polarized lights. This configuration leads to reductions in thickness, weight and cost of a point spread
function adding section 103. Further, simply by changing the diameters of anaperture part 202 a and a polarization-dependent aperture part 1201 a, it is possible to arbitrarily polarize the difference in blur amount between images of afirst polarization beam 204 and asecond polarization beam 205, and it is possible to control the distance measurement accuracy according to purposes of use. -
FIG. 12 shows an example of a configuration of an imagingoptical system 102 and animaging device 105 in the present embodiment. The point spreadfunction adding section 103 is configured to include anaperture stop 202, a polarization-dependent aperture stop 1201, and alens 203. The polarization-dependent aperture stop 1201 transmits both thefirst polarization beam 204 and thesecond polarization beam 205 within a range of a diameter D0′ from the center, for example, but in terms of the incident light thereoutside, the circular polarization-dependent aperture part 1201 a which shields thefirst polarization beam 204 and transmits thesecond polarization beam 205 is formed. Incidentally, an aperture diameter D0′ of the polarization-dependent aperture part 1201 a is assumed to be smaller than an aperture diameter D0 of theaperture part 202 a. - Further, the
lens 203 does not require the use of the birefringent material, and functions as a lens having a focal distance f with respect to both of thefirst polarization beam 204 and thesecond polarization beam 205. - Of the incident light 201 which is incident on the imaging
optical system 102, light of thefirst polarization beam 204 is narrowed down to the aperture diameter D0 at theaperture part 202 a and then narrowed down to the aperture diameter D0′ by the polarization-dependent aperture part 1201 a. Thereafter, thefirst polarization beam 204 is condensed by thelens 203 and transmitted through the polarizationbeam splitting section 104, followed by being captured by theimaging device 105. A diameter D1 of a point spread function of thefirst polarization beam 204 can be represented by an equation 6 when using D0′, a distance u between thelens 203 and a target object, a distance s between thelens 203 and theimaging device 105, and f. - On the other hand, of the incident light 201 which is incident on the imaging
optical system 102, light of thesecond polarization beam 205 is narrowed down to the aperture diameter D0 at theaperture part 202 a and then transmitted through the polarization-dependent aperture part 1201 a, but the aperture diameter does not change. Thereafter, thesecond polarization beam 205 is condensed by thelens 203 and transmitted through the polarizationbeam splitting section 104, followed by being captured by theimaging device 105. A diameter D2 of a point spread function of thesecond polarization beam 205 can be represented by an equation 7 when using D0, the distance u between thelens 203 and the target object, the distance s between thelens 203 and theimaging device 105, and the focal length f of thelens 203. -
(Equation 6) D 1 =sD 0′(1/f− 1/u−1/s) (6) -
(Equation 7) D 2 =sD 0(1/f− 1/u−1/s) (7) - With the adoption of the above configuration, the two images different in blur amount can be captured by imaging the
first polarization beam 204 and thesecond polarization beam 205 by theimaging device 105 after splitting thefirst polarization beam 204 and thesecond polarization beam 205 by the polarizationbeam splitting section 104. It is therefore possible to perform the distance measurement to the target object using the DFD method. In the present embodiment, since the polarization-dependent aperture stop 1201 using the circular aperture by the polarizer is used instead of the birefringent lens used in the first embodiment to 3, the point spreadfunction adding section 103 can be made thin and light. Further, the polarization-dependent aperture stop 1201 of the present embodiment is a simpler structure than the polarization-dependent Fresnel zone plate of the fourth embodiment and is hence capable of being reduced in manufacturing cost. Furthermore, as for theaperture stop 202, the so-called mechanical aperture stop to open and close the movable blade by using the already-described aperturestop driving section 206 is used, and theaperture part 202 a is formed as a variable aperture, whereby the difference in blur amount between the images of thefirst polarization beam 204 and thesecond polarization beam 205 can arbitrarily be polarized simply by changing the size of the aperture diameter D0, and the distance measurement accuracy can be controlled according to the purposes of use. - The above-described respective embodiments are not intended to limit the present invention, and various modifications not departing from the spirit of the present invention are intended to be included in the present invention. For example, the wavelength of the imaging light is not limited to visible light. It may be an electromagnetic wave longer in wavelength than infrared, or may be an electromagnetic wave shorter in wavelength than ultraviolet light. Also, the combination of the point spread function adding section and the polarization beam splitting section is not limited to the above either. Further, the processing order described in the above-described flowcharts is an example, and embodiments in which processing capable of obtaining the same operative effect even if the processing order is exchanged is interchanged before and after the processing are also included in the present invention. Moreover, the application examples of the present invention described with reference to
FIGS. 15 through 21 can be applied even to not only the distance measurement device according to the first embodiment, but also the distance measurement devices according to the second to fifth embodiments. - 101 . . . distance measurement device, 102 . . . imaging optical system, 103 . . . point spread function adding section, 104 . . . polarization beam splitting section, 105 . . . imaging device, 106 . . . image processing section, 107 . . . distance measurement section, 108 . . . display image forming section, 201 . . . incident light, 202 . . . aperture, 203 . . . lens, 701 . . . polarizer, 702 . . . polarization control element, 1001 . . . polarization-dependent Fresnel zone plate, 1002 . . . polarization-dependent Fresnel zone plate, 1201 . . . polarization-dependent aperture, 1501 . . . image display device, 1502 . . . image display section, 1503 . . . audio input section, 1504 . . . input switch, 2101 . . . camera holding section.
Claims (17)
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2017/009306 WO2018163335A1 (en) | 2017-03-08 | 2017-03-08 | Distance measurement device, head-mounted display device, mobile information terminal, image display device, surroundings monitoring system, and distance measurement method |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210132382A1 true US20210132382A1 (en) | 2021-05-06 |
US11422372B2 US11422372B2 (en) | 2022-08-23 |
Family
ID=63448611
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/492,094 Active 2037-12-26 US11422372B2 (en) | 2017-03-08 | 2017-03-08 | Distance measurement device, head-mounted display device, mobile information terminal, image display device, surroundings monitoring system, and distance measurement method |
Country Status (4)
Country | Link |
---|---|
US (1) | US11422372B2 (en) |
JP (1) | JP6756898B2 (en) |
CN (1) | CN110392818B (en) |
WO (1) | WO2018163335A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11348269B1 (en) * | 2017-07-27 | 2022-05-31 | AI Incorporated | Method and apparatus for combining data to construct a floor plan |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111175769B (en) * | 2020-02-14 | 2022-05-27 | 深圳奥锐达科技有限公司 | Off-axis scanning distance measuring system |
CN112595674A (en) * | 2020-11-20 | 2021-04-02 | 江西欧迈斯微电子有限公司 | Receiver, image capturing module and electronic equipment |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS61277919A (en) * | 1985-05-31 | 1986-12-08 | Canon Inc | Variable focal length lens |
JP2007205918A (en) * | 2006-02-02 | 2007-08-16 | Olympus Corp | Measuring endoscope |
JP4990234B2 (en) | 2008-07-07 | 2012-08-01 | オリンパス株式会社 | Ranging device, ranging method, ranging program or imaging device |
JP5848177B2 (en) | 2012-03-27 | 2016-01-27 | 日本放送協会 | Multi-focus camera |
JP6222898B2 (en) | 2012-07-03 | 2017-11-01 | キヤノン株式会社 | Three-dimensional measuring device and robot device |
JP5454983B2 (en) * | 2012-12-13 | 2014-03-26 | コニカミノルタ株式会社 | Obstacle measurement method, obstacle measurement device, and obstacle measurement system |
JP6292790B2 (en) | 2013-08-08 | 2018-03-14 | キヤノン株式会社 | Distance detection device, imaging device, and distance detection method |
JP2015046777A (en) * | 2013-08-28 | 2015-03-12 | キヤノン株式会社 | Imaging apparatus and control method of imaging apparatus |
JP6391316B2 (en) * | 2014-06-25 | 2018-09-19 | キヤノン株式会社 | Imaging device |
JP2016148569A (en) * | 2015-02-12 | 2016-08-18 | キヤノン株式会社 | Image measuring method and image measuring device |
JP2018136123A (en) * | 2015-06-24 | 2018-08-30 | 株式会社村田製作所 | Distance sensor and user interface apparatus |
-
2017
- 2017-03-08 US US16/492,094 patent/US11422372B2/en active Active
- 2017-03-08 WO PCT/JP2017/009306 patent/WO2018163335A1/en active Application Filing
- 2017-03-08 CN CN201780087953.8A patent/CN110392818B/en active Active
- 2017-03-08 JP JP2019504209A patent/JP6756898B2/en active Active
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11348269B1 (en) * | 2017-07-27 | 2022-05-31 | AI Incorporated | Method and apparatus for combining data to construct a floor plan |
US11481918B1 (en) * | 2017-07-27 | 2022-10-25 | AI Incorporated | Method and apparatus for combining data to construct a floor plan |
US11657531B1 (en) * | 2017-07-27 | 2023-05-23 | AI Incorporated | Method and apparatus for combining data to construct a floor plan |
US11961252B1 (en) * | 2017-07-27 | 2024-04-16 | AI Incorporated | Method and apparatus for combining data to construct a floor plan |
Also Published As
Publication number | Publication date |
---|---|
JPWO2018163335A1 (en) | 2019-11-21 |
US11422372B2 (en) | 2022-08-23 |
WO2018163335A1 (en) | 2018-09-13 |
JP6756898B2 (en) | 2020-09-16 |
CN110392818B (en) | 2021-11-09 |
CN110392818A (en) | 2019-10-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10455218B2 (en) | Systems and methods for estimating depth using stereo array cameras | |
EP2491340B1 (en) | Ranging camera apparatus | |
JP6238151B1 (en) | Head-up display device and vehicle | |
CN107615749B (en) | Signal processing apparatus and imaging apparatus | |
US9414045B2 (en) | Stereo camera | |
CN109644264B (en) | Array detector for depth mapping | |
JP5273356B2 (en) | Compound eye image input device and distance measuring device using the same | |
JP2018077190A (en) | Imaging apparatus and automatic control system | |
CN107122770B (en) | Multi-camera system, intelligent driving system, automobile, method and storage medium | |
US11422372B2 (en) | Distance measurement device, head-mounted display device, mobile information terminal, image display device, surroundings monitoring system, and distance measurement method | |
JP7098790B2 (en) | Imaging control device and moving object | |
EP3667413B1 (en) | Stereo image processing device | |
US20210152812A1 (en) | Display control device, display system, and display control method | |
US20230017668A1 (en) | Mobile communication terminal | |
KR101545971B1 (en) | System for sensing complex image | |
JP6653456B1 (en) | Imaging device | |
JP6202364B2 (en) | Stereo camera and moving object | |
JP2014041171A (en) | Polarization device and imaging device | |
JP2022175095A (en) | Image sensor, imaging apparatus, and image processing method | |
JP7446455B2 (en) | Location information acquisition system, location information acquisition method, and location information acquisition device | |
US11513266B2 (en) | Systems and methods for an improved camera system using directional optics to estimate depth | |
EP4303080A1 (en) | Imaging device and three-dimensional display device | |
JP2020064526A (en) | Image processing device | |
JP2014178494A (en) | Parallax image photographing device | |
JP2009109753A (en) | Imaging device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: MAXELL, LTD., JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YAMAGUCHI, KEITA;ISHII, TOSHIKI;SIGNING DATES FROM 20190827 TO 20190903;REEL/FRAME:050641/0928 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
AS | Assignment |
Owner name: MAXELL HOLDINGS, LTD., JAPAN Free format text: MERGER;ASSIGNOR:MAXELL, LTD.;REEL/FRAME:058255/0579 Effective date: 20211001 |
|
AS | Assignment |
Owner name: MAXELL, LTD., JAPAN Free format text: CHANGE OF NAME;ASSIGNOR:MAXELL HOLDINGS, LTD.;REEL/FRAME:058666/0407 Effective date: 20211001 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |