US20190129034A1 - Range image generation apparatus and range image generation method - Google Patents
Range image generation apparatus and range image generation method Download PDFInfo
- Publication number
- US20190129034A1 US20190129034A1 US16/094,901 US201616094901A US2019129034A1 US 20190129034 A1 US20190129034 A1 US 20190129034A1 US 201616094901 A US201616094901 A US 201616094901A US 2019129034 A1 US2019129034 A1 US 2019129034A1
- Authority
- US
- United States
- Prior art keywords
- light
- range
- range image
- image generation
- light emission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
- G01S17/32—Systems determining position data of a target for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated
- G01S17/36—Systems determining position data of a target for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated with phase comparison between the received signal and the contemporaneously transmitted signal
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
- G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
- G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
- G01S17/18—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves wherein range gates are used
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/4802—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
-
- H04N5/2256—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/74—Circuitry for compensating brightness variation in the scene by influencing the scene brightness using illuminating means
Definitions
- the present invention relates to a range image generation apparatus and a range image generation method.
- PTL1 discloses the following contents.
- TOF time-of-flight
- a light emission frequency selecting unit ( 7 ) receives reflected light S 2 from the target object at a point in time in which light is not emitted from a light source and selects illumination light S 1 having a light emission frequency hard to be influenced by the ambient light based on a frequency analysis of the reflected light S 2 .
- An image creating unit ( 6 ) receives reflected light of the illumination light S 1 having this optimal light emission frequency and emitted from a prepared light source and creates the range to the target object, such that it is possible to suppress the influence of the ambient light at the time of receiving the light and it is thus possible to decrease an influence of noise on range accuracy when a resolution of the light receiving element unit ( 2 ) is increased.
- Patent Literature 1 WO 2010/021090 A
- this method as compared with other three-dimensional recognition techniques such as a stereo image processing method, there is an advantage that it is hard to be influenced by a change in ambient light, a change in a temperature, and the like, or it is possible to measure a range with high accuracy and in real time.
- Patent Literature 1 in the case of an object having a high light reflectivity, for example, in the case in which an object surface has a mirror surface shape, the saturation of the image sensor can not be suppressed, and range accuracy can not be improved.
- an object of the present invention is to provide a range image generation apparatus capable of generating a range image with higher accuracy.
- the present application includes a plurality of means that solve at least some of the abovementioned problems, and an example of these means is as follows.
- a range image generation apparatus includes: a light emitting unit that emits light toward an object at different light emission intensities; a light receiving unit that captures an image of the object by receiving light including reflected light from the object and calculates a light exposure amount for each pixel; and a range calculating unit that calculates a range to the object by replacing a light exposure amount at a first light emission intensity by a light exposure amount at a second light emission intensity in a predetermined case and generates a range image.
- FIG. 1 is a diagram showing an example of a schematic configuration of a range image generation apparatus according to a first embodiment of the present invention.
- FIG. 2 is a diagram showing an example of a detailed configuration of a signal processing unit according to first and second embodiments of the present invention.
- FIG. 3 is a diagram showing an example of a hardware configuration of the range image generation apparatus according to the first embodiment of the present invention.
- FIG. 4 is a diagram showing an example of a timing chart of a light emission timing control signal S 1 and a light exposure timing control signal S 2 according to the first and second embodiments of the present invention.
- FIG. 5 is a flowchart showing an example of a flow of a range image generation process executed by the range image generation apparatus according to the first embodiment of the present invention.
- FIG. 6 is a diagram showing an example of a timing chart of a light emission intensity control signal S 0 , a light emission timing control signal S 1 , and a light exposure timing control signal S 2 according to the first embodiment of the present invention.
- FIG. 7 is a diagram showing an example of an image capturing configuration including the range image generation apparatus, an object OB 0 , and an object OB 1 according to the first embodiment of the present invention.
- FIG. 8 is a diagram showing an example of an image in which light exposure amounts Q 00 of the objects OB 0 and OB 1 image-captured in a strong light emission period P 0 according to the first embodiment of the present invention are two-dimensionally arranged.
- FIG. 9 is a diagram showing an example of an image in which light exposure amounts Q 10 of the objects OB 0 and OB 1 image-captured in a weak light emission period P 1 according to the first embodiment of the present invention are two-dimensionally arranged.
- FIG. 10 is a diagram showing an example of a schematic configuration of a range image generation apparatus according to a second embodiment of the present invention.
- FIG. 11 is a diagram showing an example of a timing chart of a light emission timing control signal S 1 , a light exposure timing control signal S 2 , and a light reception amount control signal S 3 according to the second embodiment of the present invention.
- FIG. 1 is a diagram showing an example of a schematic configuration of a range image generation apparatus 100 according to the present embodiment.
- the range image generation apparatus 100 adopts a time-of-flight (TOF) method, and is an apparatus that generates a range image in which a range to an object OB is two-dimensionally arranged for each pixel.
- the range image generation apparatus 100 includes an input receiving unit 11 , a control unit 12 , a light emitting unit 13 , a light receiving unit 14 , and a signal processing unit 15 .
- the input receiving unit 11 is a functional unit that receives an instruction input from a user of the range image generation apparatus 100 . Specifically, the input receiving unit 11 receives setting of a predetermined image capturing mode (for example, a short range mode, a medium range mode, a long range mode, or the like) related to a measurement range (for example, 1 to 3 m (short measurement range), 3 to 7 m (middle measurement range), 7 to 10 m (long measurement range), or the like) from the range image generation apparatus 100 to the object OB from the user. In addition, the input receiving unit 11 outputs the received setting information to a predetermined functional unit (for example, the control unit 12 or the like).
- a predetermined image capturing mode for example, a short range mode, a medium range mode, a long range mode, or the like
- a measurement range for example, 1 to 3 m (short measurement range), 3 to 7 m (middle measurement range), 7 to 10 m (long measurement range), or the like
- the control unit 12 is a functional unit that controls various processes executed by the range image generation apparatus 100 . Specifically, the control unit 12 generates a light emission intensity control signal S 0 , a light emission timing control signal S 1 , and a light exposure timing control signal S 2 depending on the measurement range.
- the light emission intensity control signal S 0 is a signal that controls a degree of strength of light emission (light emission intensity) in the light emitting unit 13 .
- the light emission timing control signal S 1 is a signal that controls a timing of light emission by the light emitting unit 13 and a length of a light emission time.
- the light exposure timing control signal S 2 is a signal that controls a timing of light exposure by the light receiving unit 14 and a length of a light exposure time.
- the control unit 12 outputs the light emission intensity control signal S 0 and the light emission timing control signal S 1 to the light emitting unit 13 and outputs the light exposure timing control signal S 2 to the light receiving unit 14 , at a predetermined timing (for example, when a generation instruction of a range image by image-capturing of the object OB is received from the user).
- the control unit 12 generates information (hereinafter, referred to as signal related information) on the timing indicated by these control signals, and outputs the information to the signal processing unit 15 .
- the light emitting unit 13 is a functional unit that emits near infrared light toward the object OB. Specifically, the light emitting unit 13 emits the near infrared light toward the object OB at a timing and a light emission intensity indicated by the light emission timing control signal S 1 and the light emission intensity control signal S 0 .
- the light receiving unit 14 is a functional unit that receives light including reflected light of the light with which the object OB is irradiated and calculates a light exposure amount. Specifically, the light receiving unit 14 captures an image of the object OB by receiving (exposing) light including reflected light from the object OB at a timing indicated by the light exposure timing control signal S 2 . In addition, the light receiving unit 14 calculates a light exposure amount for each pixel of the captured image of the object. In addition, the light receiving unit 14 converts the light exposure amount into an electrical signal by photoelectric conversion, and outputs the converted electrical signal to the signal processing unit 15 .
- the signal processing unit 15 is a functional unit that calculates a range to the object OB for each pixel and generates a range image. Specifically, the signal processing unit 15 acquires the photoelectrically converted electrical signal as an input signal from the light receiving unit 14 , and calculates the range to the object OB for each pixel using the electrical signal. In addition, the signal processing unit 15 generates a range image in which each pixel having range information to the object OB is two-dimensionally arranged.
- FIG. 2 is a diagram showing an example of a detailed configuration of the signal processing unit 15 .
- the signal processing unit 15 includes a separating unit 151 , an R/W control unit 152 , buffers 153 to 158 , a selector 159 , and a range calculating unit 160 .
- the separating unit 151 is a functional unit that separates input signals. Specifically, the separating unit 151 separates the input signals indicating the light exposure amount acquired from the light receiving unit 14 for each predetermined light emission intensity and light emission light exposure pattern with reference to the signal related information.
- the R/W control unit 152 is a functional unit that stores (writes) each of the input signals separated by the separating unit 151 into the respective buffers 153 to 158 and takes out (reads) the signals from the respective buffers 153 to 158 .
- the buffers 153 to 158 store each of the input signals separated by the separating unit 151 .
- the selector 159 selects the input signal (read out) taken out from the buffers 153 to 158 depending on a predetermined reference and outputs the selected input signal to the range calculating unit 160 .
- the range calculating unit 160 is a functional unit that calculates a range from the range image generation apparatus 100 to the object OB for each pixel using the signal output from the selector 159 . Details of a process by the range calculating unit 160 will be described below.
- FIG. 3 is a diagram showing an example of a hardware configuration of the range image generation apparatus 100 .
- the range image generation apparatus 100 is realized by, for example, a range image sensing camera (so-called, a TOF camera) that can measure a range in a depth direction for each pixel depending on a reflection time of the near infrared light with which the object OB is irradiated.
- a range image sensing camera so-called, a TOF camera
- the range image generation apparatus 100 includes an arithmetic device 20 , an input device 21 , a display device 22 , a light emitting device 23 , and a light receiving element 24 .
- the arithmetic device 20 is a central unit that executes various processes of the range image generation apparatus 100 , and includes a central processing unit (CPU) 25 and a field programmable gate array (FPGA) 26 that perform an arithmetic process and a random access memory (RAM) 27 and a read only memory (ROM) 28 as main memory units.
- CPU central processing unit
- FPGA field programmable gate array
- RAM random access memory
- ROM read only memory
- the input device 21 is a device that receives setting of an image capturing mode or an image capturing instruction from the user of the range image generation apparatus 100 , and is, for example, a touch panel or a dial button.
- the display device 22 is a device that displays setting information, menu information, or the like of the range image generation apparatus 100 , and is, for example, a liquid crystal display.
- the light emitting device 23 is a device that emits the near infrared light toward the object OB, and is an electronic flash (strobe) device.
- the light receiving element 24 is an electronic element (image capturing element) that converts light and darkness of light into an electric signal, and is, for example, a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.
- CCD charge-coupled device
- CMOS complementary metal oxide semiconductor
- the control unit 12 and the signal processing unit 15 of the range image generation apparatus 100 are realized by a predetermined program that causes the FPGA 26 (or the CPU 25 ) to perform the processes.
- a program is stored in the RAM 27 or the ROM 28 , is loaded on the RAM 27 for execution, and is executed by the FPGA (or the CPU 25 ).
- the light emitting unit 13 is realized by the light emitting device 23 .
- the light receiving unit 14 is realized by the light receiving element 24 .
- FIG. 4 is a diagram showing an example of a timing chart of the light emission timing control signal S 1 and the light exposure timing control signal S 2 . As shown, the light emission timing control signal S 1 and the light exposure timing control signal S 2 include three types of light emission light exposure patterns.
- the three types of light emission light exposure patterns are an entire light exposure period W 0 , a range measurement period W 1 , and a background light measurement period WB.
- the entire light exposure period W 0 is a period in which light exposure is performed at a timing at which all of the reflected light of the light emitted from the light emitting unit 13 to the object OB can be received.
- the control unit 12 sets a pulse width of the light emission timing control signal S 1 to ⁇ t and sets a pulse width of the light exposure timing control signal S 2 to 2 ⁇ t which is twice ⁇ t. In addition, the control unit 12 synchronizes rising timings of the light emission timing control signal S 1 and the light exposure timing control signal S 2 with each other. The control unit 12 sets a time length of the entire light exposure period W 0 to 2 ⁇ t. Therefore, it is possible to set the entire light exposure period W 0 in which all of the reflected light from the object OB can be exposed, in synchronization with the irradiation with the near infrared light by the light emitting unit 13 .
- control unit 12 specifies the measurement range based on setting information of the image capturing mode, and determines a predetermined value corresponding to the measurement range as ⁇ t.
- the range measurement period W 1 is a period in which light exposure is performed at such a timing that a light reception amount is decreased as the reflected light is delayed with respect to the light emission timing.
- the control unit 12 synchronizes falling timings of the light emission timing control signal S 1 and the light exposure timing control signal S 2 with each other.
- the control unit 12 sets a time length of the range measurement period W 1 to 2 ⁇ t, similarly to the entire light exposure period W 0 . Therefore, it is possible to set the range measurement period W 1 in which the light exposure amount is decreased as the reflected light from the object OB is delayed, in synchronization with the irradiation with the near infrared light by the light emitting unit 13 .
- the background light measurement period WB is a period in which the light exposure is performed at such a timing that only background light other than the reflected light from the object OB is received.
- the control unit 12 synchronizes a rising timing of the light emission timing control signal S 1 and a fall timing of the light exposure timing control signal S 2 with each other.
- the control unit 12 sets a time length of the background light measurement period WB to 2 ⁇ t, similarly to the entire light exposure period W 0 . Therefore, it is possible to set the background light measurement period WB in which only the background light that does not include the reflected light is exposed.
- the range image generation apparatus 100 can obtain three types of light exposure amounts Q 0 , Q 1 , and QB for each pixel of the captured image of the object OB.
- ⁇ tTOF is a time (second) from when the light emitting unit 13 emits the near infrared light until the light receiving unit 14 receives the reflected light reflected by the object OB.
- the range to the object OB is in proportion to a delay time of the reflected light, the range to the object OB is obtained by a ratio between the light exposure amount Q 0 in the entire light exposure period W 0 and the light exposure amount Q 1 in the range measurement period W 1 .
- the range calculating unit 160 calculates a value of ⁇ tTOF using the following Equation (1).
- ⁇ tTOF ⁇ t ⁇ 1 ⁇ ( Q 1 ⁇ QB ) ⁇ /( Q 0 ⁇ QB ) (1)
- the range calculating unit 160 calculates the range L(m) to the object OB using the following Equation (2). It should be noted that c is the speed of light and a unit of c is (m/sec).
- measurement is performed by changing a light emission intensity of the near infrared light emitted from the light emitting unit 13 . Details of this measurement will be described below.
- FIG. 5 is a flowchart showing an example of a flow of a range image generation process executed by the range image generation apparatus 100 .
- the range image generation process starts with, for example, start-up of the range image generation apparatus 100 .
- the input receiving unit 11 monitors whether or not a setting input of the image capturing mode related to the measurement range from the range image generation apparatus 100 to the object OB has been received from the user (step S 001 ). When it is determined that the setting input has been received (Yes in step S 001 ), the input receiving unit 11 outputs the measurement range corresponding to the set image capturing mode to the control unit 12 .
- control unit 12 generates a light emission intensity control signal S 0 , a light emission timing control signal S 1 , and a light exposure timing control signal S 2 depending on the measurement range (step S 002 ). Specifically, the control unit 12 generates a light emission intensity control signal S 0 , a light emission timing control signal S 1 , and a light exposure timing control signal S 2 shown in FIG. 6 .
- FIG. 6 is a diagram showing an example of a timing chart of the light emission intensity control signal S 0 , the light emission timing control signal S 1 , and the light exposure timing control signal S 2 .
- the light emission intensity control signal S 0 has periods P 0 and P 1 of two types of different light emission intensities, and each of the periods P 0 and P 1 includes three types of light emission light exposure patterns. It should be noted that details of the light emission light exposure patterns are the same as those described above, and a description thereof is thus omitted.
- the control unit 12 generates two types of light emission intensity control signals S 0 having different emission intensities. Specifically, the control unit 12 sets each of time lengths of the periods P 0 and P 1 including a light emission intensity period P 0 of strong light emission and a light emission intensity period P 1 of weak light emission, to a length including the three types of light emission light exposure patterns described above. In other words, the control unit 12 generates the light emission intensity control signal S 0 , the light emission timing control signal S 1 , and the light exposure timing control signal S 2 so that the light emission light exposure patterns of the light emission timing control signal S 1 and the light exposure timing control signal S 2 are included in each of the strong light emission period P 0 and the weak light emission period P 1 .
- control unit 12 outputs the generated light emission intensity control signal S 0 and light emission timing control signal S 1 to the light emitting unit 13 (step S 003 ), and outputs the light exposure timing control signal S 2 to the light receiving unit 14 (step S 004 ).
- control unit 12 generates signal related information, and outputs the signal related information to the signal processing unit 15 .
- the light emitting unit 14 irradiates the object OB with the near infrared light at a light emission timing and a light emission intensity indicated by the light emission timing control signal S 1 and the light emission intensity control signal S 0 (step S 005 ).
- the light receiving unit 14 receives the reflected light from the object OB at a timing indicated by the light exposure timing control signal S 2 to perform light exposure (step S 006 ).
- the light receiving unit 14 obtains light exposure amounts Q 00 , Q 01 , and Q 0 B for each pixel of the captured image of the object OB in the strong light emission period P 0 .
- the light receiving unit 14 obtains light exposure amounts Q 10 , Q 11 , and Q 1 B in the weak light emission period P 1 .
- the light receiving unit 14 converts the light exposure amounts into electrical signals by photoelectric conversion, and outputs the converted electrical signals to the signal processing unit 15 . Therefore, the light exposure amounts obtained depending on the timing chart of FIG. 6 are output to the signal processing unit 15 in a time division manner using the same signal line.
- the signal processing unit 15 generates a range image using the input signal (step S 007 ). Specifically, the separating unit 151 of the signal processing unit 15 separates the input signals indicating the light exposure amounts for each light emission intensity and light emission light exposure pattern with reference to the signal related information. In addition, the R/W control unit 152 writes the signals separated by the separating unit 151 into the buffers 153 to 158 , respectively. For example, the R/W control unit 152 writes the signals indicating the light exposure amounts Q 00 , Q 01 , Q 0 B, Q 10 , Q 11 , and Q 1 B into the buffers 153 to 158 , respectively.
- the R/W control unit 152 reads the signal indicating the light exposure amount from each of the buffers 153 to 158 at a predetermined timing, and inputs the signal to the selector 159 .
- the selector 159 selects an input signal of a light exposure amount that is not saturated, and outputs the selected input signal to the range calculating unit 160 .
- the selector 159 determines whether or not the light exposure amount Q 00 of the entire light exposure period W 0 included in the strong light emission period P 0 is an a predetermined saturation value QS or more, for each pixel.
- the selector 159 selects signals indicating the light exposure amounts Q 10 , Q 11 , and Q 1 B obtained in each period of the light emission light exposure patterns included in the weak light emission period P 1 for pixels of which the value of Q 00 is equal to or more than the value of QS, and outputs the selected signals to the range calculating unit 160 .
- the selector 159 selects signals indicating the light exposure amounts Q 00 , Q 01 , and Q 0 B obtained in each period of the light emission light exposure patterns included in the strong light emission period P 0 for pixels of which the value of Q 00 is less than the value of QS, and outputs the selected signals to the range calculating unit 160 .
- the range calculating unit 160 calculates a range from the range image generation apparatus 100 to the object OB for each pixel using the input signal acquired from the selector 159 . Specifically, when the signals indicating the light exposure amounts of the weak light emission period P 1 are selected by the selector 159 , the range calculating unit 160 calculates the range L(m) to the object OB by the following Equation (3) using the values of Q 10 , Q 11 , and Q 1 B.
- the range calculating unit 160 calculates the range L(m) to the object OB for each pixel by the following Equation (4) using the values of Q 00 , Q 01 , and Q 0 B.
- the range calculating unit 160 calculates the range from the range image generation apparatus 100 to the object OB for each pixel of the captured image of the object OB.
- FIG. 7 is a diagram showing an example of an image capturing configuration including the range image generation apparatus 100 , an object OB 0 , and an object OB 1 .
- FIG. 8 is a diagram showing an example of an image (hereinafter, referred to as a Q 00 image) in which light exposure amounts Q 00 of the objects OB 0 and OB 1 image-captured in the strong light emission period P 0 are two-dimensionally arranged.
- FIG. 9 is a diagram showing an example of an image (hereinafter, referred to as a Q 10 image) in which light exposure amounts Q 10 of the objects OB 0 and OB 1 image-captured in the weak light emission period P 1 are two-dimensionally arranged.
- the object OB 0 indicates an object of which a surface has a high light reflectivity (for example, an object of which a surface is formed of a white material or a material having gloss).
- the object OB 1 indicates an object of which a surface has a low light reflectivity (for example, an object of which a surface is formed of a black material or a material that does not have gloss).
- the range image generation apparatus 100 calculates the range to the object using the light exposure amounts of the weak light emission period P 1 for the pixels of which the light exposure amounts of the image-captured object are saturated and using the light exposure amounts of the strong light emission period P 0 for the pixels of which the light exposure amounts of the image-captured object are not saturated, by the range image generation process described above. In other words, the range image generation apparatus 100 can accurately calculate the range to the object by replacing the pixels saturated in the Q 00 image by the light exposure amounts of the corresponding pixels in the Q 10 image.
- the range image generation apparatus 100 can generate the range image with higher accuracy.
- the range image generation apparatus 100 captures the image of the object at two types of different light emission intensities, and replaces the pixels of which the light exposure amounts are saturated by the light exposure amounts (range information) of the pixels that are not saturated to generate the range image. Therefore, the range image can be generated using measurement information that does not include saturation of the light exposure amounts or signal noise as much as possible.
- FIG. 10 is a diagram showing an example of a schematic configuration of a range image generation apparatus 100 according to a second embodiment.
- the range image generation apparatus 100 according to the present embodiment is different from the range image generation apparatus according to the first embodiment in that it includes a light reception amount control unit 16 .
- the light reception amount control unit 16 changes a light reception amount of a light receiving unit 14 based on a light reception amount control signal S 3 generated by a control unit 12 .
- the basic components of the range image generation apparatus 100 are the same as those of the range image generation apparatus according to the first embodiment, the basic components are denoted by the same reference numerals and a description thereof is omitted.
- the light reception amount control unit 1612 is realized by, for example, a device such as an iris or a liquid crystal shutter.
- the light reception amount control unit 16 changes a value of the iris based on the light reception amount control signal S 3 output from the control unit 12 , and adjusts an amount of light received by the light receiving unit 14 . Therefore, it is possible to change a light amount, that is, a light exposure amount, of reflected light from an object OB received by the light receiving unit 14 .
- the control unit 12 generates a light emission timing control signal S 1 , a light exposure timing control signal S 2 , and the light reception amount control signal S 3 .
- the light reception amount control signal S 3 is a signal for controlling a value of the iris by the light reception amount control unit 16 . It should be noted that the light emission timing control signal S 1 and the light exposure timing control signal S 2 are the same as those described above, and a description thereof is thus omitted.
- FIG. 11 is a diagram showing an example of a timing chart of the light emission timing control signal S 1 , the light exposure timing control signal S 2 , and the light reception amount control signal S 3 .
- the light reception amount control signal S 3 has periods P 2 and P 3 of two kinds of different light reception amounts (strong light amount and weak light amount).
- each of the periods P 2 and P 3 includes three types of light emission light exposure patterns described in the first embodiment.
- the control unit 12 generates the light reception amount control signal S 3 including a strong light amount period P 2 and a weak light amount period P 3 set to time lengths in which the light emission light exposure patterns are included. In addition, the control unit 12 outputs the light reception amount control signal S 3 to the light reception amount control unit 16 at a predetermined timing.
- the light reception amount control unit 16 controls the value of the iris based on the light reception amount control signal S 3 , and controls the light reception amount of the light receiving unit 14 .
- the control unit 12 outputs the light emission timing control signal S 1 to the light emitting unit 13 and outputs the light exposure timing control signal S 2 to the light receiving unit 14 , at a predetermined timing.
- the control unit 12 generates signal related information on the light emission timing control signal S 1 , the light exposure timing control signal S 2 , and the light reception amount control signal S 3 , and outputs the signal related information to a signal processing unit 15 .
- the light emitting unit 13 irradiates the object OB with near infrared light based on the light emission timing control signal S 1 .
- the light reception amount control unit 1612 changes the value of the iris based on the light reception amount control signal S 3 .
- the light receiving unit 14 performs light exposure by reflected light that is reflected on the object OB and returns. Therefore, the light receiving unit 14 obtains light exposure amounts Q 20 , Q 21 , and Q 2 B for each pixel of a captured image of the object OB in the strong light amount period P 2 .
- the light receiving unit 14 obtains light exposure amounts Q 30 , Q 31 , and Q 3 B in the weak light amount period P 3 .
- the light receiving unit 14 converts the light exposure amounts into electrical signals by photoelectric conversion, and outputs the converted electrical signals to the signal processing unit 15 .
- the signal processing unit 15 generates a range image using an input signal, as described above. Specifically, a separating unit 151 of the signal processing unit 15 separates input signals indicating the light exposure amounts for each light reception amount and light emission light exposure pattern with reference to the signal related information. In addition, an R/W control unit 152 writes the signals separated by the separating unit 151 into predetermined buffers 153 to 158 , respectively.
- the R/W control unit 152 reads the signal indicating the light exposure amount from each of the buffers 153 to 158 at a predetermined timing, and inputs the signal to a selector 159 .
- the selector 159 selects an input signal of a light exposure amount that is not saturated, and outputs the selected input signal to a range calculating unit 160 .
- the selector 159 determines whether or not the light exposure amount Q 20 of an entire light exposure period W 0 included in the strong light amount period P 2 is an a predetermined saturation value QS or more, for each pixel.
- the selector 159 selects signals indicating the light exposure amounts Q 30 , Q 31 , and Q 3 B obtained in each period of the light emission light exposure patterns included in the weak light amount period P 3 for pixels of which the value of Q 20 is equal to or more than the value of QS, and outputs the selected signals to the range calculating unit 160 .
- the selector 159 selects signals indicating the light exposure amounts Q 20 , Q 21 , and Q 2 B obtained in each period of the light emission light exposure patterns included in the strong light amount period P 2 for pixels of which the value of Q 20 is less than the value of QS, and outputs the selected signals to the range calculating unit 160 .
- the range calculating unit 160 calculates a range from the range image generation apparatus 100 to the object OB for each pixel using the input signal acquired from the selector 159 . Specifically, when the signals indicating the light exposure amounts of the weak light amount period P 3 are selected by the selector 159 , the range calculating unit 160 calculates the range L(m) to the object OB by the following Equation (5) using the values of Q 30 , Q 31 , and Q 3 B.
- the range calculating unit 160 calculates the range L(m) to the object OB for each pixel by the following Equation (6) using the values of Q 20 , Q 21 , and Q 2 B.
- the range image generation apparatus 100 according to the second embodiment has been described hereinabove.
- the range image generation apparatus 100 as described above can also generate the range image with higher accuracy.
- the range image generation apparatus 100 according to the present embodiment can provide a range image with high accuracy by adding simple hardware (iris or liquid crystal shutter).
- the generated range image is used for analysis (for example, human flow analysis or analysis of a worker's work in a factory) of three-dimensional recognition such as identification of a depth direction of the object.
- the respective functional units of the range image generation apparatus 100 described above are classified depending on main processing contents in order to make the functions of the range image generation apparatus 100 realized in the present embodiment easy to understand, and the present invention is not limited by a method of classifying the respective functions and names of the respective functions.
- the respective components of the range image generation apparatus 100 can also be classified into more components depending on processing contents.
- one component can also be classified so as to execute more processes.
- the present invention is not limited to the abovementioned embodiments, but includes various modified examples.
- the abovementioned embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to including all the components described.
- some of the components of any embodiment can be replaced by components of another embodiment, and components of another embodiment can be added to components of any embodiment.
- control lines or information lines considered to be necessary for the description are shown, and all control lines or information lines are not necessarily shown for a product. In fact, it may be considered that almost all the components are connected to each other.
Abstract
Description
- The present invention relates to a range image generation apparatus and a range image generation method.
- PTL1 discloses the following contents. When a range image showing a range to a target object present in a target space is estimated by a time-of-flight (TOF) method, if it is intended to realize an increase in a resolution of the range image or an increase in a frame rate of the range image, there is a risk that charged coupled device (CCD) saturation will occur and range accuracy will be decreased due to an influence of shot noise, ambient light, or the like. A light emission frequency selecting unit (7) receives reflected light S2 from the target object at a point in time in which light is not emitted from a light source and selects illumination light S1 having a light emission frequency hard to be influenced by the ambient light based on a frequency analysis of the reflected light S2. An image creating unit (6) receives reflected light of the illumination light S1 having this optimal light emission frequency and emitted from a prepared light source and creates the range to the target object, such that it is possible to suppress the influence of the ambient light at the time of receiving the light and it is thus possible to decrease an influence of noise on range accuracy when a resolution of the light receiving element unit (2) is increased.
- [Patent Literature 1] WO 2010/021090 A
- In recent years, the need for accurately grasping an object or a situation around the object in various fields such as a digital signage, a game, an autonomous mobile robot, an automobile, and security has increased. Therefore, approaches to increasing accuracy of object recognition using a so-called range image that two-dimensionally expresses a range to the object in addition to an image acquired by a camera have increased. There are a plurality of three-dimensional recognition techniques (methods) using the range image, but in a time-of-flight (TOF) method, a camera and a projector are controlled in combination, and a time until light projected by the projector is reflected on a target object and then returns is measured in real time for each pixel. According to this method, as compared with other three-dimensional recognition techniques such as a stereo image processing method, there is an advantage that it is hard to be influenced by a change in ambient light, a change in a temperature, and the like, or it is possible to measure a range with high accuracy and in real time.
- On the other hand, in the TOF method, since near infrared light is generally used, in an environment where the near infrared light is extremely strong (for example, the outdoors when the sun shines) or an object having gloss on a surface of a metal or the like, a signal obtained by exposing light by an image sensor is saturated, such that accurate range information can not be acquired. In a technique of
Patent Literature 1, it is described to suppress saturation of the image sensor by estimating the light emission frequency hard to be influenced by the ambient light and changing a frequency of illumination. However, according to the technique ofPatent Literature 1, in the case of an object having a high light reflectivity, for example, in the case in which an object surface has a mirror surface shape, the saturation of the image sensor can not be suppressed, and range accuracy can not be improved. - Therefore, an object of the present invention is to provide a range image generation apparatus capable of generating a range image with higher accuracy.
- The present application includes a plurality of means that solve at least some of the abovementioned problems, and an example of these means is as follows.
- To solve the abovementioned problems, a range image generation apparatus according to the present invention includes: a light emitting unit that emits light toward an object at different light emission intensities; a light receiving unit that captures an image of the object by receiving light including reflected light from the object and calculates a light exposure amount for each pixel; and a range calculating unit that calculates a range to the object by replacing a light exposure amount at a first light emission intensity by a light exposure amount at a second light emission intensity in a predetermined case and generates a range image.
- According to the range image generation apparatus according to the present invention, it is possible to generate a range image with higher accuracy. It should be noted that objects, configurations, and effects other than those described above will be clarified from a description of embodiments provided below.
-
FIG. 1 is a diagram showing an example of a schematic configuration of a range image generation apparatus according to a first embodiment of the present invention. -
FIG. 2 is a diagram showing an example of a detailed configuration of a signal processing unit according to first and second embodiments of the present invention. -
FIG. 3 is a diagram showing an example of a hardware configuration of the range image generation apparatus according to the first embodiment of the present invention. -
FIG. 4 is a diagram showing an example of a timing chart of a light emission timing control signal S1 and a light exposure timing control signal S2 according to the first and second embodiments of the present invention. -
FIG. 5 is a flowchart showing an example of a flow of a range image generation process executed by the range image generation apparatus according to the first embodiment of the present invention. -
FIG. 6 is a diagram showing an example of a timing chart of a light emission intensity control signal S0, a light emission timing control signal S1, and a light exposure timing control signal S2 according to the first embodiment of the present invention. -
FIG. 7 is a diagram showing an example of an image capturing configuration including the range image generation apparatus, an object OB0, and an object OB1 according to the first embodiment of the present invention. -
FIG. 8 is a diagram showing an example of an image in which light exposure amounts Q00 of the objects OB0 and OB1 image-captured in a strong light emission period P0 according to the first embodiment of the present invention are two-dimensionally arranged. -
FIG. 9 is a diagram showing an example of an image in which light exposure amounts Q10 of the objects OB0 and OB1 image-captured in a weak light emission period P1 according to the first embodiment of the present invention are two-dimensionally arranged. -
FIG. 10 is a diagram showing an example of a schematic configuration of a range image generation apparatus according to a second embodiment of the present invention. -
FIG. 11 is a diagram showing an example of a timing chart of a light emission timing control signal S1, a light exposure timing control signal S2, and a light reception amount control signal S3 according to the second embodiment of the present invention. - Hereinafter, a range image generation apparatus according to a first embodiment of the present invention will be described.
-
FIG. 1 is a diagram showing an example of a schematic configuration of a rangeimage generation apparatus 100 according to the present embodiment. The rangeimage generation apparatus 100 adopts a time-of-flight (TOF) method, and is an apparatus that generates a range image in which a range to an object OB is two-dimensionally arranged for each pixel. Specifically, the rangeimage generation apparatus 100 includes aninput receiving unit 11, acontrol unit 12, alight emitting unit 13, alight receiving unit 14, and asignal processing unit 15. - The
input receiving unit 11 is a functional unit that receives an instruction input from a user of the rangeimage generation apparatus 100. Specifically, theinput receiving unit 11 receives setting of a predetermined image capturing mode (for example, a short range mode, a medium range mode, a long range mode, or the like) related to a measurement range (for example, 1 to 3 m (short measurement range), 3 to 7 m (middle measurement range), 7 to 10 m (long measurement range), or the like) from the rangeimage generation apparatus 100 to the object OB from the user. In addition, theinput receiving unit 11 outputs the received setting information to a predetermined functional unit (for example, thecontrol unit 12 or the like). - The
control unit 12 is a functional unit that controls various processes executed by the rangeimage generation apparatus 100. Specifically, thecontrol unit 12 generates a light emission intensity control signal S0, a light emission timing control signal S1, and a light exposure timing control signal S2 depending on the measurement range. It should be noted that the light emission intensity control signal S0 is a signal that controls a degree of strength of light emission (light emission intensity) in thelight emitting unit 13. In addition, the light emission timing control signal S1 is a signal that controls a timing of light emission by thelight emitting unit 13 and a length of a light emission time. In addition, the light exposure timing control signal S2 is a signal that controls a timing of light exposure by thelight receiving unit 14 and a length of a light exposure time. - The
control unit 12 outputs the light emission intensity control signal S0 and the light emission timing control signal S1 to thelight emitting unit 13 and outputs the light exposure timing control signal S2 to thelight receiving unit 14, at a predetermined timing (for example, when a generation instruction of a range image by image-capturing of the object OB is received from the user). In addition, thecontrol unit 12 generates information (hereinafter, referred to as signal related information) on the timing indicated by these control signals, and outputs the information to thesignal processing unit 15. - The
light emitting unit 13 is a functional unit that emits near infrared light toward the object OB. Specifically, thelight emitting unit 13 emits the near infrared light toward the object OB at a timing and a light emission intensity indicated by the light emission timing control signal S1 and the light emission intensity control signal S0. - The
light receiving unit 14 is a functional unit that receives light including reflected light of the light with which the object OB is irradiated and calculates a light exposure amount. Specifically, thelight receiving unit 14 captures an image of the object OB by receiving (exposing) light including reflected light from the object OB at a timing indicated by the light exposure timing control signal S2. In addition, thelight receiving unit 14 calculates a light exposure amount for each pixel of the captured image of the object. In addition, thelight receiving unit 14 converts the light exposure amount into an electrical signal by photoelectric conversion, and outputs the converted electrical signal to thesignal processing unit 15. - The
signal processing unit 15 is a functional unit that calculates a range to the object OB for each pixel and generates a range image. Specifically, thesignal processing unit 15 acquires the photoelectrically converted electrical signal as an input signal from thelight receiving unit 14, and calculates the range to the object OB for each pixel using the electrical signal. In addition, thesignal processing unit 15 generates a range image in which each pixel having range information to the object OB is two-dimensionally arranged. -
FIG. 2 is a diagram showing an example of a detailed configuration of thesignal processing unit 15. Thesignal processing unit 15 includes aseparating unit 151, an R/W control unit 152,buffers 153 to 158, aselector 159, and arange calculating unit 160. - The separating
unit 151 is a functional unit that separates input signals. Specifically, the separatingunit 151 separates the input signals indicating the light exposure amount acquired from thelight receiving unit 14 for each predetermined light emission intensity and light emission light exposure pattern with reference to the signal related information. - The R/
W control unit 152 is a functional unit that stores (writes) each of the input signals separated by the separatingunit 151 into therespective buffers 153 to 158 and takes out (reads) the signals from therespective buffers 153 to 158. - The
buffers 153 to 158 store each of the input signals separated by the separatingunit 151. - The
selector 159 selects the input signal (read out) taken out from thebuffers 153 to 158 depending on a predetermined reference and outputs the selected input signal to therange calculating unit 160. - The
range calculating unit 160 is a functional unit that calculates a range from the rangeimage generation apparatus 100 to the object OB for each pixel using the signal output from theselector 159. Details of a process by therange calculating unit 160 will be described below. - The schematic configuration of the range
image generation apparatus 100 according to the present embodiment has been described hereinabove. -
FIG. 3 is a diagram showing an example of a hardware configuration of the rangeimage generation apparatus 100. The rangeimage generation apparatus 100 is realized by, for example, a range image sensing camera (so-called, a TOF camera) that can measure a range in a depth direction for each pixel depending on a reflection time of the near infrared light with which the object OB is irradiated. - As shown, the range
image generation apparatus 100 includes anarithmetic device 20, aninput device 21, adisplay device 22, alight emitting device 23, and alight receiving element 24. - The
arithmetic device 20 is a central unit that executes various processes of the rangeimage generation apparatus 100, and includes a central processing unit (CPU) 25 and a field programmable gate array (FPGA) 26 that perform an arithmetic process and a random access memory (RAM) 27 and a read only memory (ROM) 28 as main memory units. - The
input device 21 is a device that receives setting of an image capturing mode or an image capturing instruction from the user of the rangeimage generation apparatus 100, and is, for example, a touch panel or a dial button. - The
display device 22 is a device that displays setting information, menu information, or the like of the rangeimage generation apparatus 100, and is, for example, a liquid crystal display. - The
light emitting device 23 is a device that emits the near infrared light toward the object OB, and is an electronic flash (strobe) device. - The
light receiving element 24 is an electronic element (image capturing element) that converts light and darkness of light into an electric signal, and is, for example, a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. - The
control unit 12 and thesignal processing unit 15 of the rangeimage generation apparatus 100 are realized by a predetermined program that causes the FPGA 26 (or the CPU 25) to perform the processes. Such a program is stored in theRAM 27 or theROM 28, is loaded on theRAM 27 for execution, and is executed by the FPGA (or the CPU 25). In addition, thelight emitting unit 13 is realized by thelight emitting device 23. In addition, thelight receiving unit 14 is realized by thelight receiving element 24. - The hardware configuration of the range
image generation apparatus 100 according to the present embodiment has been described hereinabove. -
FIG. 4 is a diagram showing an example of a timing chart of the light emission timing control signal S1 and the light exposure timing control signal S2. As shown, the light emission timing control signal S1 and the light exposure timing control signal S2 include three types of light emission light exposure patterns. - As shown, the three types of light emission light exposure patterns are an entire light exposure period W0, a range measurement period W1, and a background light measurement period WB. The entire light exposure period W0 is a period in which light exposure is performed at a timing at which all of the reflected light of the light emitted from the
light emitting unit 13 to the object OB can be received. - The
control unit 12 sets a pulse width of the light emission timing control signal S1 to Δt and sets a pulse width of the light exposure timing control signal S2 to 2Δt which is twice Δt. In addition, thecontrol unit 12 synchronizes rising timings of the light emission timing control signal S1 and the light exposure timing control signal S2 with each other. Thecontrol unit 12 sets a time length of the entire light exposure period W0 to 2Δt. Therefore, it is possible to set the entire light exposure period W0 in which all of the reflected light from the object OB can be exposed, in synchronization with the irradiation with the near infrared light by thelight emitting unit 13. - It is necessary to increase a set value of Δt in proportion to a measurement range to the object OB to be measured. For this reason, the
control unit 12 specifies the measurement range based on setting information of the image capturing mode, and determines a predetermined value corresponding to the measurement range as Δt. - The range measurement period W1 is a period in which light exposure is performed at such a timing that a light reception amount is decreased as the reflected light is delayed with respect to the light emission timing. In the range measurement period W1, the
control unit 12 synchronizes falling timings of the light emission timing control signal S1 and the light exposure timing control signal S2 with each other. Thecontrol unit 12 sets a time length of the range measurement period W1 to 2Δt, similarly to the entire light exposure period W0. Therefore, it is possible to set the range measurement period W1 in which the light exposure amount is decreased as the reflected light from the object OB is delayed, in synchronization with the irradiation with the near infrared light by thelight emitting unit 13. - The background light measurement period WB is a period in which the light exposure is performed at such a timing that only background light other than the reflected light from the object OB is received. In the background light measurement period WB, the
control unit 12 synchronizes a rising timing of the light emission timing control signal S1 and a fall timing of the light exposure timing control signal S2 with each other. Thecontrol unit 12 sets a time length of the background light measurement period WB to 2Δt, similarly to the entire light exposure period W0. Therefore, it is possible to set the background light measurement period WB in which only the background light that does not include the reflected light is exposed. - With the light emission light exposure patterns as described above, the range
image generation apparatus 100 can obtain three types of light exposure amounts Q0, Q1, and QB for each pixel of the captured image of the object OB. - It should be noted that ΔtTOF is a time (second) from when the
light emitting unit 13 emits the near infrared light until thelight receiving unit 14 receives the reflected light reflected by the object OB. Here, since the range to the object OB is in proportion to a delay time of the reflected light, the range to the object OB is obtained by a ratio between the light exposure amount Q0 in the entire light exposure period W0 and the light exposure amount Q1 in the range measurement period W1. However, in either case, since the background light is included, it is necessary to cancel the background light using the light exposure amount QB of such a period. Therefore, therange calculating unit 160 calculates a value of ΔtTOF using the following Equation (1). -
ΔtTOF=Δt×{1−(Q1−QB)}/(Q0−QB) (1) - In addition, ΔtTOF is a round trip time from the
light emitting unit 13 to the object OB. Therefore, therange calculating unit 160 calculates the range L(m) to the object OB using the following Equation (2). It should be noted that c is the speed of light and a unit of c is (m/sec). -
L=(c×ΔtTOF)/2=(c×Δt)/2×{1−(Q1−QB)/(Q0−QB)} (2) - Conditions of both of the above Equations (1) and (2) are that the light exposure amounts Q0 and Q1 are not saturated. Since the light exposure amount Q1 is a light exposure amount decreased in proportion to the delay time of the reflected light, Q0>Q1. Therefore, when a control is performed so that the light exposure amount Q0 is not saturated, it is possible to create a condition that neither of the light exposure amounts Q0 nor Q1 is saturated.
- In the present embodiment, in order to obtain a light exposure amount that is not saturated, measurement is performed by changing a light emission intensity of the near infrared light emitted from the
light emitting unit 13. Details of this measurement will be described below. - [Description of Operation]
-
FIG. 5 is a flowchart showing an example of a flow of a range image generation process executed by the rangeimage generation apparatus 100. The range image generation process starts with, for example, start-up of the rangeimage generation apparatus 100. - When the range image generation process starts, the
input receiving unit 11 monitors whether or not a setting input of the image capturing mode related to the measurement range from the rangeimage generation apparatus 100 to the object OB has been received from the user (step S001). When it is determined that the setting input has been received (Yes in step S001), theinput receiving unit 11 outputs the measurement range corresponding to the set image capturing mode to thecontrol unit 12. - Then, the
control unit 12 generates a light emission intensity control signal S0, a light emission timing control signal S1, and a light exposure timing control signal S2 depending on the measurement range (step S002). Specifically, thecontrol unit 12 generates a light emission intensity control signal S0, a light emission timing control signal S1, and a light exposure timing control signal S2 shown inFIG. 6 . -
FIG. 6 is a diagram showing an example of a timing chart of the light emission intensity control signal S0, the light emission timing control signal S1, and the light exposure timing control signal S2. As shown, the light emission intensity control signal S0 has periods P0 and P1 of two types of different light emission intensities, and each of the periods P0 and P1 includes three types of light emission light exposure patterns. It should be noted that details of the light emission light exposure patterns are the same as those described above, and a description thereof is thus omitted. - The
control unit 12 generates two types of light emission intensity control signals S0 having different emission intensities. Specifically, thecontrol unit 12 sets each of time lengths of the periods P0 and P1 including a light emission intensity period P0 of strong light emission and a light emission intensity period P1 of weak light emission, to a length including the three types of light emission light exposure patterns described above. In other words, thecontrol unit 12 generates the light emission intensity control signal S0, the light emission timing control signal S1, and the light exposure timing control signal S2 so that the light emission light exposure patterns of the light emission timing control signal S1 and the light exposure timing control signal S2 are included in each of the strong light emission period P0 and the weak light emission period P1. - Then, the
control unit 12 outputs the generated light emission intensity control signal S0 and light emission timing control signal S1 to the light emitting unit 13 (step S003), and outputs the light exposure timing control signal S2 to the light receiving unit 14 (step S004). In addition, thecontrol unit 12 generates signal related information, and outputs the signal related information to thesignal processing unit 15. - Then, the
light emitting unit 14 irradiates the object OB with the near infrared light at a light emission timing and a light emission intensity indicated by the light emission timing control signal S1 and the light emission intensity control signal S0 (step S005). In addition, thelight receiving unit 14 receives the reflected light from the object OB at a timing indicated by the light exposure timing control signal S2 to perform light exposure (step S006). - Therefore, the
light receiving unit 14 obtains light exposure amounts Q00, Q01, and Q0B for each pixel of the captured image of the object OB in the strong light emission period P0. In addition, thelight receiving unit 14 obtains light exposure amounts Q10, Q11, and Q1B in the weak light emission period P1. In addition, thelight receiving unit 14 converts the light exposure amounts into electrical signals by photoelectric conversion, and outputs the converted electrical signals to thesignal processing unit 15. Therefore, the light exposure amounts obtained depending on the timing chart ofFIG. 6 are output to thesignal processing unit 15 in a time division manner using the same signal line. - Then, the
signal processing unit 15 generates a range image using the input signal (step S007). Specifically, the separatingunit 151 of thesignal processing unit 15 separates the input signals indicating the light exposure amounts for each light emission intensity and light emission light exposure pattern with reference to the signal related information. In addition, the R/W control unit 152 writes the signals separated by the separatingunit 151 into thebuffers 153 to 158, respectively. For example, the R/W control unit 152 writes the signals indicating the light exposure amounts Q00, Q01, Q0B, Q10, Q11, and Q1B into thebuffers 153 to 158, respectively. - Next, the R/
W control unit 152 reads the signal indicating the light exposure amount from each of thebuffers 153 to 158 at a predetermined timing, and inputs the signal to theselector 159. Theselector 159 selects an input signal of a light exposure amount that is not saturated, and outputs the selected input signal to therange calculating unit 160. - Specifically, the
selector 159 determines whether or not the light exposure amount Q00 of the entire light exposure period W0 included in the strong light emission period P0 is an a predetermined saturation value QS or more, for each pixel. Theselector 159 selects signals indicating the light exposure amounts Q10, Q11, and Q1B obtained in each period of the light emission light exposure patterns included in the weak light emission period P1 for pixels of which the value of Q00 is equal to or more than the value of QS, and outputs the selected signals to therange calculating unit 160. - On the other hand, the
selector 159 selects signals indicating the light exposure amounts Q00, Q01, and Q0B obtained in each period of the light emission light exposure patterns included in the strong light emission period P0 for pixels of which the value of Q00 is less than the value of QS, and outputs the selected signals to therange calculating unit 160. - The
range calculating unit 160 calculates a range from the rangeimage generation apparatus 100 to the object OB for each pixel using the input signal acquired from theselector 159. Specifically, when the signals indicating the light exposure amounts of the weak light emission period P1 are selected by theselector 159, therange calculating unit 160 calculates the range L(m) to the object OB by the following Equation (3) using the values of Q10, Q11, and Q1B. -
L=(c×Δt)/2×{1−(Q11−Q1B)/(Q10−Q1B)} (3) - On the other hand, when the signals indicating the light exposure amounts of the strong light emission period P0 are selected by the
selector 159, therange calculating unit 160 calculates the range L(m) to the object OB for each pixel by the following Equation (4) using the values of Q00, Q01, and Q0B. -
L=(c×Δt)/2×{1−(Q01−Q0B)/(Q00−Q0B)} (4) - In this manner, the
range calculating unit 160 calculates the range from the rangeimage generation apparatus 100 to the object OB for each pixel of the captured image of the object OB. - Next, an example of generation of a range image will be described with reference to
FIGS. 7 to 9 . -
FIG. 7 is a diagram showing an example of an image capturing configuration including the rangeimage generation apparatus 100, an object OB0, and an object OB1. -
FIG. 8 is a diagram showing an example of an image (hereinafter, referred to as a Q00 image) in which light exposure amounts Q00 of the objects OB0 and OB1 image-captured in the strong light emission period P0 are two-dimensionally arranged. -
FIG. 9 is a diagram showing an example of an image (hereinafter, referred to as a Q10 image) in which light exposure amounts Q10 of the objects OB0 and OB1 image-captured in the weak light emission period P1 are two-dimensionally arranged. - It should be noted that the object OB0 indicates an object of which a surface has a high light reflectivity (for example, an object of which a surface is formed of a white material or a material having gloss). In addition, the object OB1 indicates an object of which a surface has a low light reflectivity (for example, an object of which a surface is formed of a black material or a material that does not have gloss).
- As shown in
FIG. 8 , in the Q00 image in the strong light emission period P0, a light exposure amount of the object OB0 is saturated, such that it is impossible to calculate an accurate range from the range image apparatus to the object OB0. On the other hand, even in the Q00 image in the strong light emission period P0, a light exposure amount of the object OB1 is not saturated, and an accurate range can thus be calculated. - In addition, as shown in
FIG. 9 , in the Q10 image in the weak light emission period P1, light exposure amounts of either of the objects OB0 and OB1 are not saturated. However, the object OB1 appears very dark, such that a signal to noise (S/N) ratio of a signal is decreased due to shot noise or quantization at the time of analog to digital (A/D) conversion in thelight receiving unit 14, which can cause an error in range calculation. - Therefore, the range
image generation apparatus 100 calculates the range to the object using the light exposure amounts of the weak light emission period P1 for the pixels of which the light exposure amounts of the image-captured object are saturated and using the light exposure amounts of the strong light emission period P0 for the pixels of which the light exposure amounts of the image-captured object are not saturated, by the range image generation process described above. In other words, the rangeimage generation apparatus 100 can accurately calculate the range to the object by replacing the pixels saturated in the Q00 image by the light exposure amounts of the corresponding pixels in the Q10 image. - As described above, the range
image generation apparatus 100 can generate the range image with higher accuracy. In particular, the rangeimage generation apparatus 100 captures the image of the object at two types of different light emission intensities, and replaces the pixels of which the light exposure amounts are saturated by the light exposure amounts (range information) of the pixels that are not saturated to generate the range image. Therefore, the range image can be generated using measurement information that does not include saturation of the light exposure amounts or signal noise as much as possible. -
FIG. 10 is a diagram showing an example of a schematic configuration of a rangeimage generation apparatus 100 according to a second embodiment. The rangeimage generation apparatus 100 according to the present embodiment is different from the range image generation apparatus according to the first embodiment in that it includes a light reception amount control unit 16. The light reception amount control unit 16 changes a light reception amount of alight receiving unit 14 based on a light reception amount control signal S3 generated by acontrol unit 12. It should be noted that since basic components of the rangeimage generation apparatus 100 are the same as those of the range image generation apparatus according to the first embodiment, the basic components are denoted by the same reference numerals and a description thereof is omitted. - The light reception amount control unit 1612 is realized by, for example, a device such as an iris or a liquid crystal shutter. The light reception amount control unit 16 changes a value of the iris based on the light reception amount control signal S3 output from the
control unit 12, and adjusts an amount of light received by thelight receiving unit 14. Therefore, it is possible to change a light amount, that is, a light exposure amount, of reflected light from an object OB received by thelight receiving unit 14. - The
control unit 12 generates a light emission timing control signal S1, a light exposure timing control signal S2, and the light reception amount control signal S3. The light reception amount control signal S3 is a signal for controlling a value of the iris by the light reception amount control unit 16. It should be noted that the light emission timing control signal S1 and the light exposure timing control signal S2 are the same as those described above, and a description thereof is thus omitted. -
FIG. 11 is a diagram showing an example of a timing chart of the light emission timing control signal S1, the light exposure timing control signal S2, and the light reception amount control signal S3. As shown, the light reception amount control signal S3 has periods P2 and P3 of two kinds of different light reception amounts (strong light amount and weak light amount). In addition, each of the periods P2 and P3 includes three types of light emission light exposure patterns described in the first embodiment. - The
control unit 12 generates the light reception amount control signal S3 including a strong light amount period P2 and a weak light amount period P3 set to time lengths in which the light emission light exposure patterns are included. In addition, thecontrol unit 12 outputs the light reception amount control signal S3 to the light reception amount control unit 16 at a predetermined timing. - The light reception amount control unit 16 controls the value of the iris based on the light reception amount control signal S3, and controls the light reception amount of the
light receiving unit 14. As in the first embodiment, thecontrol unit 12 outputs the light emission timing control signal S1 to thelight emitting unit 13 and outputs the light exposure timing control signal S2 to thelight receiving unit 14, at a predetermined timing. In addition, thecontrol unit 12 generates signal related information on the light emission timing control signal S1, the light exposure timing control signal S2, and the light reception amount control signal S3, and outputs the signal related information to asignal processing unit 15. - The
light emitting unit 13 irradiates the object OB with near infrared light based on the light emission timing control signal S1. The light reception amount control unit 1612 changes the value of the iris based on the light reception amount control signal S3. Thelight receiving unit 14 performs light exposure by reflected light that is reflected on the object OB and returns. Therefore, thelight receiving unit 14 obtains light exposure amounts Q20, Q21, and Q2B for each pixel of a captured image of the object OB in the strong light amount period P2. In addition, thelight receiving unit 14 obtains light exposure amounts Q30, Q31, and Q3B in the weak light amount period P3. In addition, thelight receiving unit 14 converts the light exposure amounts into electrical signals by photoelectric conversion, and outputs the converted electrical signals to thesignal processing unit 15. - The
signal processing unit 15 generates a range image using an input signal, as described above. Specifically, aseparating unit 151 of thesignal processing unit 15 separates input signals indicating the light exposure amounts for each light reception amount and light emission light exposure pattern with reference to the signal related information. In addition, an R/W control unit 152 writes the signals separated by the separatingunit 151 intopredetermined buffers 153 to 158, respectively. - Next, the R/
W control unit 152 reads the signal indicating the light exposure amount from each of thebuffers 153 to 158 at a predetermined timing, and inputs the signal to aselector 159. Theselector 159 selects an input signal of a light exposure amount that is not saturated, and outputs the selected input signal to arange calculating unit 160. - Specifically, the
selector 159 determines whether or not the light exposure amount Q20 of an entire light exposure period W0 included in the strong light amount period P2 is an a predetermined saturation value QS or more, for each pixel. Theselector 159 selects signals indicating the light exposure amounts Q30, Q31, and Q3B obtained in each period of the light emission light exposure patterns included in the weak light amount period P3 for pixels of which the value of Q20 is equal to or more than the value of QS, and outputs the selected signals to therange calculating unit 160. - On the other hand, the
selector 159 selects signals indicating the light exposure amounts Q20, Q21, and Q2B obtained in each period of the light emission light exposure patterns included in the strong light amount period P2 for pixels of which the value of Q20 is less than the value of QS, and outputs the selected signals to therange calculating unit 160. - The
range calculating unit 160 calculates a range from the rangeimage generation apparatus 100 to the object OB for each pixel using the input signal acquired from theselector 159. Specifically, when the signals indicating the light exposure amounts of the weak light amount period P3 are selected by theselector 159, therange calculating unit 160 calculates the range L(m) to the object OB by the following Equation (5) using the values of Q30, Q31, and Q3B. -
L=(c×Δt)/2×{1−(Q31−Q3B)/(Q30−Q3B)} (5) - On the other hand, when the signals indicating the light exposure amounts of the strong light amount period P2 are selected by the
selector 159, therange calculating unit 160 calculates the range L(m) to the object OB for each pixel by the following Equation (6) using the values of Q20, Q21, and Q2B. -
L=(c×Δt)/2×{1−(Q21−Q2B)/(Q20−Q2B)} (6) - The range
image generation apparatus 100 according to the second embodiment has been described hereinabove. The rangeimage generation apparatus 100 as described above can also generate the range image with higher accuracy. In particular, for example, even in the case where it is difficult to control a light emission amount due to a restriction or the like of a light source side, the rangeimage generation apparatus 100 according to the present embodiment can provide a range image with high accuracy by adding simple hardware (iris or liquid crystal shutter). - The generated range image is used for analysis (for example, human flow analysis or analysis of a worker's work in a factory) of three-dimensional recognition such as identification of a depth direction of the object.
- The respective functional units of the range
image generation apparatus 100 described above are classified depending on main processing contents in order to make the functions of the rangeimage generation apparatus 100 realized in the present embodiment easy to understand, and the present invention is not limited by a method of classifying the respective functions and names of the respective functions. In addition, the respective components of the rangeimage generation apparatus 100 can also be classified into more components depending on processing contents. In addition, one component can also be classified so as to execute more processes. - In addition, the present invention is not limited to the abovementioned embodiments, but includes various modified examples. For example, the abovementioned embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to including all the components described. In addition, some of the components of any embodiment can be replaced by components of another embodiment, and components of another embodiment can be added to components of any embodiment. In addition, it is possible to add, delete, and replace other components with respect to some of the components of the respective embodiments.
- In the above description, control lines or information lines considered to be necessary for the description are shown, and all control lines or information lines are not necessarily shown for a product. In fact, it may be considered that almost all the components are connected to each other.
-
- 100 range image generation apparatus
- 11 input receiving unit
- 12 control unit
- 13 light emitting unit
- 14 light receiving unit
- 15 signal processing unit
- 151 separating unit
- 152 R/W control unit
- 153 to 158 buffer
- 159 selector
- 160 range calculating unit
- 20 arithmetic device
- 21 input device
- 22 display device
- 23 light emitting device
- 24 light receiving element
Claims (10)
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2016/062419 WO2017183114A1 (en) | 2016-04-19 | 2016-04-19 | Distance image generation device and distance image generation method |
Publications (2)
Publication Number | Publication Date |
---|---|
US20190129034A1 true US20190129034A1 (en) | 2019-05-02 |
US10928518B2 US10928518B2 (en) | 2021-02-23 |
Family
ID=60115758
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/094,901 Active US10928518B2 (en) | 2016-04-19 | 2016-04-19 | Range image generation apparatus and range image generation method |
Country Status (5)
Country | Link |
---|---|
US (1) | US10928518B2 (en) |
JP (1) | JP6751137B2 (en) |
CN (1) | CN109073754B (en) |
TW (1) | TWI648549B (en) |
WO (1) | WO2017183114A1 (en) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190171222A1 (en) * | 2017-12-05 | 2019-06-06 | Shenzhen Silver Star Intelligent Technology Co., Ltd. | Mobile Robot |
EP3663801A1 (en) * | 2018-12-07 | 2020-06-10 | Infineon Technologies AG | Time of flight sensor module, method, apparatus and computer program for determining distance information based on time of flight sensor data |
WO2021004944A1 (en) * | 2019-07-05 | 2021-01-14 | Sony Semiconductor Solutions Corporation | Time-of-flight sensing circuitry with different imaging modes and method for operating such a time-of-flight sensing circuitry |
US20210297574A1 (en) * | 2020-03-18 | 2021-09-23 | Sony Olympus Medical Solutions Inc. | Control device and medical observation system |
US11347232B2 (en) * | 2017-10-26 | 2022-05-31 | Shenzhen Silver Star Intelligent Technology Co., Ltd. | Mobile robot |
EP4060284A4 (en) * | 2019-11-15 | 2023-01-04 | Panasonic Intellectual Property Management Co., Ltd. | Sensing device and information processing device |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP7135846B2 (en) * | 2018-12-27 | 2022-09-13 | 株式会社デンソー | Object detection device and object detection method |
JP7234171B2 (en) * | 2019-04-25 | 2023-03-07 | キヤノン株式会社 | Imaging device and its control method |
TWI703341B (en) * | 2019-06-04 | 2020-09-01 | 精準基因生物科技股份有限公司 | Time-of-flight ranging device and time-of-flight ranging method |
JP7175872B2 (en) * | 2019-11-14 | 2022-11-21 | 株式会社日立エルジーデータストレージ | rangefinder |
WO2021107036A1 (en) * | 2019-11-27 | 2021-06-03 | ヌヴォトンテクノロジージャパン株式会社 | Distance measurement and imaging device |
JP2022141316A (en) * | 2021-03-15 | 2022-09-29 | ソニーセミコンダクタソリューションズ株式会社 | Image capturing device and image capturing system |
CN113296114A (en) * | 2021-05-21 | 2021-08-24 | Oppo广东移动通信有限公司 | dTOF depth image acquisition method, dTOF depth image acquisition device, electronic device, and medium |
JP2023068471A (en) * | 2021-11-02 | 2023-05-17 | 株式会社デンソー | Control device, control method, and control program |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160178734A1 (en) * | 2013-06-27 | 2016-06-23 | Panasonic Intellectual Property Management Co., Ltd. | Distance measuring device and solid-state image sensor |
US20160320486A1 (en) * | 2014-01-14 | 2016-11-03 | Panasonic Intellectual Property Management Co., Ltd. | Range imaging apparatus and range imaging method |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4399693B2 (en) | 2000-09-21 | 2010-01-20 | ソニー株式会社 | Imaging device |
US6753876B2 (en) * | 2001-12-21 | 2004-06-22 | General Electric Company | Method for high dynamic range image construction based on multiple images with multiple illumination intensities |
JP2005295442A (en) * | 2004-04-05 | 2005-10-20 | Hitachi Kokusai Electric Inc | Image pickup device and image pickup method |
JP4777087B2 (en) * | 2005-03-03 | 2011-09-21 | 富士フイルム株式会社 | Imaging apparatus, imaging method, imaging program, image recording output system, and image recording output method |
JP4321540B2 (en) * | 2006-03-30 | 2009-08-26 | 株式会社豊田中央研究所 | Object detection device |
JP4897350B2 (en) * | 2006-05-16 | 2012-03-14 | 株式会社リコー | Image recording method and image recording apparatus |
JP5308088B2 (en) * | 2008-07-24 | 2013-10-09 | パナソニック株式会社 | Distance image sensor |
JPWO2010021090A1 (en) | 2008-08-20 | 2012-01-26 | パナソニック株式会社 | Distance estimation device, distance estimation method, program, integrated circuit, and camera |
JP2010204304A (en) * | 2009-03-02 | 2010-09-16 | Panasonic Corp | Image capturing device, operator monitoring device, method for measuring distance to face |
EP2395369A1 (en) * | 2010-06-09 | 2011-12-14 | Thomson Licensing | Time-of-flight imager. |
JP5802917B2 (en) * | 2011-04-06 | 2015-11-04 | オプテックス株式会社 | Distance image camera and distance measuring method using the same |
JP2012225807A (en) | 2011-04-20 | 2012-11-15 | Optex Co Ltd | Distance image camera and distance image synthesis method |
JP2016040520A (en) * | 2013-01-10 | 2016-03-24 | 三洋電機株式会社 | Object detection device |
JP6424338B2 (en) * | 2014-06-09 | 2018-11-21 | パナソニックIpマネジメント株式会社 | Ranging device |
-
2016
- 2016-04-19 US US16/094,901 patent/US10928518B2/en active Active
- 2016-04-19 WO PCT/JP2016/062419 patent/WO2017183114A1/en active Application Filing
- 2016-04-19 CN CN201680084945.3A patent/CN109073754B/en active Active
- 2016-04-19 JP JP2018512687A patent/JP6751137B2/en active Active
-
2017
- 2017-03-27 TW TW106110113A patent/TWI648549B/en active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160178734A1 (en) * | 2013-06-27 | 2016-06-23 | Panasonic Intellectual Property Management Co., Ltd. | Distance measuring device and solid-state image sensor |
US20160320486A1 (en) * | 2014-01-14 | 2016-11-03 | Panasonic Intellectual Property Management Co., Ltd. | Range imaging apparatus and range imaging method |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11347232B2 (en) * | 2017-10-26 | 2022-05-31 | Shenzhen Silver Star Intelligent Technology Co., Ltd. | Mobile robot |
US20190171222A1 (en) * | 2017-12-05 | 2019-06-06 | Shenzhen Silver Star Intelligent Technology Co., Ltd. | Mobile Robot |
US10725477B2 (en) * | 2017-12-05 | 2020-07-28 | Shenzhen Silver Star Intelligent Technology Co., Ltd. | Mobile robot |
EP3663801A1 (en) * | 2018-12-07 | 2020-06-10 | Infineon Technologies AG | Time of flight sensor module, method, apparatus and computer program for determining distance information based on time of flight sensor data |
US11879993B2 (en) | 2018-12-07 | 2024-01-23 | Infineon Technologies Ag | Time of flight sensor module, method, apparatus and computer program for determining distance information based on time of flight sensor data |
WO2021004944A1 (en) * | 2019-07-05 | 2021-01-14 | Sony Semiconductor Solutions Corporation | Time-of-flight sensing circuitry with different imaging modes and method for operating such a time-of-flight sensing circuitry |
EP4060284A4 (en) * | 2019-11-15 | 2023-01-04 | Panasonic Intellectual Property Management Co., Ltd. | Sensing device and information processing device |
US20210297574A1 (en) * | 2020-03-18 | 2021-09-23 | Sony Olympus Medical Solutions Inc. | Control device and medical observation system |
Also Published As
Publication number | Publication date |
---|---|
TW201812334A (en) | 2018-04-01 |
JP6751137B2 (en) | 2020-09-02 |
CN109073754B (en) | 2022-07-01 |
WO2017183114A1 (en) | 2017-10-26 |
JPWO2017183114A1 (en) | 2019-02-28 |
CN109073754A (en) | 2018-12-21 |
TWI648549B (en) | 2019-01-21 |
US10928518B2 (en) | 2021-02-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10928518B2 (en) | Range image generation apparatus and range image generation method | |
US11769775B2 (en) | Distance-measuring imaging device, distance measuring method of distance-measuring imaging device, and solid-state imaging device | |
JP5743390B2 (en) | Ranging device and ranging method | |
US11448757B2 (en) | Distance measuring device | |
CN108369089B (en) | 3D image measuring device and method | |
US8159598B2 (en) | Distance estimation apparatus, distance estimation method, storage medium storing program, integrated circuit, and camera | |
EP3379291A1 (en) | Imaging device and solid-state imaging element used in same | |
US9068831B2 (en) | Image processing apparatus and image processing method | |
WO2016208215A1 (en) | Distance sensor and user interface device | |
WO2015128915A1 (en) | Distance measurement device and distance measurement method | |
US10872434B2 (en) | Image processing apparatus and method | |
EP3040754B1 (en) | Imaging device, method, and program | |
JP2010190675A (en) | Distance image sensor system and method of generating distance image | |
CN104219455B (en) | The method and information process unit that imaging device, detection flash | |
WO2017056776A1 (en) | Projector device equipped with distance image acquisition device and projection method | |
US20170307869A1 (en) | Microscope and method for obtaining a high dynamic range synthesized image of an object | |
US11754717B2 (en) | Distance measurement device having external light illuminance measurement function and external light illuminance measurement method | |
JP2020052001A (en) | Depth acquisition device, depth acquisition method, and program | |
US20200145641A1 (en) | Image processing apparatus and method | |
US11610339B2 (en) | Imaging processing apparatus and method extracting a second RGB ToF feature points having a correlation between the first RGB and TOF feature points | |
WO2021084891A1 (en) | Movement amount estimation device, movement amount estimation method, movement amount estimation program, and movement amount estimation system | |
JP2017191082A (en) | Bright-spot image acquisition apparatus and bright-spot image acquisition method | |
KR102101864B1 (en) | Apparatus for detecting depth map | |
JP2014021100A (en) | Three-dimensional information detection device, and three-dimensional information detection method | |
US20230078063A1 (en) | Distance measurement device and distance measurement system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: HITACHI-LG DATA STORAGE, INC., JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YOSHIDA, DAISUKE;NONAKA, YUUICHI;SIGNING DATES FROM 20181017 TO 20190206;REEL/FRAME:048404/0228 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |