WO2023286542A1 - 物体検知装置および物体検知方法 - Google Patents

物体検知装置および物体検知方法 Download PDF

Info

Publication number
WO2023286542A1
WO2023286542A1 PCT/JP2022/024738 JP2022024738W WO2023286542A1 WO 2023286542 A1 WO2023286542 A1 WO 2023286542A1 JP 2022024738 W JP2022024738 W JP 2022024738W WO 2023286542 A1 WO2023286542 A1 WO 2023286542A1
Authority
WO
WIPO (PCT)
Prior art keywords
object detection
exposure
image
light
pixel signal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/024738
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
哲郎 奥山
良直 河合
正樹 金丸
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nuvoton Technology Corp Japan
Original Assignee
Nuvoton Technology Corp Japan
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nuvoton Technology Corp Japan filed Critical Nuvoton Technology Corp Japan
Priority to JP2022544426A priority Critical patent/JP7220835B1/ja
Priority to CN202280049134.5A priority patent/CN117616310A/zh
Priority to EP22841886.9A priority patent/EP4372419A4/en
Publication of WO2023286542A1 publication Critical patent/WO2023286542A1/ja
Priority to US18/408,397 priority patent/US12529796B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G01S17/08—Systems determining position data of a target for measuring distance only
    • G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G01S17/08—Systems determining position data of a target for measuring distance only
    • G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
    • G01S17/18—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves wherein range gates are used
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G01S17/894—Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/483—Details of pulse systems
    • G01S7/484—Transmitters
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/483—Details of pulse systems
    • G01S7/486—Receivers
    • G01S7/4861—Circuits for detection, sampling, integration or read-out
    • G01S7/4863—Detector arrays, e.g. charge-transfer gates
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/483—Details of pulse systems
    • G01S7/486—Receivers
    • G01S7/4865—Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single two-dimensional [2D] image sensor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/81—Camera processing pipelines; Components thereof for suppressing or minimising disturbance in the image signal generation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/131—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing infrared wavelengths
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/703—SSIS architectures incorporating pixels for producing signals other than image signals
    • H04N25/705—Pixels for depth measurement, e.g. RGBZ

Definitions

  • the present disclosure relates to an object detection device and an object detection method that generate a range image.
  • Japanese Patent Laid-Open No. 2004-100001 treats pixels for which distance measurement success or failure is different between a background distance image and a target distance image in the same manner as pixels in which a distance change is detected, thereby reducing omissions in extracting pixels in which a moving object appears.
  • An intruder detection device is disclosed.
  • Patent document 2 associates three-dimensional data such as distance image data with two-dimensional image data, and uses the distance image data and the two-dimensional image data complementarily to obtain flexible and highly reliable recognition results. Disclosed is a method for integrating three-dimensional data and two-dimensional image data, and a robust monitoring/watching system using the method.
  • an object of the present disclosure is to provide an object detection device and an object detection method that suppress generation of invalid pixels caused by nearby dust.
  • an object detection device comprising: a light emitting unit that emits light; a sensor that generates a first pixel signal by exposure and generates a second pixel signal by exposing reflected light according to a combination of the light emission and the second exposure; and the first pixel signal and the second pixel signal. and a distance calculation unit that generates a distance image by ratio calculation using the distance from the object included in the first interval from 0 to a predetermined value from the object detection device.
  • the timing is set so that the reflected wave is not exposed, and the time difference between the light emission and the second exposure is greater than the time difference between the light emission and the first exposure.
  • an object detection method is an object detection method in an object detection device including a light emitting unit that emits light and a sensor that has pixels that expose reflected light, wherein the light emitted by the light emitting unit and the generating a first pixel signal by exposing the reflected light in accordance with a set of the first exposure, generating a second pixel signal by exposing the reflected light in accordance with the set of the light emission and the second exposure;
  • a distance image is generated by a ratio calculation using the pixel signal and the second pixel signal, and the timing of the first exposure is for an object included in a first section in which the distance from the object detection device is 0 to a predetermined value.
  • the time difference between the light emission and the second exposure is greater than the time difference between the light emission and the first exposure.
  • the object detection device and object detection method of the present disclosure can suppress the generation of invalid pixels caused by nearby dust.
  • FIG. 1 is a block diagram showing a configuration example of an object detection device according to an embodiment.
  • FIG. 2 is a diagram showing a pixel arrangement example of the image sensor according to the embodiment.
  • FIG. 3 is a time chart showing an operation example of the object detection device according to the embodiment.
  • FIG. 4 is a flow chart showing an example of an object detection method according to the embodiment.
  • FIG. 5 is a diagram for explaining flare caused by dust.
  • FIG. 6 is a diagram schematically showing a luminance image and a distance image with and without dust.
  • FIG. 7 is a diagram for explaining the influence of dust on the distance image.
  • FIG. 5 is a diagram for explaining lens flare caused by dust.
  • the figure schematically shows a light source of an object detection device and a camera including a plurality of optical lenses and an image sensor.
  • the illumination light from the light source is reflected by the object OB to return to the camera as reflected light L1 and by the dust D0 to return to the camera as reflected light L2.
  • the reflected light L2 When the distance from the camera to the dust D0 is long, the reflected light L2 is weak and does not affect the imaging of the camera. However, when the distance from the camera to the dust D0 is short, the reflected light L2 becomes strong.
  • the strong reflected light L2 is incident on the image sensor and causes reflected light L3 as lens flare on the surfaces of the plurality of optical lenses. Part of the reflected light L3 is incident on the image sensor and causes a ghost.
  • the reflected light L2 of the dust D0 is incident on the image sensor, and the reflected light L3 and the reflected light L1 caused by the lens flare overlap each other.
  • FIG. 6 is a diagram schematically showing an example of a luminance image and a range image with and without dust.
  • (a1) of the figure shows an example of a luminance image when there is no dust near the camera.
  • This luminance image is, for example, an infrared image representing the road surface in front of the onboard camera.
  • (b1) in the figure is an image representing the same road surface as (a1), and shows an example of an infrared image when dust is soaring in the vicinity of the camera.
  • (a2) in the figure shows an example of a distance image when there is no dust near the camera.
  • This distance image is, for example, an image representing the road surface in front of the vehicle-mounted camera. Each pixel shows a distance value rather than luminance.
  • (b2) in the figure is a distance image representing the same road surface as (a2), and shows an example of the distance image when dust is flying near the camera.
  • the first section R1 in which the dust D0 causes invalid pixels depends on the intensity of the irradiation light, the size of the dust, and the like.
  • FIG. 7 is a diagram for explaining the influence of dust on the range image.
  • the light emission pulse in the figure indicates the timing of the pulsed light from the light source in FIG.
  • An exposure pulse e1 and an exposure pulse e2 indicate the timing at which the image sensor exposes the reflected light.
  • the figure schematically shows the timing of the reflected light L0 from the object.
  • the image sensor generates the first pixel signal S0 from the reflected light L0 according to the set of the light emission pulse and the exposure pulse e1. Also, the image sensor generates a second pixel signal S1 from the reflected light L0 according to a set of the light emission pulse and the exposure pulse e2.
  • a pixel distance value can be calculated by a ratio operation using the first pixel signal S0 and the second pixel signal S1 by the TOF (Time Of Flight) method.
  • the invalid pixels include a part of the vehicle body on which the object detection device is mounted within the angle of view, and the same problem occurs when strong reflected light is generated from the part of the vehicle body.
  • an object detection device is an object detection device that includes a light emitting unit that emits light, and reflected light that is emitted according to a set of the light emitted by the light emitting unit and the first exposure. a sensor that generates a first pixel signal by exposure and generates a second pixel signal by exposing reflected light according to a combination of the light emission and the second exposure; and the first pixel signal and the second pixel signal. and a distance calculation unit that generates a distance image by ratio calculation using the distance from the object included in the first interval from 0 to a predetermined value from the object detection device.
  • the timing is set so that the reflected wave is not exposed, and the time difference between the light emission and the second exposure is greater than the time difference between the light emission and the first exposure.
  • an object detection method is an object detection method in an object detection device including a light emitting unit that emits light and a sensor that has pixels that expose reflected light, wherein the light emitted by the light emitting unit and the generating a first pixel signal by exposing the reflected light in accordance with a set of the first exposure, generating a second pixel signal by exposing the reflected light in accordance with the set of the light emission and the second exposure;
  • a distance image is generated by a ratio calculation using the pixel signal and the second pixel signal, and the timing of the first exposure is for an object included in a first section in which the distance from the object detection device is 0 to a predetermined value.
  • the time difference between the light emission and the second exposure is greater than the time difference between the light emission and the first exposure.
  • the moving object here includes, for example, vehicles such as automobiles, agricultural machinery, and two-wheeled vehicles, ships, and flying objects such as drones.
  • FIG. 1 is a block diagram showing a configuration example of an object detection device 100 according to an embodiment.
  • the object detection device 100 shown in FIG. 1 is a block diagram showing a configuration example of an object detection device 100 according to an embodiment.
  • the object detection device 100 shown in FIG. 1 is a block diagram showing a configuration example of an object detection device 100 according to an embodiment.
  • the object detection device 100 shown in FIG. 1 is a block diagram showing a configuration example of an object detection device 100 according to an embodiment.
  • the light emitting unit 4 emits pulsed light, which is infrared light, according to the light emission control signal from the signal processing unit 5 .
  • the light emitting unit 4 for example, an element such as a light emitting diode (LED) or a laser diode having a relatively high response speed and capable of high speed blinking is used.
  • the image sensor 3 is a solid-state imaging device having a plurality of pixels arranged in a matrix.
  • An example of the pixel array of the image sensor 3 is shown in FIG.
  • the image sensor 3 has first pixels 31 and second pixels 32 .
  • the first pixels 31 sensitive to visible light are denoted by letters "BW" in the sense of "black and white”.
  • An optical filter that suppresses infrared light may be arranged in the first pixel 31 .
  • the second pixels 32 sensitive to infrared light are labeled with "IR” in the sense of "infrared”.
  • An optical filter that suppresses visible light may be arranged in the second pixel 32 .
  • the image sensor 3 generates pixel signals by exposing according to exposure control signals from the signal processing section 5 . Specifically, the image sensor 3 generates the first pixel signal in each second pixel 32 by exposing the reflected light according to the first exposure pulse in the exposure control signal. Similarly, the image sensor 3 generates a second pixel signal at each second pixel 32 by exposing the reflected light according to the second exposure pulse in the exposure control signal. Also, the image sensor 3 generates a black and white pixel signal at each first pixel 31 by exposing it to visible light.
  • the pixel array of the image sensor 3 is not limited to that shown in FIG.
  • the first pixels 31 and the second pixels 32 may be arranged alternately in the row direction.
  • the first pixels 31 and the second pixels 32 may be alternately arranged in the row direction and the column direction.
  • the number of rows of the first pixels 31 and the number of rows of the second pixels 32 are the same in FIG. 2, they may be different.
  • the image sensor 3 includes R pixels sensitive to red light, G pixels sensitive to green light, B pixels sensitive to blue light, and IR pixels sensitive to infrared light. good too.
  • R pixels, G pixels, B pixels and IR pixels may be arranged in a square.
  • a second image sensor that captures a black-and-white image may be provided. The second image sensor images the same subject as the image sensor 3 .
  • the signal processing unit 5 controls the image sensor 3 to generate a distance image and two types of luminance images. Each pixel of the range image indicates a range value.
  • the two types of luminance images are a black-and-white image based on the black-and-white pixel signals of the first pixels 31 and an infrared image based on the first pixel signals.
  • the signal processor 5 includes a timing controller 7 and a distance calculator 8 .
  • the timing control section 7 generates a light emission control signal for instructing the light emitting section 4 to emit light and an exposure control signal for instructing the image sensor 3 to perform exposure.
  • the distance calculation unit 8 generates a distance image D3 by ratio calculation using the first pixel signal and the second pixel signal from the image sensor 3 .
  • the information processing unit 6 acquires the distance image D3 as a three-dimensional image and the black-and-white image and the infrared image as two-dimensional images from the signal processing unit 5, and detects the captured object in the acquired image. Therefore, the information processing section 6 includes a three-dimensional object detection section 9 , a nearby object detection section 10 , a two-dimensional object detection section 11 , a first integration section 12 , a second integration section 13 and an output section 14 .
  • the nearby object detection unit 10 acquires the first pixel signal from the signal processing unit 5, generates a luminance image from the first pixel signal, and detects an object from the luminance image.
  • the luminance image here is an infrared image. This infrared image is an image of reflected light from an object at a distance shorter than the shortest distance shown by the distance image D3. Therefore, the nearby object detection unit 10 can detect the presence or absence of a near object not included in the distance image D3, and outputs the detection result as the nearby object information D1.
  • the two-dimensional object detection unit 11 acquires black-and-white pixel signals from the signal processing unit 5, generates a black-and-white image from the black-and-white pixel signals, and detects an object and its type in the black-and-white image.
  • the two-dimensional object detection unit 11 uses AI (artificial intelligence) to detect an object from a black-and-white image, and determines whether the detected object is a target of the object detection unit (for example, a person or a car).
  • 2D type information D2 indicating the type of the discriminated object is generated.
  • the two-dimensional object detection unit 11 may acquire a black-and-white image from the second image sensor instead of acquiring black-and-white pixel signals from the signal processing unit 5 .
  • the first integration unit 12 integrates the 3D object information D4 from the 3D object detection unit 9 and the 2D type information D2 from the 2D object detection unit 11, and outputs the integration result as 3D information D5.
  • the 3D information D5 indicates the three-dimensional position, distance, size, and type of the detected object.
  • the first integration unit 12 may integrate the 3D object information D4 and the 2D type information D2 using an OR condition, an AND condition, or an integration combining an OR condition and an AND condition. .
  • the method of such integration may be appropriately determined according to the characteristics of the distance image D3 and the black-and-white image.
  • the second integration unit 13 integrates the nearby object information D1 from the nearby object detection unit 10 and the 2D type information D2 from the two-dimensional object detection unit 11, and outputs the integration result as 2D information D6.
  • the 2D information D6 indicates the two-dimensional position and type of the detected object.
  • This integration may be performed by integrating the nearby object information D1 and the 2D type information D2 under an OR condition, by an AND condition, or by combining an OR condition and an AND condition.
  • Such integration method may be appropriately determined according to the characteristics of the infrared image and the black-and-white image.
  • the output unit 14 outputs the 3D information D5 from the first integration unit 12 and the 2D information from the second integration unit 13 as detection results.
  • the information processing section 6 may be configured by a computer system having one or more processors and one or more memories.
  • This computer system may be one of SoC (System on a Chip), server, and cloud computing.
  • the processor implements the functions of the information processing section 6 by executing programs recorded in the memory.
  • the program may be recorded in memory in advance, recorded in a non-temporary recording medium such as a memory card and provided, or provided through an electric communication line.
  • the program is a program for causing one or more processors to function as the information processing section 6 .
  • the three-dimensional object detection unit 9 detects an object from the distance image D3 and generates 3D object information D4 indicating the position and size of the object. For example, the three-dimensional object detection unit 9 converts the distance image D3 into point cloud data represented by xyz three-dimensional coordinates, and detects an object by searching for a set of points within a certain range in the point cloud data.
  • FIG. 3 is a time chart showing an operation example of the object detection device 100 according to the embodiment.
  • the set of the light emission pulse and the first exposure pulse and the set of the light emission pulse and the second exposure pulse in the same figure indicate separate exposure operations. In the figure, two exposure operations are shown together for convenience of explanation.
  • An example of the timing of reflected light L1 from an object, reflected light L2 from dust, or reflected light L3 due to flare is also shown.
  • Light emission pulse indicates a light emission pulse included in the light emission control signal from the timing control unit 7 to the light emission unit 4. It is assumed that the light emission pulse is of positive logic, and the light emitting section 4 emits pulsed light in the high level section. Assume that the pulse width of the pulsed light is Tp. Note that the light emission pulse may be a negative logic signal.
  • First exposure pulse indicates the first exposure pulse included in the exposure control signal from the timing control unit 7 to the image sensor 3. Assume that the first exposure pulse is of negative logic and exposes the image sensor 3 in the low level section.
  • the timing of the first exposure pulse is set to the timing at which the reflected light from the object included in the first section R1 in which the distance from the object detection device 100 is from 0 to a predetermined value is not exposed. That is, the time A from the start timing of the light emission pulse to the start timing of the first exposure pulse is set to the timing at which strong reflected light from nearby dust is not exposed, as shown in FIG.
  • the first exposure pulse may be a positive logic signal.
  • the first section R1 in which the above distance is from 0 to a predetermined value is, as shown in FIG. Say.
  • the predetermined value and the first interval R1 are not fixed, but depend on the environment such as the intensity of the pulsed light emitted by the light emitting unit 4, the pulse width Tp, and the brightness captured by the image sensor 3.
  • the predetermined value may be, for example, several tens of centimeters to about one meter.
  • the above time A also depends on the environment, so an appropriate value may be set dynamically.
  • “Second exposure pulse” indicates the second exposure pulse included in the exposure control signal from the timing control unit 7 to the image sensor 3. Assume that the second exposure pulse is of negative logic and exposes the image sensor 3 in the low level section.
  • the second exposure pulse may be a positive logic signal.
  • the light emission pulse, the first exposure pulse and the second exposure pulse have the same pulse width Tp.
  • the time from the end timing of the pulsed light to the start timing of the second exposure pulse is the same as the time from the end timing of the pulsed light to the end timing of the first exposure pulse. That is, the time from the start timing of the light emission pulse to the start timing of the second exposure pulse is Tp+A.
  • the distance z is calculated, for example, by the ratio calculation shown in (Equation 1).
  • ⁇ t is the time it takes for the pulsed light from the light emitting unit 4 to be reflected by the object and return to the image sensor 3 .
  • ⁇ t can be expressed as A+B.
  • A is a known time from the start timing of the light emission pulse to the start timing of the first exposure pulse.
  • B is unknown but is calculated from S0 and S1.
  • S0 is the first pixel signal generated by exposure with the first exposure pulse.
  • S1 is a second pixel signal generated by exposure with a second exposure pulse.
  • the nearby object detection unit 10 and the two-dimensional object detection unit 11 operate to overcome this disadvantage. That is, the nearby object detection unit 10 generates a luminance image from the first pixel signal and detects an object from the luminance image.
  • the two-dimensional object detection unit 11 acquires a black-and-white image, which is a two-dimensional image, and detects an object and its type in the two-dimensional image.
  • Equation 1 shows a ratio calculation ignoring background light, but the object detection apparatus 100 generates a background light signal BG indicating the magnitude of the background light by exposing without reflected light. , the first pixel signal S0 and the second pixel signal S1 may be subtracted from each of the background light signal BG and then the ratio calculation of (Equation 1) may be performed.
  • FIG. 4 is a flowchart showing an example of an object detection method according to the embodiment.
  • the object detection device 100 first exposes the image sensor 3 to reflected light in accordance with a set of pulsed light and a first exposure pulse, so that first pixels corresponding to each of the plurality of second pixels 32 are exposed.
  • a pixel signal is output (S101).
  • the object detection device 100 outputs second pixel signals corresponding to each of the plurality of second pixels 32 by exposing the image sensor 3 to reflected light according to a combination of the pulsed light and the second exposure pulse ( S102). Further, the object detection device 100 outputs black and white pixel signals from each of the plurality of first pixels 31 in the image sensor 3 (S103).
  • the exposures in steps S101 and S102 are global exposures (or global shutter function) that expose all pixels simultaneously.
  • the first pixel signal may be an integrated value resulting from multiple exposures by the first exposure pulse.
  • the second pixel signal may also be an integrated value resulting from multiple exposures by the second exposure pulse.
  • the object detection device 100 calculates the distance z using the first pixel signal S0 and the second pixel signal S1 using the above (Equation 1) for each of the plurality of second pixels 32 (S104 to S106). Thereby, the distance calculator 8 generates a distance image D3 having the pixel value of the distance z.
  • the object detection device 100 generates a luminance image from the first pixel signal and detects an object from the luminance image.
  • This luminance image is an infrared image obtained by the first exposure pulse, and an object in the unmeasurable range, that is, an object near the object detection device 100 is detected, and the detection result is generated as nearby object information D1 (S107).
  • the object detection device 100 detects a two-dimensional object and its type from the black-and-white image, and generates the detection result as 2D type information D2 (S108).
  • the object detection device 100 detects a three-dimensional object from the distance image D3 and generates the detection result as 3D object information D4 (S109).
  • the object detection device 100 integrates and outputs the nearby object information D1, the 2D type information D2, and the 3D object information D4 in the first integration unit 12, the second integration unit 13, and the output unit 14 (S110).
  • steps S101 to S103 may be performed in a different order. Further, steps S107 to S109 may also be performed in a different order.
  • the image sensor 3 may be of a dTOF (direct TOF) method using pulsed light or a CW-iTOF (Continuous Wave-iTOF) method also using periodic light (for example, sinusoidal laser light).
  • dTOF direct TOF
  • CW-iTOF Continuous Wave-iTOF
  • the object detection device is an object detection device that exposes the reflected light according to a combination of a light emitting unit that emits light and the first exposure. and a sensor that generates a second pixel signal by exposing reflected light according to a combination of the light emission and the second exposure, and the first pixel signal and the second pixel signal. and a distance calculation unit for generating a distance image by calculating the ratio, wherein the timing of the first exposure is determined based on the reflected waves from the object included in the first section in which the distance from the object detection device is from 0 to a predetermined value. The timing is set so that no exposure is performed, and the time difference between the light emission and the second exposure is greater than the time difference between the light emission and the first exposure.
  • a three-dimensional object detection unit that detects an object from the range image may be provided.
  • a two-dimensional object detection unit that acquires a first luminance image including the same imaging target as the distance image, detects an object in the first luminance image, and a detection result of the three-dimensional object detection unit and the two-dimensional
  • a first integration unit that integrates the detection result of the object detection unit may be provided.
  • the first intensity image may be, for example, the black and white image described above.
  • the accuracy of object detection can be improved through integration.
  • a proximity object detection unit that generates a second luminance image from the first pixel signal and detects an object from the second luminance image may be provided.
  • a two-dimensional object detection unit that acquires a first luminance image including the same imaging target as the distance image, detects an object in the first luminance image, and a detection result of the proximity object detection unit and the two-dimensional object.
  • a second integration unit that integrates the detection result of the detection unit may be provided.
  • the accuracy of object detection can be improved through integration.
  • a three-dimensional object detection unit that detects an object from the range image
  • a first integration unit that integrates the detection result of the three-dimensional object detection unit and the detection result of the two-dimensional object detection unit
  • the first integration and an output unit that outputs, as a detection result, an integration result of the second integration unit and an integration result of the second integration unit.
  • the light emitting unit may emit pulsed light as the light emission.
  • the sensor that generates the first pixel signal and the second pixel signal may be an image sensor having a plurality of pixels.
  • an image sensor can be used as the sensor.
  • an image sensor that generates the first luminance image may be provided.
  • the image sensor for monochrome images can be used separately from the sensors for distance images and luminance images.
  • the sensor may generate the first luminance image.
  • one sensor generates the above black and white image, the above distance image, and the above brightness image, so it is suitable for downsizing the object detection device.
  • an object detection method is an object detection method in an object detection device including a light emitting unit that emits light and a sensor that has pixels that expose reflected light, wherein the light emitted by the light emitting unit and the generating a first pixel signal by exposing the reflected light in accordance with a set of the first exposure, generating a second pixel signal by exposing the reflected light in accordance with the set of the light emission and the second exposure;
  • a distance image is generated by a ratio calculation using the pixel signal and the second pixel signal, and the timing of the first exposure is for an object included in a first section in which the distance from the object detection device is 0 to a predetermined value.
  • the time difference between the light emission and the second exposure is greater than the time difference between the light emission and the first exposure.
  • the object detection device 100 has the light emitting unit 4 that emits pulsed light and a plurality of pixels, and exposes the first pixel by exposing the reflected light according to a set of the pulsed light and the first exposure pulse.
  • An image sensor 3 that generates a signal and exposes the reflected light according to a combination of the pulsed light and the second exposure pulse to generate a second pixel signal, and a ratio calculation using the first pixel signal and the second pixel signal.
  • a nearby object detection unit 10 for generating a luminance image from the first pixel signal and detecting an object from the luminance image.
  • the timing is set so that the reflected wave from the object included in the first interval R1 from 0 to a predetermined value from the detection device 100 is not exposed, and the time difference between the pulsed light and the second exposure pulse is set to the pulsed light and the first exposure pulse. Greater than the time difference from the exposure pulse.
  • the start timing of the first exposure pulse may be later than the end timing of the pulsed light.
  • the distance indicated by the distance image D3 may indicate a distance farther than the first section R1, and may not indicate a distance corresponding to the first section R1.
  • the timing control section 7 for generating the first exposure pulse and the second exposure pulse in synchronization with the pulsed light may be provided, and the timing control section 7 does not need to generate the exposure pulse that overlaps with the pulsed light.
  • the time from the end timing of the pulsed light to the start timing of the second exposure pulse may be the same as the time from the end timing of the pulsed light to the end timing of the first exposure pulse.
  • the object detection device 100 acquires a two-dimensional image generated by the image sensor 3, and detects an object in the two-dimensional image by the two-dimensional object detection unit 11 and the detection result of the nearby object detection unit 10.
  • An integration unit (for example, the first combining unit 12) that integrates the detection result of the dimensional object detection unit 11 may be provided.
  • objects can be detected from 2D images without being affected by dust.
  • the two-dimensional object detection unit 11 may generate type information of the detected object.
  • the type of object can be detected from the two-dimensional image without being affected by dust.
  • the object detection device 100 includes a three-dimensional object detection unit 9 that detects an object from a range image and generates three-dimensional information of the object, and an integration unit (for example, the first combining unit 12 and the second combining unit 13 ) may integrate the detection result of the nearby object detection unit 10 , the detection result of the two-dimensional object detection unit 11 , and the detection result of the three-dimensional object detection unit 9 .
  • an integration unit for example, the first combining unit 12 and the second combining unit 13
  • the integration unit may integrate the detection result of the nearby object detection unit 10 , the detection result of the two-dimensional object detection unit 11 , and the detection result of the three-dimensional object detection unit 9 .
  • An object detection method is an object detection method in an object detection device 100 including a light emitting unit 4 that emits pulsed light and an image sensor 3 that has pixels that expose light reflected by the pulsed light.
  • the image sensor 3 generates a first pixel signal for each pixel by exposing the reflected light in accordance with the combination of the pulsed light and the first exposure pulse, and the image sensor 3 generates a first pixel signal by combining the pulsed light and the second exposure pulse.
  • a second pixel signal is generated for each pixel by exposing reflected light according to the set, a distance image is generated by calculating a ratio between the first pixel signal and the second pixel signal, and a luminance image is generated from the first pixel signal.
  • the timing of the first exposure pulse is set to the timing at which the reflected wave from the object included in the first interval from 0 to the predetermined value from the object detection device 100 is not exposed.
  • the time difference between the pulsed light and the second exposure pulse is greater than the time difference between the pulsed light and the first exposure pulse.
  • each component may be configured by dedicated hardware or implemented by executing a software program suitable for each component.
  • Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor.
  • the software that realizes the object detection device and the like of each of the above embodiments is the following program.
  • this program causes the computer to execute the object detection method shown in FIG.
  • the image sensor 3 only needs to have a global shutter function, and may be a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type.
  • CCD Charge Coupled Device
  • CMOS Complementary Metal Oxide Semiconductor
  • the image sensor 3 may be configured with two solid-state imaging devices instead of one solid-state imaging device, or may be configured with three solid-state imaging devices. However, it is necessary that the pixel positions of the first pixel signal, the second pixel signal, and the black-and-white pixel signal can be associated with each other.
  • the object detection device 100 has been described above based on the embodiment, the present disclosure is not limited to this embodiment. As long as it does not deviate from the spirit of the present disclosure, various modifications that a person skilled in the art can think of are applied to the present embodiment, and a form constructed by combining the components of different embodiments may also be one or more of the present disclosure. may be included within the scope of the embodiments.
  • the present disclosure can be used for an object detection device 100 that detects objects in 2D images and 3D images.
  • Image sensor 4 Light emitting unit 5
  • Signal processing unit 6 Information processing unit 7
  • Distance calculation unit 9 Three-dimensional object detection unit 10 Nearby object detection unit 11
  • Two-dimensional object detection unit 12 First integration unit 13
  • Second integration unit 14 Output unit 31 First pixel 32 Second pixel 100 Object detection device

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • General Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Optical Radar Systems And Details Thereof (AREA)
PCT/JP2022/024738 2021-07-16 2022-06-21 物体検知装置および物体検知方法 Ceased WO2023286542A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2022544426A JP7220835B1 (ja) 2021-07-16 2022-06-21 物体検知装置および物体検知方法
CN202280049134.5A CN117616310A (zh) 2021-07-16 2022-06-21 物体检测装置及物体检测方法
EP22841886.9A EP4372419A4 (en) 2021-07-16 2022-06-21 OBJECT DETECTION DEVICE AND OBJECT DETECTION METHOD
US18/408,397 US12529796B2 (en) 2021-07-16 2024-01-09 Object detection device and object detection method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021118091 2021-07-16
JP2021-118091 2021-07-16

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/408,397 Continuation US12529796B2 (en) 2021-07-16 2024-01-09 Object detection device and object detection method

Publications (1)

Publication Number Publication Date
WO2023286542A1 true WO2023286542A1 (ja) 2023-01-19

Family

ID=84919217

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/024738 Ceased WO2023286542A1 (ja) 2021-07-16 2022-06-21 物体検知装置および物体検知方法

Country Status (6)

Country Link
US (1) US12529796B2 (https=)
EP (1) EP4372419A4 (https=)
JP (1) JP7220835B1 (https=)
CN (1) CN117616310A (https=)
TW (1) TW202323856A (https=)
WO (1) WO2023286542A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024204561A1 (ja) * 2023-03-30 2024-10-03 ヌヴォトンテクノロジージャパン株式会社 物体検知装置、物体検知システム及び物体検知方法
WO2026029088A1 (ja) * 2024-08-01 2026-02-05 ヌヴォトンテクノロジージャパン株式会社 距離画像撮像装置及び駆動方法

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12323689B1 (en) * 2024-09-18 2025-06-03 SimpliSafe, Inc. Lighted camera apparatus

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10253760A (ja) * 1997-03-10 1998-09-25 Hamamatsu Photonics Kk 距離測定装置
JPH11325823A (ja) * 1998-05-14 1999-11-26 Omron Corp 距離計測装置およびこれを用いた物体検出装置ならびに受光信号処理回路
JP2007122508A (ja) 2005-10-28 2007-05-17 Secom Co Ltd 侵入検知装置
JP2012146132A (ja) 2011-01-12 2012-08-02 Osaka City Univ ３次元データと２次元データの統合方法及びこれを用いた見守りシステム、監視システム
WO2018110183A1 (ja) * 2016-12-14 2018-06-21 パナソニックＩｐマネジメント株式会社 撮像制御装置、撮像制御方法、プログラムおよび記録媒体
JP2018185179A (ja) * 2017-04-25 2018-11-22 株式会社リコー 測距装置、監視装置、３次元計測装置、移動体、ロボット及び測距方法
WO2021005659A1 (ja) * 2019-07-05 2021-01-14 パナソニックセミコンダクターソリューションズ株式会社 情報処理システム、センサシステム、情報処理方法及びプログラム

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH04169805A (ja) * 1990-11-01 1992-06-17 Matsushita Electric Ind Co Ltd 三次元画像測定装置
US9170095B1 (en) * 2014-04-08 2015-10-27 Sony Corporation Distance detection device and method including dynamically adjusted frame rate
JP7531140B2 (ja) * 2018-12-06 2024-08-09 パナソニックＩｐマネジメント株式会社 距離情報取得装置、距離情報取得方法、およびプログラム
WO2020195755A1 (ja) * 2019-03-26 2020-10-01 パナソニックＩｐマネジメント株式会社 測距撮像システム、測距撮像方法、及びプログラム

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10253760A (ja) * 1997-03-10 1998-09-25 Hamamatsu Photonics Kk 距離測定装置
JPH11325823A (ja) * 1998-05-14 1999-11-26 Omron Corp 距離計測装置およびこれを用いた物体検出装置ならびに受光信号処理回路
JP2007122508A (ja) 2005-10-28 2007-05-17 Secom Co Ltd 侵入検知装置
JP2012146132A (ja) 2011-01-12 2012-08-02 Osaka City Univ ３次元データと２次元データの統合方法及びこれを用いた見守りシステム、監視システム
WO2018110183A1 (ja) * 2016-12-14 2018-06-21 パナソニックＩｐマネジメント株式会社 撮像制御装置、撮像制御方法、プログラムおよび記録媒体
JP2018185179A (ja) * 2017-04-25 2018-11-22 株式会社リコー 測距装置、監視装置、３次元計測装置、移動体、ロボット及び測距方法
WO2021005659A1 (ja) * 2019-07-05 2021-01-14 パナソニックセミコンダクターソリューションズ株式会社 情報処理システム、センサシステム、情報処理方法及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4372419A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024204561A1 (ja) * 2023-03-30 2024-10-03 ヌヴォトンテクノロジージャパン株式会社 物体検知装置、物体検知システム及び物体検知方法
WO2026029088A1 (ja) * 2024-08-01 2026-02-05 ヌヴォトンテクノロジージャパン株式会社 距離画像撮像装置及び駆動方法

Also Published As

Publication number Publication date
US20240142628A1 (en) 2024-05-02
CN117616310A (zh) 2024-02-27
US12529796B2 (en) 2026-01-20
TW202323856A (zh) 2023-06-16
EP4372419A4 (en) 2024-11-13
EP4372419A1 (en) 2024-05-22
JP7220835B1 (ja) 2023-02-10
JPWO2023286542A1 (https=) 2023-01-19

Similar Documents

Publication Publication Date Title
JP7220835B1 (ja) 物体検知装置および物体検知方法
US11638073B2 (en) Ranging device and ranging methhod
US20150193934A1 (en) Motion sensor apparatus having a plurality of light sources
JP7259660B2 (ja) イメージレジストレーション装置、画像生成システム及びイメージレジストレーションプログラム
KR102488663B1 (ko) 깊이 프로세서 및 3차원 이미지 기기
CN110986816B (zh) 一种深度测量系统及其测量方法
US20150062302A1 (en) Measurement device, measurement method, and computer program product
US12422557B2 (en) Object recognition abnormality detection apparatus, object recognition abnormality detection program product, and object recognition abnormality detection method
CN107678041A (zh) 用于检测对象的系统和方法
KR20190000052A (ko) 광 송출장치 및 이를 이용한 ToF(Time of Flight)모듈
US20230003895A1 (en) Method and apparatus for controlling distance measurement apparatus
US20240369712A1 (en) Abnormality determination device, abnormality determination method, and non-transitory computer-readable storage medium storing abnormality determination program
US20240221399A1 (en) Vehicle detection device, vehicle detection method, and non transitory computer-readable medium
CN114599999A (zh) 移动量估计装置、移动量估计方法、移动量估计程序以及移动量估计系统
WO2020195755A1 (ja) 測距撮像システム、測距撮像方法、及びプログラム
JP7746300B2 (ja) ゲーティングカメラ、車両用センシングシステム、車両用灯具
US20220311923A1 (en) System, position detecting device, position detecting method, and program
CN115063464B (zh) 深度值确定装置、方法、深度感测模组和电子设备
US20260086237A1 (en) Hybrid three-dimensional sensing system and method
CN118339475A (zh) 异常判定装置、异常判定方法、以及异常判定程序
KR20250106037A (ko) 차량 제어 장치 및 그 방법
TWI596360B (zh) 攝像設備以及攝像方法
WO2026034368A1 (ja) 測定装置、測定システムの制御プログラム、測定システムの制御方法、および、記録媒体
WO2025112766A1 (zh) 飞时测距相机及其检测方法
CN115695688A (zh) 图像生成装置、图像生成方法以及车辆控制系统

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2022544426

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22841886

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202280049134.5

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 2022841886

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022841886

Country of ref document: EP

Effective date: 20240216