WO2024062874A1 - Dispositif de traitement d'informations, procédé de traitement d'informations et programme - Google Patents

Dispositif de traitement d'informations, procédé de traitement d'informations et programme Download PDF

Info

Publication number
WO2024062874A1
WO2024062874A1 PCT/JP2023/031534 JP2023031534W WO2024062874A1 WO 2024062874 A1 WO2024062874 A1 WO 2024062874A1 JP 2023031534 W JP2023031534 W JP 2023031534W WO 2024062874 A1 WO2024062874 A1 WO 2024062874A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
sensor
light
distance
pattern
Prior art date
Application number
PCT/JP2023/031534
Other languages
English (en)
Japanese (ja)
Inventor
秀一 澁井
正彦 南雲
均 中田
大岳 北見
貴志 猿田
Original Assignee
ソニーセミコンダクタソリューションズ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーセミコンダクタソリューションズ株式会社 filed Critical ソニーセミコンダクタソリューションズ株式会社
Publication of WO2024062874A1 publication Critical patent/WO2024062874A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G01S17/894—3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/495—Counter-measures or counter-counter-measures using electronic or electro-optical means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Definitions

  • the present technology relates to an information processing device, an information processing method, and a program for distance measurement.
  • ToF Time of Flight
  • Such distance measuring sensors are installed in various devices, including AMR (Autonomous Mobile Robot). It is expected that AMR equipped with a ranging sensor will be used in various fields in the future. Since such an AMR emits laser light or the like for distance measurement, there is an increased chance that the AMR receives not only the laser light emitted by itself but also the laser light emitted by others.
  • AMR Autonomous Mobile Robot
  • AMR may not be able to perform proper distance measurement.
  • the present technology was developed in view of these circumstances, and its purpose is to suppress the decrease in distance measurement accuracy due to disturbances.
  • first image data as distance image data obtained based on a light reception signal of a first sensor which is a light reception sensor that performs a light reception operation for distance measurement
  • the second image data which is the image data obtained based on the light reception signal of the second sensor, which is the light reception sensor of the type
  • the external an AI (Artificial Intelligence) image processing unit that performs processing to infer a distance measurement error area that occurs in the first image data due to interference light in the wavelength band received by the first sensor emitted by the object, as a correction target area.
  • FIG. 2 is a diagram for explaining an outline of the operation of AMR.
  • FIG. 2 is a block diagram showing a configuration example of an iToF sensor, an RGB sensor, and a control unit. It is a figure showing the example of composition of the 1st light sensing portion.
  • FIG. 2 is a diagram illustrating a configuration example of pixels included in a pixel array section.
  • FIG. 3 is a diagram for explaining an example in which charges are distributed to two charge storage units. It is a figure showing an example of a 1st pattern.
  • FIG. 11 is a diagram showing an example of a second pattern. It is a figure which shows an example of a 3rd pattern. It is a figure which shows an example of a 4th pattern.
  • FIG. 7 is a diagram for explaining correction of a correction target area.
  • FIG. 11 is a diagram for explaining correction of a correction target area together with FIG. 10;
  • FIG. 12 is a diagram for explaining correction of a correction target area together with FIGS. 10 and 11.
  • FIG. 3 is a flowchart illustrating an example of processing executed by a control unit of AMR.
  • FIG. 2 is a block diagram showing a configuration example of AMR.
  • FIG. 7 is a diagram showing a configuration example of an AMR iToF sensor and an RGB sensor according to a second embodiment.
  • FIG. 11 is a block diagram illustrating a configuration example of an AMR according to a second embodiment.
  • FIG. 1 is a diagram for explaining an overview of the operation of AMR1.
  • the AMR 1 is a robot that moves autonomously, and moves while determining an appropriate route while detecting obstacles and the like based mainly on image data captured in the direction of travel.
  • the AMR 1 includes a first optical system 2, an iToF (indirect time of flight) sensor 3, a second optical system 4, an R (Red) G (Green) B (Blue) sensor 5, a control unit 6, and a drive unit 7. It is composed of
  • the first optical system 2 is provided at the front stage of the iToF sensor 3 so that the iToF sensor 3 receives appropriate incident light. Equipped with etc.
  • the first optical system 2 ensures a good light receiving state of the iToF sensor 3 by appropriately driving an actuator (not shown) based on control by the control unit 6. Note that a part of the first optical system 2 may be provided as a part of the iToF sensor 3 as a microlens or the like.
  • the iToF sensor 3 is a sensor that receives reflected light of a specific wavelength band irradiated onto a subject, and obtains distance information to the subject for each pixel based on its flight time.
  • the iToF sensor 3 generates distance image data based on distance information for each pixel and outputs it to the subsequent control section 6.
  • the iToF sensor 3 may be configured to output a light reception signal, and the subsequent control unit 6 or the like may be configured to generate distance image data based on the light reception signal.
  • the iToF sensor 3 is configured as, for example, a CMOS (Complementary Metal Oxide Semiconductor) type image sensor, and has sensitivity to IR (Infrared) light.
  • CMOS Complementary Metal Oxide Semiconductor
  • IR Infrared
  • the second optical system 4 is provided in front of the RGB sensor 5 so that the RGB sensor 5 receives appropriate incident light, and includes, for example, a cover lens, a zoom lens, a focus lens, and other lenses, as well as an aperture mechanism.
  • the second optical system 4 ensures a good light receiving state of the RGB sensor 5 by appropriately driving an actuator (not shown) based on control by the control unit 6. Note that a part of the second optical system 4 may be provided as a part of the RGB sensor 5 as a microlens or the like.
  • the RGB sensor 5 generates RGB image data as color image data by receiving light from the subject and outputs it to the control section 6 at the subsequent stage.
  • the RGB sensor 5 is configured as, for example, a CMOS type or CCD (Charge Coupled Device) type image sensor, and has sensitivity to light in the visible light band.
  • CMOS complementary metal-oxide-semiconductor
  • CCD Charge Coupled Device
  • the control unit 6 receives distance image data from the iToF sensor 3 and receives RGB image data from the RGB sensor 5. The control unit 6 recognizes the object present in the imaging direction based on the distance image data and determines an appropriate travel route.
  • the control unit 6 issues a drive instruction to the drive unit 7 to move along the travel route.
  • the drive unit 7 includes various drive mechanisms, actuators, etc., and contributes to the movement of the AMR 1.
  • FIG. 2 shows a more detailed configuration of the iToF sensor 3, RGB sensor 5, and control section 6.
  • the iToF sensor 3 includes a light emitting unit 8, a first light receiving unit 9, a first signal processing unit 10, and a light emission control unit 11.
  • the light emitting unit 8 is configured to include an LED (Light Emitting Diode) that emits IR light based on a light emission signal that is a periodic signal of a specific frequency and is applied by the light emission control unit 11.
  • LED Light Emitting Diode
  • the first light receiving section 9 includes a pixel array section 13 in which pixels 12 having light receiving elements that receive reflected light emitted from the light emitting section 8 and reflected by the subject are two-dimensionally arranged in a matrix. .
  • Each pixel 12 is configured with two charge storage sections 14a and 14b, and responds to a signal (transfer control signals TRTa, TRTb to be described later) based on a light emission signal applied by the light emission control section 11.
  • the charge storage sections 14a and 14b in which charges are stored are switched.
  • the pixel 12 outputs a light reception signal according to the amount of charge accumulated in the charge accumulation section 14a and the charge accumulation section 14b, respectively. That is, the pixel 12 outputs a light reception signal corresponding to the amount of charge accumulated in the charge storage section 14a and a light reception signal corresponding to the amount of charge accumulated in the charge storage section 14b. These two light reception signals are used in a subsequent stage to obtain distance information to the subject.
  • FIG. 3 An example of the configuration of the first light receiving section 9 is shown in FIG. 3.
  • the first light receiving section 9 includes a pixel array section 13, a vertical drive section 15, a column processing section 16, a horizontal drive section 17, and a system control section 18.
  • the pixel array section 13, the vertical drive section 15, the column processing section 16, the horizontal drive section 17, and the system control section 18 are formed on a semiconductor substrate (chip) not shown.
  • pixels 12 are arranged in a matrix, each having a PD (Photo Diode) 19 (not shown in FIG. 3) as a photoelectric conversion element that generates and internally accumulates photoelectric charges corresponding to the amount of incident light. are arranged two-dimensionally.
  • PD Photo Diode
  • pixel drive lines 20 are formed for each row of the pixel array in a matrix form along the arrangement direction of the pixels 12 in the pixel row (horizontal direction in FIG. 3), and vertical signal lines 20 are formed for each column. 21 are formed along the arrangement direction of the pixels 12 in the pixel column (vertical direction in FIG. 3).
  • One end of the pixel drive line 20 is connected to an output end corresponding to each row of the vertical drive section 15.
  • the vertical drive unit 15 is a pixel drive unit that is composed of a shift register, an address decoder, etc., and drives each pixel 12 of the pixel array unit 13, either all pixels at the same time or row by row.
  • the pixel signals output from each pixel 12 in a pixel row selected and scanned by the vertical drive unit 15 are supplied to the column processing unit 16 through each vertical signal line 21.
  • the column processing unit 16 performs a predetermined signal processing on the pixel signals output from each pixel in the selected row through the vertical signal line 21 for each pixel column in the pixel array unit 13, and temporarily holds the pixel signals after signal processing.
  • the column processing unit 16 performs at least noise removal processing, such as CDS (Correlated Double Sampling) processing, as signal processing.
  • This correlated double sampling process by the column processing unit 16 removes fixed pattern noise specific to pixels, such as reset noise and threshold variation of amplification transistors.
  • the column processing section 16 may also be provided with, for example, an A/D (Analog/Digital) conversion function, and output the signal level as a digital signal.
  • the horizontal drive section 17 is composed of a shift register, an address decoder, etc., and sequentially selects unit circuits corresponding to the pixel columns of the column processing section 16. By this selective scanning by the horizontal driving section 17, pixel signals subjected to signal processing in the column processing section 16 are sequentially outputted to the first signal processing section 10 in FIG.
  • the system control unit 18 includes a timing generator that generates various timing signals, and operates the vertical drive unit 15, column processing unit 16, horizontal drive unit 17, etc. based on the various timing signals generated by the timing generator. Performs drive control.
  • a pixel drive line 20 is wired along the row direction for each pixel row, and two vertical signal lines 21 are wired along the column direction for each pixel column. has been done.
  • the pixel drive line 20 transmits a drive signal for driving when reading a signal from the pixel 12.
  • FIG. 3 shows an example in which one pixel drive line 20 is wired for one pixel 12, the number of pixel drive lines 20 is not limited to one.
  • One end of the pixel drive line 20 is connected to an output end corresponding to each row of the vertical drive section 15.
  • FIG. 4 shows an example of the configuration of each pixel 12 included in the pixel array section 13.
  • the pixel 12 includes a PD 19 as a photoelectric conversion element, and is configured so that charges generated by the PD 19 are distributed to taps 22a and 22b. Of the charges generated by the PD 19, the charges distributed to the tap 22a are read out from the vertical signal line 21a and output as the light reception signal S1. Furthermore, the charge distributed to the tap 22b is read out from the vertical signal line 21b and output as a light reception signal S2.
  • Tap 22a is composed of a transfer transistor 23a, a charge storage section 14a (Floating Diffusion), a reset transistor 24, an amplification transistor 25a, and a selection transistor 26a.
  • tap 22b is composed of a transfer transistor 23b, a charge storage section 14b, a reset transistor 24, an amplification transistor 25b, and a selection transistor 26b.
  • the reset transistor 24 may be shared by the charge storage section 14a and the charge storage section 14b, or may be provided in each of the charge storage section 14a and the charge storage section 14b. good.
  • the reset transistor 24 When the reset transistor 24 is provided in each of the charge storage section 14a and the charge storage section 14b, the reset timing can be controlled individually for the charge storage section 14a and the charge storage section 14b, so that fine control can be performed. becomes possible.
  • the charge storage section 14a and the charge storage section 14b are provided with a common reset transistor 24, the reset timing can be made the same for the charge storage section 14a and the charge storage section 14b, and control becomes simple.
  • the circuit configuration can also be simplified.
  • distribution of charges in the pixel 12 will be explained.
  • distribution refers to the fact that charges generated by photoelectric conversion in the PD 19 are transferred to different charge storage sections 14a and 14b by being transferred by either of the transfer transistors 23a and 23b depending on the timing of charge generation. means.
  • the light emitting unit 8 outputs irradiation light that is modulated so that the irradiation is turned on and off once in one cycle Tp of the light emission signal, which is a periodic signal, and the distance to the subject is adjusted.
  • the reflected light is received by the PD 19 after a corresponding delay time Td.
  • the transfer control signal TRTa is supplied by the pixel drive line 20a and controls on/off of the transfer transistor 23a. Further, the transfer control signal TRTb is supplied by the pixel drive line 20b and controls on/off of the transfer transistor 23b. As shown in the figure, the transfer control signal TRTa has the same phase as the irradiation light, while the transfer control signal TRTb has the inverted phase of the transfer control signal TRTa.
  • the selection transistor 26a when the selection transistor 26a is turned on according to the selection signal SELa, the charges accumulated in the charge storage section 14a are read out via the vertical signal line 21a.
  • the first light receiving section 9 outputs a light receiving signal S1 corresponding to the amount of charge.
  • the selection transistor 26b is turned on according to the selection signal SELb, the charge stored in the charge storage section 14b is read out via the vertical signal line 21b, and the light reception signal S2 corresponding to the amount of charge is sent to the first light reception. It is output from section 9.
  • the charges accumulated in the charge accumulation section 14a are discharged when the reset transistor 24 is turned on in accordance with the reset signal RST supplied by the pixel drive line 20c. Similarly, the charges accumulated in the charge accumulation section 14b are discharged when the reset transistor 24 is turned on according to the reset signal RST.
  • the pixel 12 can distribute the charge generated by the reflected light received by the PD 19 to the tap 22a and the tap 22b according to the delay time Td, and output the light reception signal S1 and the light reception signal S2.
  • the delay time Td depends on the time it takes for the light emitted by the light emitting unit 8 to travel to the subject, reflect on the subject, and then fly to the first light receiving unit 9, that is, it depends on the distance to the subject. be. Therefore, the iToF sensor 3 can determine the distance (depth) to the subject according to the delay time Td based on the light reception signal S1 and the light reception signal S2.
  • the first light receiving section 9 performs light reception control using the synchronization signal of the light emission signal. Therefore, the synchronization signal of the light emission signal is applied from the light emission control section 11 to the first light receiving section 9 .
  • the first signal processing section 10 includes a distance image data generation section 27 and a brightness image data generation section 28.
  • the distance image data generation unit 27 uses the light reception signals S1 and S2 output from the first light reception unit 9 to generate distance image data in which distance information representing the distance to the subject is associated with each pixel 12.
  • the distance image data generated by the distance image data generation section 27 is based on distance information obtained by receiving reflected light of the IR light emitted by the light emitting section 8 of the AMR 1.
  • the opposing AMR 1' AMR 1'
  • a distance measurement error may be included due to the IR light emitted by the opposing AMR 1'.
  • the distance image data output from the distance image data generation unit 27 and which may contain distance measurement errors is referred to as "primary distance image data DD1.”
  • the first method is to use the primary distance image data DD1 at a subsequent stage without correcting it. This is suitable when the primary distance image data DD1 does not include a distance measurement error or when the included distance measurement error is small.
  • the second method is to correct the primary distance image data DD1 and use it at a later stage. This is suitable when the distance measurement error included in the primary distance image data DD1 is not small and it is difficult to handle it without correction, but it is possible to correct it.
  • the third method is to obtain distance image data anew using another method without correcting the primary distance image data DD1. This is suitable when the distance measurement error contained in the primary distance image data DD1 is large and cannot be appropriately corrected, or when the area containing the distance measurement error is large.
  • the luminance image data generating unit 28 performs processing to realize the third method of the three methods described above. Specifically, the luminance image data generating unit 28 generates luminance image data BD using the light receiving signals S1 and S2 output from the first light receiving unit 9.
  • the brightness image data BD is used together with the RGB image data CD output from the RGB sensor 5 to generate distance measurement image data using a distance measurement method using stereo images.
  • distance image data generated by a distance measurement method using stereo images instead of the primary distance image data DD1 will be referred to as "alternative distance image data SD.”
  • the primary distance image data DD1 can be obtained by calculating the distribution of the amount of charge accumulated in the charge storage section 14a and the charge storage section 14b shown in FIG. 4 using the light reception signals S1 and S2.
  • the luminance image data BD can be obtained according to the total amount of charge accumulated in both the charge accumulation section 14a and the charge accumulation section 14b shown in FIG. 4. That is, by calculating the total amount of charge accumulated in both the charge accumulation section 14a and the charge accumulation section 14b using the light receiving signals S1 and S2, it is possible to obtain the luminance information of each pixel 12 in the luminance image data BD.
  • the primary distance image data DD1 and the brightness image data BD are information that can be calculated by performing different calculations using the light reception signal S1 and the light reception signal S2.
  • the primary distance image data DD1 is used as is, or after being corrected, it is used as distance information in subsequent processing such as object detection and travel route determination.
  • the brightness image data BD is treated as one image data in the stereo image distance measuring method and is used to generate the alternative distance image data SD.
  • the light emission control section 11 receives an instruction from the first signal processing section 10 and supplies a light emission signal to the light emission section 8, and also supplies a synchronization signal of the light emission signal to the first light receiving section 9.
  • the iToF sensor 3 which includes the above-mentioned parts, outputs primary distance image data DD1 and brightness image data BD to the control unit 6.
  • the RGB sensor 5 in this example includes a second light receiving section 29 and a second signal processing section 30.
  • the RGB sensor 5 is composed of a CMOS sensor, a CCD sensor, etc., as described above.
  • the spatial resolution of the RGB sensor 5 is higher than that of the iToF sensor 3.
  • the second light receiving section 29 has a pixel array section in which each pixel is two-dimensionally arranged with R, G, or B color filters arranged in a Bayer array, etc., and the R, G, or B wavelength band in which each pixel receives light.
  • a signal obtained by photoelectrically converting the visible light is supplied to the second signal processing section 30 as a light reception signal.
  • the second signal processing section 30 includes an RGB image data generation section 31.
  • the RGB image data generation section 31 performs CDS processing, AGC (Automatic Gain Control) processing, etc. on the R signal, G signal, or B signal supplied from the second light receiving section 29, and further performs A/D conversion processing. By doing this, digital data is generated. Further, the RGB image data generation unit 31 generates RGB image data CD consisting of an R signal, a G signal, and a B signal for each pixel by performing a synchronization process on the digital data, and is supplied to the control section 6.
  • a polarizing filter that transmits light in a predetermined polarization direction may be provided on the incident surface of the image sensor of the RGB sensor 5.
  • a polarized image signal is generated based on light polarized in a predetermined polarization direction by the polarizing filter.
  • the polarizing filter has, for example, four polarization directions, in which case polarized image signals in four directions are generated.
  • the generated polarized image signal is supplied to the control section 6.
  • the control unit 6 uses the primary distance image data DD1 and brightness image data BD output from the iToF sensor 3 and the RGB image data CD output from the RGB sensor 5 to generate final distance image data.
  • This distance image data will be referred to as "final distance image data DD2.”
  • the control unit 6 includes an AI (Artificial Intelligence) image processing unit 32, a correction processing unit 33, and an alternative distance image data generation unit 34 in order to generate the final distance image data DD2 (see FIG. 2).
  • AI Artificial Intelligence
  • the AI image processing unit 32 performs various types of processing using an artificial intelligence model (hereinafter referred to as the "AI model") obtained through machine learning. Specifically, the AI image processing unit 32 performs a process of inferring the type of interference mode of the IR light (hereinafter referred to as "interfering light”) irradiated by the above-mentioned opposed AMR 1'.
  • AI model an artificial intelligence model obtained through machine learning. Specifically, the AI image processing unit 32 performs a process of inferring the type of interference mode of the IR light (hereinafter referred to as "interfering light") irradiated by the above-mentioned opposed AMR 1'.
  • the AI image processing unit 32 performs a process of inferring an area including a distance measurement error caused by interference light in the primary distance image data DD1 as a correction target area ArC.
  • the correction processing unit 33 corrects the distance information for the correction target area ArC inferred by the AI image processing unit 32.
  • This correction processing may be realized by performing inference using an AI model different from the AI model used by the AI image processing unit 32, or may be realized by rule-based processing.
  • the alternative distance image data generation unit 34 uses the brightness image data BD output from the brightness image data generation unit 28 of the first signal processing unit 10 and the RGB image data output from the RGB image data generation unit 31 of the second signal processing unit 30. Substitute distance image data SD is generated based on the image data CD.
  • the brightness image data BD is an IR image
  • the RGB image data CD is a color image. Therefore, both image data are of different types.
  • the alternative distance image data generation unit 34 can generate alternative distance image data SD using these different types of image data.
  • the alternative distance image data generation unit 34 first identifies a pixel in the RGB image data CD that corresponds to an arbitrary pixel in the brightness image data BD as a corresponding point. The corresponding points are identified for each pixel in the brightness image data BD.
  • RGB image data After converting RGB image data to monochrome image data, it is conceivable to use a block matching method on the luminance image data BD as an IR image and the monochrome RGB image data.
  • An AI model may be used in the block matching method.
  • the number of pixels in the luminance image data BD is said to be smaller than the number of pixels in the RGB image data CD, so by identifying the corresponding point of each pixel in the luminance image data BD on the RGB image data CD, It is possible to prevent a situation in which there are no suitable pixels to be used.
  • the corresponding point of each pixel of the RGB image data CD may be specified on the luminance image data BD. In this case, there may be cases where there is no appropriate corresponding point on the brightness image data BD, or cases where corresponding points of a plurality of pixels of the RGB image data CD are the same pixel on the brightness image data BD.
  • the above-mentioned AI image processing unit 32 uses the AI model to classify the degree of influence of the interference light emitted from the opposing AMR 1' on the primary distance image data DD1 as a "disturbance pattern PT.”
  • FIG. 6 shows an example of the first pattern PT1 regarding the interference pattern PT.
  • the first pattern PT1 is a pattern that does not cause distance measurement errors due to interference light.
  • the first pattern PT1 is a state in which the opposing AMR 1' is not emitting light or a state in which direct light and reflected light of the interference light from the opposing AMR 1' are not incident on the sensor surface of the iToF sensor 3 due to the angle of the interference light. It is said that
  • FIG. 7 shows an example of the second pattern PT2 regarding the interference pattern PT.
  • the second pattern PT2 is a pattern with a large distance measurement error due to interference light. For example, there is a state in which direct interference light from the opposing AMR 1' is incident over substantially the entire sensor surface of the iToF sensor 3. Further, for example, there is a case where the interference light is light substantially perpendicular to the sensor surface of the iToF sensor 3. That is, when the area including the distance measurement error is wide and the correction target area ArC is wide, the second pattern PT2 is inferred.
  • FIG. 8 shows an example of the third pattern PT3 regarding the interference pattern PT.
  • the third pattern PT3 is a pattern in which there is a distance measurement error due to the interference light, and the influence of the distance measurement error due to the interference light is smaller than the second pattern PT2. For example, this is a state where the opposing AMR 1' is emitting interference light toward the ground or a wall, and the reflected light of the interference light is incident on a part of the sensor surface of the iToF sensor 3. Another case is that the interference light is non-parallel to the sensor surface of the iToF sensor 3.
  • FIG. 9 shows an example of the fourth pattern PT4 regarding the interference pattern PT.
  • the fourth pattern PT4 is a state in which interference light is irradiated from the opposing AMR 1' in the same manner as the third pattern PT3, and there is an obstacle BO that blocks the direct light or a part of the reflected light of the interference light. It is a state of In other words, the oncoming AMR 1' is emitting interference light toward the ground, floor, or wall, and part of the reflected light of the interference light that is not blocked by the obstacle BO is iToF. This is a state in which the light is incident on a part of the sensor surface of the sensor 3.
  • the type of the interference pattern PT can also be expressed as a type depending on the direction of the interference light.
  • the AI image processing unit 32 of the control unit 6 of the AMR 1 performs a process of inferring the correction target area ArC according to the disturbance pattern PT using the AI model.
  • the AI image processing unit 32 determines that the correction target area ArC does not exist in the primary distance image data DD1, and avoids the inference process for the correction target area ArC.
  • the AI image processing unit 32 does not infer the correction target area ArC. Since it can be estimated that the second pattern PT2 includes a distance measurement error over almost the entire sensor surface of the iToF sensor 3, the AI image processing unit 32 determines that the accuracy of the distance information after correction cannot be guaranteed. , avoiding estimation of the correction target area ArC.
  • the alternative distance image data generation unit 34 of the control unit 6 generates alternative distance image data SD.
  • the generated alternative distance image data SD is treated as final distance image data DD2.
  • the AI image processing unit 32 performs a process of inferring the correction target area ArC on the primary distance image data DD1 by providing the primary distance image data DD1 as input data to the AI model.
  • the area Ar1 (the area shown by the dashed line in Figure 8) where the interfering light is reflected on the floor surface is obtained as the output data of the AI model as the correction target area ArC.
  • the interference pattern PT of the primary distance image data DD1 is the fourth pattern PT4 shown in FIG.
  • processing for inferring the correction target area ArC on the primary distance image data DD1 is performed.
  • an area Ar2 (area indicated by a broken line in FIG. 9) on the floor surface where the interfering light is reflected is obtained as the correction target area ArC as output data of the AI model.
  • the region Ar3 (region indicated by the dashed line in FIG. 9) in which the obstacle BO is imaged in the primary distance image data DD1 is not regarded as the correction target region ArC since normal distance information is obtained.
  • correction processing section 33 of the control section 6 corrects the distance information for the correction target area ArC specified on the primary distance image data DD1.
  • an AI model may be used to correct the distance information
  • rule-based processing may also be applied.
  • an arbitrary pixel in the correction target area ArC is selected as the correction target pixel GC (see FIG. 10).
  • the correction processing unit 33 selects the pixel G1 outside the correction target area ArC in the area recognized as the floor surface, acquires its distance information, and selects the pixel G1 located outside the correction target area ArC in the area recognized as the floor surface, and selects the pixel G1 located in the horizontal direction of the pixel G1. Correction is performed by replacing it with distance information (see FIG. 11).
  • the pixels G1 selected outside the correction target area ArC may be selected from the pixels closest to the correction target area ArC (see FIG. 12).
  • Corrected distance image data which is distance image data obtained after distance information is corrected, is treated as final distance image data DD2.
  • the AI image processing unit 32 performs a process of classifying the interference pattern PT regarding the primary distance image data DD1 using the AI model.
  • the AI model used for classifying the disturbance pattern PT is referred to as a first AI model M1.
  • the learning data used for machine learning to generate the first AI model M1 is each distance image data corresponding to the images of the first pattern PT1 to the fourth pattern PT4 as shown in each figure of FIGS. 6 to 9. , is the label information of the first pattern PT1 to the fourth pattern PT4 as correct data.
  • RGB image data CD obtained from the RGB sensor 5 may be used as learning data for generating the first AI model M1. Thereby, it is possible to learn feature amounts regarding the presence or absence of influence by IR light and the magnitude of influence, and it is possible to obtain the first AI model M1 with high inference accuracy.
  • the AI image processing unit 32 also performs a process of identifying the correction target area ArC in the primary distance image data DD1 using an AI model.
  • the AI model used to identify the correction target area ArC is referred to as the second AI model M2.
  • the learning data used for machine learning to generate the second AI model M2 are distance image data belonging to the first pattern PT1 (first normal image data) and distance image data belonging to the third pattern PT3 (first correction required). (image data), and information on the correction target area ArC for distance image data belonging to the third pattern PT3 as correct data.
  • the distance image data belonging to the first pattern PT1 and the distance image data belonging to the third pattern PT3 are distance image data obtained as a result of imaging the same subject at the same angle of view, and are By using distance image data that differs only in the presence or absence of light emission, it is possible to obtain the second AI model M2 that has suitably learned the range affected by the interference light from the opposing AMR 1'.
  • RGB image data CD (second normal image data) obtained from the RGB sensor 5 may be used as learning data for generating the second AI model M2.
  • the AI image processing unit 32 described above uses the primary distance image data DD1 output from the iToF sensor 3 and the RGB image data CD output from the RGB sensor 5 as input data for inference of the correction target area ArC to an AI model. It is something that can be given. That is, the second AI model M2 calculates the primary distance by taking into account both the characteristics of the object obtained from the RGB image data CD, from which the influence of IR light as interference light has been eliminated, and the characteristics of the distance information in the primary distance image data DD1. A correction target area ArC in the image data DD1 is inferred.
  • the second AI model M2 uses the distance image data belonging to the first pattern PT1 as learning data
  • the second AI model M2 uses the distance image data of the third pattern PT3 that includes a distance measurement error due to the influence of interference light. It is assumed that the difference between the distance image data and distance image data from which distance measurement errors due to the influence of interfering light have been removed, that is, the distance image data of the first pattern PT1 as corrected distance image data, has been learned.
  • the second AI model M2 can be used not only to specify the correction target area ArC but also to correct the correction target area ArC.
  • control unit 6 acquires RGB image data CD acquired from the RGB sensor 5 in step S102.
  • step S103 the control unit 6 performs inference processing using the first AI model M1 to identify the interference pattern PT.
  • the first AI model M1 receives primary distance image data DD1 and RGB image data CD as input data.
  • step S104 and step S105 the control unit 6 performs branch processing according to the identified disturbance pattern PT.
  • step S104 the control unit 6 determines whether the identified interference pattern PT is the first pattern PT1 or not. If it is determined that it is the first pattern PT1, that is, if the first AI model M1 infers that appropriate distance information has been obtained for each pixel and no correction is necessary, the control unit 6 proceeds to step S110 and outputs the primary distance image data DD1 as the final distance image data DD2.
  • step S104 determines in the subsequent step S105 whether or not the identified interference pattern PT is the second pattern PT2. do.
  • step S106 inference processing using the second AI model M2 is performed.
  • the second AI model M2 receives, as input data, the primary distance image data DD1, the RGB image data CD, and information about the disturbance pattern PT which is the inference result of the first AI model M1. Thereby, the correction target area ArC on the primary distance image data DD1 is specified.
  • step S107 the control unit 6 performs a process of correcting the distance information regarding the correction target area ArC.
  • the correction process rule-based processing or inference using an AI model may be performed.
  • step S110 the control unit 6 outputs the corrected distance image data obtained by the correction process in step S107 as final distance image data DD2.
  • step S105 if it is determined that the identified interference pattern PT is the second pattern PT2, that is, the primary distance image data DD1 includes a distance measurement error, and even if corrected, the appropriate distance information is If it is inferred by the first AI model M1 that it cannot be obtained, the control unit 6 proceeds to step S108 and obtains the brightness image data BD output from the iToF sensor 3.
  • step S109 the control unit 6 uses the brightness image data BD and the RGB image data CD to generate alternative distance image data SD using a distance measurement method using stereo images.
  • step S110 the control unit 6 outputs the alternative distance image data SD obtained in step S109 as final distance image data DD2.
  • FIG. 14 is a block diagram showing an example of the internal configuration of the AMR 1. As shown in FIG.
  • the AMR 1 includes the above-mentioned iToF sensor 3, RGB sensor 5, control unit 6, imaging optical system 35, optical system driving unit 36, memory unit 37, and communication unit 38.
  • the iToF sensor 3, the RGB sensor 5, the control section 6, the memory section 37, and the communication section 38 are connected via a bus 39, and are capable of mutual data communication.
  • the control unit 6 includes a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.
  • the CPU performs overall control of the AMR 1 by executing various processes according to programs stored in the ROM or programs loaded into the RAM. Since the control unit 6 has already been described, further explanation will be omitted.
  • the imaging optical system 35 includes the first optical system 2 and the second optical system 4 described above, and it guides light (IR light and visible light) from the subject to the first optical system of the iToF sensor 3. The light is focused on the light receiving section 9 and the light receiving surface of the second light receiving section 29 of the RGB sensor 5.
  • the optical system driving unit 36 collectively refers to the driving units for each optical component of the imaging optical system 35. Specifically, the optical system driving unit 36 has actuators for driving the zoom lens, focus lens, and aperture mechanism, and driving circuits for the actuators.
  • the memory unit 37 is a nonvolatile storage device such as an HDD (Hard Disk Drive) or a flash memory device, and is used as a storage destination (recording destination) for image data output from the iToF sensor 3 and the RGB sensor 5.
  • HDD Hard Disk Drive
  • flash memory device a nonvolatile storage device such as an SSD (Hard Disk Drive) or a flash memory device, and is used as a storage destination (recording destination) for image data output from the iToF sensor 3 and the RGB sensor 5.
  • the communication unit 38 performs various data communications with external devices according to the communication control of the control unit 6.
  • the communication unit 38 of the AMR 1 may be able to perform data communication with the communication unit of the opposing AMR 1'.
  • the implementation of the present technology is not limited to this, and inference processing using an AI model may be performed inside the iToF sensor 3.
  • FIG. 15 The configuration of the iToF sensor 3A and RGB sensor 5A included in such an AMR 1 is shown in FIG. 15. Note that descriptions of configurations similar to those in FIG. 2 will be omitted as appropriate.
  • the iToF sensor 3A includes a control section 6 in addition to a light emitting section 8, a first light receiving section 9, a first signal processing section 10, and a light emission control section 11. That is, the iToF sensor 3A functions as the AI image processing section 32, correction processing section 33, and alternative distance image data generation section 34 included in the control section 6.
  • the RGB sensor 5A has a configuration similar to that shown in FIG. 2, including a second light receiving unit 29 and a second signal processing unit 30, but the RGB image data CD generated in the RGB image data generating unit 31 of the second signal processing unit 30 is supplied to the iToF sensor 3A.
  • FIG. 16 is a block diagram showing an example of the internal configuration of the AMR 1 according to the second embodiment.
  • the AMR 1 includes an iToF sensor 3A, an RGB sensor 5A, an imaging optical system 35, an optical system drive section 36, a memory section 37, a communication section 38, and an external control section 40. There is.
  • the iToF sensor 3A, the RGB sensor 5A, the memory section 37, the communication section 38, and the external control section 40 are connected via a bus 39, and are capable of mutual data communication.
  • Imaging optical system 35 Descriptions of the imaging optical system 35, optical system drive section 36, memory section 37, and communication section 38 are omitted to avoid duplication with FIG. 14.
  • the extra-sensor control unit 40 realizes, for example, some of the functions of the control unit 6 described above, and includes, for example, processing for driving the drive unit of the AMR 1 based on distance image data and communication with the opposing AMR 1'. Perform processing, etc.
  • the iToF sensor 3A includes a light emitting section 8, a first light receiving section 9, a first signal processing section 10, an in-sensor control section 41, an AI image processing section 42, a memory section 43, and a communication I/F 44. and are capable of data communication with each other via a bus 45.
  • the light emitting section 8, the first light receiving section 9, and the first signal processing section 10 will not be repeatedly explained.
  • the sensor internal control unit 41 realizes some of the functions of the control unit 6 and the light emission control unit 11 in FIG. 15, and specifically performs various processes that can be executed without using an AI model.
  • the sensor internal control unit 41 is realized by, for example, a CPU or an MPU (Micro Processor Unit).
  • the in-sensor control unit 41 issues instructions to the light emission control unit 11 to realize light emission control. Further, the in-sensor control unit 41 issues instructions to the first light receiving unit 9 and controls execution of the imaging operation. Similarly, the first signal processing section 10 also controls the execution of processing.
  • the AI image processing unit 42 performs processing using an AI model in the control unit 6, and is realized by, for example, a DSP (Digital Signal Processor).
  • DSP Digital Signal Processor
  • the processing executed by the AI image processing unit 42 includes processing for inferring the disturbance pattern PT and processing for inferring the correction target area ArC, as described above. Further, the correction process may be realized by inferring distance information after correction with respect to the correction target area ArC.
  • Switching of the AI model used by the AI image processing unit 42 is performed, for example, based on the processing of the extra-sensor control unit 40 and the intra-sensor control unit 41. Furthermore, switching between AI models is realized by, for example, storing a plurality of AI models in the memory unit 43 or the memory unit 37.
  • the memory unit 43 can be used as a so-called frame memory in which the primary distance image data DD1 and luminance image data BD obtained by the first signal processing unit 10 are stored.
  • the memory unit 43 can also be used for temporary storage of data used in the process in which the AI image processing unit 42 executes processing using an AI model.
  • the memory unit 43 stores information on programs, AI applications, and AI models used by the AI image processing unit 42.
  • the AI application refers to an application that uses an AI model.
  • an application that generates primary distance image data DD1 using an AI model is an example of an AI application.
  • the communication I/F 44 is an interface that communicates with the external control section 40, the memory section 37, etc. located outside the iToF sensor 3A.
  • the communication I/F 44 performs communication to acquire programs executed by the sensor internal control unit 41 and AI applications and AI models used by the AI image processing unit 42 from the outside, and stores them in the memory unit 43 included in the iToF sensor 3A. Make me remember.
  • the AI model is stored in the memory unit 43 included in the iToF sensor 3A, and can be used by the AI image processing unit 42.
  • Distance image data such as the final distance image data DD2 as a processing result of the processing using the AI model performed by the AI image processing unit 42 is output to the outside of the iToF sensor 3A via the communication I/F 44.
  • the RGB sensor 5A includes a second light receiving section 29, a second signal processing section 30, an internal sensor control section 46, a memory section 47, and a communication I/F 48, each of which exchanges data with each other via a bus 49. It is said that communication is possible.
  • the second light receiving section 29 and the second signal processing section 30 will not be repeatedly explained.
  • the in-sensor control unit 46 instructs the second light receiving unit 29 to control the execution of the imaging operation.
  • the second signal processing section 30 also controls the execution of processing. Thereby, RGB image data CD is obtained in the RGB sensor 5A.
  • the memory section 47 can be used as a so-called frame memory in which the RGB image data CD obtained by the second signal processing section 30 is stored.
  • the communication I/F 48 is an interface that communicates with the external control unit 40, the memory unit 37, etc. located outside the RGB sensor 5A.
  • the communication I/F 44 performs communication to acquire a program and the like to be executed by the in-sensor control unit 41 from the outside, and stores it in the memory unit 47 included in the RGB sensor 5A.
  • the above-mentioned AMR 1 describes that the communication unit 38 may communicate with the opposing AMR 1'. In this communication by the communication unit 38, the AMR 1 may be configured to acquire manufacturer information, model information, etc. of the opposing AMR 1' from the opposing AMR 1'.
  • the AMR 1 can obtain information about the interfering light emitted from the opposing AMR 1', making it possible to improve correction accuracy. Further, since processing can be performed using various AI models in consideration of the size, shape, etc. of the casing of the opposing AMR 1', it is possible to improve inference accuracy.
  • the AMR1 described above uses an AI model to perform a correction process to reduce the distance measurement error included in the correction target area ArC.
  • the AMR 1 may determine the material of the subject using an AI model or the like, and perform correction on the correction target area ArC in consideration of the material of the subject in the correction process.
  • a plurality of images with different materials for floors and walls are provided as learning data.
  • the AI model obtained through this learning is an AI model that infers the correction target area ArC, it can infer an appropriate area as the correction target area ArC depending on the material. It becomes possible to improve inference accuracy.
  • an AI model that performs correction processing by inferring distance information after correction it is possible to appropriately correct the correction target area ArC according to the material of the subject, so that the correction result can be It becomes possible to make the distance information of the final distance image data DD2 highly accurate.
  • the iToF sensor 3 (3A) is the first sensor and the RGB sensor 5 (5A) is the second sensor.
  • the types of sensors included in the AMR 1 are not limited to these.
  • the AMR 1 may be provided with a dToF (direct ToF) sensor instead of the iToF sensor 3 (3A) as the first sensor, or may be provided with a distance measurement sensor provided with phase difference detection pixels, or may be provided with any other distance measurement sensor.
  • dToF direct ToF
  • the AMR 1 may be provided with a sensor that outputs a monochrome image instead of the RGB sensor 5 (5A), may be provided with an IR sensor, or may be provided with a polarization sensor as the second sensor. .
  • the AMR 1 can use a sensor capable of detecting the external shape of the subject as the second sensor. This makes it possible to identify the corresponding point of any pixel in the distance image data output from the distance measurement sensor on the image data output from the second sensor, and replaces it with the stereo image distance measurement method.
  • Distance image data SD can be generated.
  • the AMR1 as an information processing device is equipped with an AI image processing unit 32, 42 that performs processing to infer, as input data to a machine-learned artificial intelligence model (second AI model M2), a ranging error area that occurs in the first image data due to interfering light (e.g., interfering light of IR light) in the light receiving wavelength band of the first sensor emitted by an external object (opposing AMR1'), as a correction target area ArC, by providing the first image data (primary distance image data DD1) as distance image data obtained based on the light receiving signals S1, S2 of the first sensor (iToF sensor 3, 3A), which is a light receiving sensor that performs light receiving operation for distance measurement, and the second image data (RGB image data CD), which is image data obtained based on the light receiving signals (R signal, G signal, B signal) of the second sensor (RGB sensor 5, 5A), which is a light receiving sensor of a different type from the first sensor,
  • a machine-learned artificial intelligence model a
  • the distance information about the correction target area ArC in the first image data (primary distance image data DD1) is corrected, and the corrected distance image data is It may also include a correction processing section 33 that outputs.
  • the correction processing section 33 that outputs.
  • the above-mentioned external object may be the AMR (opposing AMR 1'). If information processing devices such as AMRs that move while generating distance image data about objects located in front of them pass each other, there is a concern that the distance measurement accuracy will decrease due to the laser light emitted for distance measurement. Ru. By specifying the correction target area ArC in such an information processing device, it is possible to suppress a decrease in distance measurement accuracy.
  • the artificial intelligence model (second AI model M2) in AMR1 as an information processing device uses the first normal image data (primary distance image data DD1) in which the correction target area ArC is assumed not to exist.
  • Image data for example, distance image data regarding FIG. 6
  • first correction-required image data for example, distance image data regarding FIG. 8
  • second image data RGB
  • the second normal image data based on the image data CD may be obtained by machine learning using learning data set as a set.
  • the first normal image data and the first image data requiring correction are distance image data
  • the second normal image data is, for example, an RGB image, a monochrome image, a polarized light image, or the like.
  • the distance image data of the second pattern PT2 shown in FIG. 7 and the distance image data of the fourth pattern PT4 shown in FIG. 9 as learning data it becomes possible to recognize features such as the outline of the object to be measured, and improve the accuracy of inference of the correction target area ArC. Can be done. Furthermore, by not using the distance image data of the second pattern PT2 shown in FIG. 7 and the distance image data of the fourth pattern PT4 shown in FIG. 9 as learning data, it is possible to reduce the learning data, and The time required can be shortened. Note that by further using the distance image data of the fourth pattern PT4 shown in FIG. 9 as learning data, the correction accuracy in the case where an obstacle BO exists between the oncoming AMR 1' and the oncoming AMR 1' may be improved. Moreover, by further using the distance image data of the second pattern PT2 shown in FIG. 7 as learning data, it is possible to determine whether or not to perform correction using the AI model.
  • the learning data for the second AI model M2 may further include corrective data for the correction target area ArC. This makes it possible to improve the estimation accuracy of the correction target area ArC.
  • the AI image processing units 32 and 42 in the AMR 1 as an information processing device use an artificial intelligence model (first AI model M1) to prevent interference with interference light. Processing may be performed to infer a disturbance pattern PT which is a type of aspect. Thereby, for example, the correction target area ArC can be inferred using the second AI model M2 learned for each disturbance pattern PT, so that the inference accuracy can be improved.
  • first AI model M1 an artificial intelligence model
  • the correction target area ArC can be inferred using the second AI model M2 learned for each disturbance pattern PT, so that the inference accuracy can be improved.
  • the interference patterns PT may be classified into types according to the direction of interference light. For example, if interference light is irradiated approximately orthogonally to the sensor surface of the first sensor (iToF sensors 3, 3A), the interference light will affect the entire sensor surface and the accuracy of distance image data will be significantly reduced. It will drop. Further, when the reflected light of the interference light is incident on the sensor surface of the first sensor, the accuracy of the distance image data decreases in a part of the area that receives the reflected light. By estimating such a pattern of interference light, it becomes possible to use an appropriate second AI model M2 for each interference pattern PT, and it is possible to improve inference accuracy.
  • the interference pattern PT is such that the direct light of the interference light hits the sensor surface of the first sensor (iToF sensor 3, 3A).
  • a first pattern PT1 in which no direct light is incident a second pattern PT2 in which direct light is incident on the sensor surface, and a third pattern in which direct light is not incident on the sensor surface and there is no obstacle BO that blocks at least a portion of the interfering light.
  • the pattern may include at least PT3 and a fourth pattern PT4 in which direct light is not incident on the sensor surface and an obstacle BO is present.
  • distance information about the correction target area ArC in the first image data is corrected and corrected distance image data is output.
  • the correction processing unit 33 may perform correction on the fourth pattern PT4 except for the area where the obstacle BO is imaged.
  • the distance information obtained about the obstacle BO is considered to be correct.
  • the artificial intelligence model (first AI model M1) that infers the disturbance pattern PT is the one in which the correction target area ArC does not exist.
  • the first normal image data (for example, the distance image data for FIG. 6) is the first image data (primary distance image data DD1) that has been determined to be the same, and the first correction-required image is the first image data having the correction target area ArC.
  • Data for example, the distance image data for FIG. 8
  • second normal image data based on the second image data RGB image data CD
  • correct data for the interference pattern PT are used. It may also be obtained through machine learning.
  • the interference pattern PT By performing supervised learning in which the interference pattern PT is given as correct data, the accuracy of inference of the interference pattern PT for distance image data can be improved, and subsequent correction processing can be performed appropriately. Note that by further using the distance image data of the second pattern PT2 shown in FIG. 7 and the distance image data of the fourth pattern PT4 shown in FIG. 9 as learning data, the interference pattern PT for various situations is inferred. This makes it possible to perform appropriate correction processing and the like.
  • the stereo It may also include an alternative distance image data generation unit 34 that generates alternative distance image data SD by a distance measurement method using images. That is, for example, a distance measurement method using stereo images is adopted as an alternative to distance measurement using the iToF method. Therefore, it is possible to generate alternative distance image data SD that eliminates distance measurement errors caused by interfering light.
  • the first sensor (iToF sensor 3, 3A) in the AMR 1 as an information processing device receives a light reception signal S1 from which distance image data and brightness image data BD are obtained
  • the alternative distance image data generation unit 34 generates brightness image data BD and second image data as two images used in the distance measurement method using stereo images.
  • Image data RGB image data CD
  • the light reception signals S1 and S2 output from the first sensor are used both for generating distance image data using the iToF distance measurement method and for generating distance image data using the distance measurement method using stereo images.
  • the two-lens sensors can handle the generation of distance image data and alternative distance image data SD.
  • This makes it possible to reduce component costs and downsize the AMR1 compared to a total of 3-lens configuration, which has a single-lens sensor that generates distance image data and two-lens sensors that obtain stereo images. It becomes possible.
  • the correction processing unit 33 in the AMR 1 serving as the information processing apparatus may perform correction using information on peripheral pixels of the correction target area ArC. Thereby, correction processing can be performed with simple processing.
  • the first sensors (iToF sensors 3, 3A) in the AMR 1 serving as the information processing device may be sensors having sensitivity to IR light.
  • the interfering light for the first sensor is IR light. Therefore, if the second sensor is, for example, the RGB sensor 5 or 5A, second image data (RGB image data CD) that is not affected by interference light, which is IR light, is output, so that the correction target area ArC can be inferred. The accuracy and accuracy of correction processing can be improved.
  • the information processing method executed by the AMR1 as an information processing device is to process distance image data obtained based on the light reception signals S1 and S2 of the first sensor (iToF sensor 3, 3A), which is a light reception sensor that performs a light reception operation for distance measurement.
  • the first image data primary distance image data DD1
  • the light reception signals (R signal, G signal, B signal) of the second sensor RGB sensor 5, 5A) which is a light reception sensor of a different type from the first sensor.
  • the second image data (RGB image data CD), which is image data obtained based on , as input data to the machine-learned artificial intelligence model (second AI model M2)
  • the information is used to infer a distance measurement error area that occurs in the first image data due to interference light in the wavelength band received by the first sensor emitted by an external object (opposing AMR 1') as a correction target area ArC. This is done by the processing device.
  • the program in this technology is based on first image data (primary This is image data obtained based on the distance image data DD1) and the light reception signals (R signal, G signal, B signal) of the second sensor (RGB sensor 5, 5A) which is a light reception sensor of a different type from the first sensor.
  • first image data primary This is image data obtained based on the distance image data DD1
  • the light reception signals R signal, G signal, B signal
  • the second sensor RGB sensor 5, 5A
  • the arithmetic processing unit executes a function of inferring a distance measurement error area that occurs in the first image data due to interference light in the wavelength band received by the first sensor emitted by the opposing AMR 1') as a correction target area ArC.
  • the AMR 1 as the information processing device described above, it is possible to suppress a decrease in distance measurement accuracy due to disturbance.
  • HDD Hard Disk Drive
  • ROM Read Only Memory
  • MO Magnetic Optical
  • DVDs Digital Versatile Discs
  • Blu-ray Discs registered trademark
  • magnetic disks and semiconductor memory.
  • a removable recording medium can be provided as so-called package software.
  • a program can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.
  • LAN Local Area Network
  • the present technology can also adopt the following configuration.
  • An information processing device comprising an AI (Artificial Intelligence) image processing unit that performs a process of inferring a distance measurement error area that occurs in the first image data due to interference light in a received light wavelength band as a correction target area.
  • AI Artificial Intelligence
  • the information processing device further comprising a correction processing unit that corrects distance information regarding the correction target area in the first image data and outputs corrected distance image data.
  • the external object is an AMR (Autonomous Mobile Robot).
  • the artificial intelligence model includes first normal image data that is the first image data in which the correction target area does not exist, and first correction-required image data that is the first image data that has the correction target area. and second normal image data based on the second image data.
  • Information processing device (5) The information processing device according to (4) above, wherein the learning data further includes correct data regarding the correction target area.
  • the information processing device performs a process of inferring a disturbance pattern that is a type of disturbance mode of the disturbance light using an artificial intelligence model. .
  • the interference pattern is classified into types according to the direction of the interference light.
  • the interference pattern includes a first pattern in which the direct light of the interference light does not enter the sensor surface of the first sensor, a second pattern in which the direct light does not enter the sensor surface, and a second pattern in which the direct light does not enter the sensor surface.
  • the information processing device comprising a correction processing unit that corrects distance information regarding the correction target area in the first image data and outputs corrected distance image data;
  • the information processing device comprising a correction processing unit that corrects distance information regarding the correction target area in the first image data and outputs corrected distance image data;
  • the information processing device comprising a correction processing unit that corrects distance information regarding the correction target area in the first image data and outputs corrected distance image data;
  • the information processing device comprising a correction processing unit that performs the correction on the fourth pattern except for an area where the obstacle is imaged.
  • the artificial intelligence model that infers the disturbance pattern uses first normal image data that is the first image data in which the correction target area does not exist, and first normal image data that is the first image data that has the correction target area.
  • the above-mentioned image data is obtained by machine learning using learning data that is a set of first correction image data, second normal image data based on the second image data, and correct data for the interference pattern.
  • the information processing device according to any one of (7) to (9) above.
  • (11) an alternative distance image data generation unit that generates alternative distance image data by a distance measurement method using stereo images when the AI image processing unit obtains an inference result that the interference pattern is the second pattern;
  • the information processing device according to any one of (8) to (9) above.
  • the first sensor is a light-receiving sensor that performs a light-receiving operation using an iToF (indirect time of flight) ranging method that outputs a light-receiving signal from which the distance image data and brightness image data are obtained;
  • the correction processing unit performs the correction using information on peripheral pixels of the correction target area.
  • the second image data which is image data obtained based on the light reception signal of A program that causes an arithmetic processing unit to execute a function of inferring a distance measurement error area that occurs in the first image data due to interference light in a received light wavelength band as a correction target area.
  • AMR information processing equipment
  • AMR information processing equipment
  • 3A iToF sensor first sensor
  • 5A RGB sensor second sensor
  • Correction processing section 34
  • Primary distance image data first image data, distance image data
  • BD Brightness image data
  • CD RGB image data second image data
  • SD Alternative distance image data
  • ArC Correction target area
  • PT Obstruction pattern PT1 1st pattern PT2 2nd pattern PT3 3rd pattern PT4 4th pattern BO Obstacle
  • M1 1st AI model artificial intelligence model
  • M2 2nd AI model artificial intelligence model

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Software Systems (AREA)
  • Electromagnetism (AREA)
  • Theoretical Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Optical Radar Systems And Details Thereof (AREA)

Abstract

Un dispositif de traitement d'informations selon la présente invention comprend une unité de traitement d'image d'intelligence artificielle (IA) qui : reçoit une entrée de premières données d'image et de secondes données d'image en tant que données d'entrée pour un modèle d'intelligence artificielle qui a été entraîné par apprentissage automatique, lesdites premières données d'image étant des données d'image de distance qui sont obtenues sur la base d'un signal de réception de lumière d'un premier capteur servant de capteur de réception de lumière qui effectue une opération de réception de lumière pour une mesure de distance, lesdites secondes données d'image étant des données d'image obtenues sur la base d'un signal de réception de lumière d'un second capteur qui est un capteur de réception de lumière différent du premier capteur ; et utilise le modèle d'intelligence artificielle pour effectuer un processus de déduction, en tant que région cible de correction, d'une région d'erreur de mesure de distance qui apparaît dans les premières données d'image en raison d'une lumière d'interférence qui a été émise par un objet extérieur et qui est dans une bande de longueur d'onde de réception de lumière du premier capteur.
PCT/JP2023/031534 2022-09-20 2023-08-30 Dispositif de traitement d'informations, procédé de traitement d'informations et programme WO2024062874A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022148872 2022-09-20
JP2022-148872 2022-09-20

Publications (1)

Publication Number Publication Date
WO2024062874A1 true WO2024062874A1 (fr) 2024-03-28

Family

ID=90454198

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/031534 WO2024062874A1 (fr) 2022-09-20 2023-08-30 Dispositif de traitement d'informations, procédé de traitement d'informations et programme

Country Status (1)

Country Link
WO (1) WO2024062874A1 (fr)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014181949A (ja) * 2013-03-18 2014-09-29 Sanyo Electric Co Ltd 情報取得装置および物体検出装置
US20170262768A1 (en) * 2016-03-13 2017-09-14 Microsoft Technology Licensing, Llc Depth from time-of-flight using machine learning
JP2019015706A (ja) * 2017-07-11 2019-01-31 ソニーセミコンダクタソリューションズ株式会社 撮像装置及びモニタリング装置
WO2020066637A1 (fr) * 2018-09-28 2020-04-02 パナソニックＩｐマネジメント株式会社 Dispositif d'acquisition de profondeur, procédé d'acquisition de profondeur et programme
JP2021012133A (ja) * 2019-07-08 2021-02-04 株式会社リコー 測距装置、情報処理装置、測距方法、車載装置、移動体、測距システム
US20210166124A1 (en) * 2019-12-03 2021-06-03 Sony Semiconductor Solutions Corporation Apparatuses and methods for training a machine learning network for use with a time-of-flight camera
WO2022034856A1 (fr) * 2020-08-13 2022-02-17 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme
WO2022201803A1 (fr) * 2021-03-25 2022-09-29 ソニーセミコンダクタソリューションズ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations, et programme

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014181949A (ja) * 2013-03-18 2014-09-29 Sanyo Electric Co Ltd 情報取得装置および物体検出装置
US20170262768A1 (en) * 2016-03-13 2017-09-14 Microsoft Technology Licensing, Llc Depth from time-of-flight using machine learning
JP2019015706A (ja) * 2017-07-11 2019-01-31 ソニーセミコンダクタソリューションズ株式会社 撮像装置及びモニタリング装置
WO2020066637A1 (fr) * 2018-09-28 2020-04-02 パナソニックＩｐマネジメント株式会社 Dispositif d'acquisition de profondeur, procédé d'acquisition de profondeur et programme
JP2021012133A (ja) * 2019-07-08 2021-02-04 株式会社リコー 測距装置、情報処理装置、測距方法、車載装置、移動体、測距システム
US20210166124A1 (en) * 2019-12-03 2021-06-03 Sony Semiconductor Solutions Corporation Apparatuses and methods for training a machine learning network for use with a time-of-flight camera
WO2022034856A1 (fr) * 2020-08-13 2022-02-17 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme
WO2022201803A1 (fr) * 2021-03-25 2022-09-29 ソニーセミコンダクタソリューションズ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations, et programme

Similar Documents

Publication Publication Date Title
KR102352333B1 (ko) 고체 촬상 장치 및 그 제조 방법, 및 전자 기기
US10191245B2 (en) Image sensor
JP5895355B2 (ja) 撮像装置
US10666875B2 (en) Solid-state imaging device
US20110221931A1 (en) Temperature information output apparatus, imaging apparatus, method of outputting temperature information
WO2016133053A1 (fr) Appareil de mesure d'image télémétrique
US20160353043A1 (en) Image sensor and image apparatus
JP6257396B2 (ja) 焦点検出装置及びその制御方法、プログラム、記憶媒体
JP2017059563A (ja) 撮像素子
KR20160008364A (ko) 이미지 센서 및 이미지 센서를 포함하는 영상 촬영 장치
US20080205872A1 (en) Focus detection device
JP6381274B2 (ja) 撮像装置、その制御方法、および制御プログラム
JP6362511B2 (ja) 撮像装置及びその制御方法
JP6740666B2 (ja) 撮像素子、焦点検出装置および撮像装置
WO2024062874A1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations et programme
JP7146022B2 (ja) 半導体装置、撮像装置、および撮像装置が実行する方法
US11249279B2 (en) Imaging apparatus and control method of imaging apparatus
JP6946045B2 (ja) 焦点検出装置及び撮像システム
US20200092489A1 (en) Optical apparatus, control method, and non-transitory computer-readable storage medium
US10375294B2 (en) Focus detection apparatus and focus detection method
JP2020039113A (ja) 撮像装置、撮像システム、および撮像装置の駆動方法
US10341556B2 (en) Image capturing apparatus and control method therefor
JP4950699B2 (ja) 焦点検出装置、その駆動方法及びカメラシステム
KR102253023B1 (ko) 옵티컬 블랙 영역을 구비하는 이미지 센서 및 이를 이용한 다크 레벨 보상 방법
JP2009053540A (ja) 撮像素子

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23867998

Country of ref document: EP

Kind code of ref document: A1