WO2024062874A1 - Information processing device, information processing method, and program - Google Patents

Abstract

An information processing device according to the present invention comprises an artificial intelligence (AI) image processing unit that: receives input of first image data and second image data as input data for an artificial intelligence model which has been trained with machine learning, said first image data being distance image data which is obtained on the basis of a light reception signal of a first sensor serving as a light-receiving sensor that performs a light reception operation for distance measurement, said second image data being image data obtained on the basis of a light reception signal of a second sensor that is a light-receiving sensor differing in type from the first sensor; and uses the artificial intelligence model to perform a process for inferring, as a correction target region, a distance measurement error region which appears in the first image data due to interference light that has been emitted from an outside object and that is in a light-reception wavelength band of the first sensor.

Description

Information processing device, information processing method, program

The present technology relates to an information processing device, an information processing method, and a program for distance measurement.

Regarding distance measuring devices that measure the distance to a target, there is a ToF (Time of Flight) sensor as shown in Patent Document 1 below, for example.

Special table 2014-524016 publication

Such distance measuring sensors are installed in various devices, including AMR (Autonomous Mobile Robot).
It is expected that AMR equipped with a ranging sensor will be used in various fields in the future. Since such an AMR emits laser light or the like for distance measurement, there is an increased chance that the AMR receives not only the laser light emitted by itself but also the laser light emitted by others.

Therefore, AMR may not be able to perform proper distance measurement.

The present technology was developed in view of these circumstances, and its purpose is to suppress the decrease in distance measurement accuracy due to disturbances.

In the information processing device according to the present technology, first image data as distance image data obtained based on a light reception signal of a first sensor, which is a light reception sensor that performs a light reception operation for distance measurement, is different from the first sensor. By providing the second image data, which is the image data obtained based on the light reception signal of the second sensor, which is the light reception sensor of the type, as input data to the machine-learned artificial intelligence model, using the artificial intelligence model, the external an AI (Artificial Intelligence) image processing unit that performs processing to infer a distance measurement error area that occurs in the first image data due to interference light in the wavelength band received by the first sensor emitted by the object, as a correction target area. ing.
That is, by using a distance measurement sensor and other sensors in combination, it becomes possible to specify a pixel area whose distance measurement results are affected by laser light emitted by an external object as a correction target area.

FIG. 2 is a diagram for explaining an outline of the operation of AMR. FIG. 2 is a block diagram showing a configuration example of an iToF sensor, an RGB sensor, and a control unit. It is a figure showing the example of composition of the 1st light sensing portion. FIG. 2 is a diagram illustrating a configuration example of pixels included in a pixel array section. FIG. 3 is a diagram for explaining an example in which charges are distributed to two charge storage units. It is a figure showing an example of a 1st pattern. FIG. 11 is a diagram showing an example of a second pattern. It is a figure which shows an example of a 3rd pattern. It is a figure which shows an example of a 4th pattern. FIG. 7 is a diagram for explaining correction of a correction target area. FIG. 11 is a diagram for explaining correction of a correction target area together with FIG. 10; FIG. 12 is a diagram for explaining correction of a correction target area together with FIGS. 10 and 11. FIG. 3 is a flowchart illustrating an example of processing executed by a control unit of AMR. FIG. 2 is a block diagram showing a configuration example of AMR. FIG. 7 is a diagram showing a configuration example of an AMR iToF sensor and an RGB sensor according to a second embodiment. FIG. 11 is a block diagram illustrating a configuration example of an AMR according to a second embodiment.

Hereinafter, embodiments will be described in the following order.
<1. Configuration of information processing device>
<2. Configuration of each sensor and control section>
<3. Functional configuration of control section>
<4. Types of interference modes>
<5. Inference of correction target area>
<6. Correction of correction target area>
<7. Learning data>
<8. Processing flow related to correction>
<9. AMR configuration>
<10. Second embodiment>
<11. Modified example>
<12. Summary>
<13. This technology＞

<1. Configuration of information processing device>
An AMR (Autonomous Mobile Robot) will be cited as an example of an information processing device according to an embodiment of the present technology, and will be described with reference to the attached drawings.

FIG. 1 is a diagram for explaining an overview of the operation of AMR1.
The AMR 1 is a robot that moves autonomously, and moves while determining an appropriate route while detecting obstacles and the like based mainly on image data captured in the direction of travel.

The AMR 1 includes a first optical system 2, an iToF (indirect time of flight) sensor 3, a second optical system 4, an R (Red) G (Green) B (Blue) sensor 5, a control unit 6, and a drive unit 7. It is composed of

The first optical system 2 is provided at the front stage of the iToF sensor 3 so that the iToF sensor 3 receives appropriate incident light. Equipped with etc.

The first optical system 2 ensures a good light receiving state of the iToF sensor 3 by appropriately driving an actuator (not shown) based on control by the control unit 6.
Note that a part of the first optical system 2 may be provided as a part of the iToF sensor 3 as a microlens or the like.

The iToF sensor 3 is a sensor that receives reflected light of a specific wavelength band irradiated onto a subject, and obtains distance information to the subject for each pixel based on its flight time. The iToF sensor 3 generates distance image data based on distance information for each pixel and outputs it to the subsequent control section 6. Note that the iToF sensor 3 may be configured to output a light reception signal, and the subsequent control unit 6 or the like may be configured to generate distance image data based on the light reception signal.

The iToF sensor 3 is configured as, for example, a CMOS (Complementary Metal Oxide Semiconductor) type image sensor, and has sensitivity to IR (Infrared) light.

The second optical system 4 is provided in front of the RGB sensor 5 so that the RGB sensor 5 receives appropriate incident light, and includes, for example, a cover lens, a zoom lens, a focus lens, and other lenses, as well as an aperture mechanism.

The second optical system 4 ensures a good light receiving state of the RGB sensor 5 by appropriately driving an actuator (not shown) based on control by the control unit 6.
Note that a part of the second optical system 4 may be provided as a part of the RGB sensor 5 as a microlens or the like.

The RGB sensor 5 generates RGB image data as color image data by receiving light from the subject and outputs it to the control section 6 at the subsequent stage.

The RGB sensor 5 is configured as, for example, a CMOS type or CCD (Charge Coupled Device) type image sensor, and has sensitivity to light in the visible light band.

The control unit 6 receives distance image data from the iToF sensor 3 and receives RGB image data from the RGB sensor 5.
The control unit 6 recognizes the object present in the imaging direction based on the distance image data and determines an appropriate travel route.

The control unit 6 issues a drive instruction to the drive unit 7 to move along the travel route.

The drive unit 7 includes various drive mechanisms, actuators, etc., and contributes to the movement of the AMR 1.

<2. Configuration of each sensor and control section>
Here, a more detailed configuration for realizing the present technology will be described. Specifically, FIG. 2 shows a more detailed configuration of the iToF sensor 3, RGB sensor 5, and control section 6.

The iToF sensor 3 includes a light emitting unit 8, a first light receiving unit 9, a first signal processing unit 10, and a light emission control unit 11.

The light emitting unit 8 is configured to include an LED (Light Emitting Diode) that emits IR light based on a light emission signal that is a periodic signal of a specific frequency and is applied by the light emission control unit 11.

The first light receiving section 9 includes a pixel array section 13 in which pixels 12 having light receiving elements that receive reflected light emitted from the light emitting section 8 and reflected by the subject are two-dimensionally arranged in a matrix. .

Each pixel 12 is configured with two charge storage sections 14a and 14b, and responds to a signal (transfer control signals TRTa, TRTb to be described later) based on a light emission signal applied by the light emission control section 11. The charge storage sections 14a and 14b in which charges are stored are switched.

The pixel 12 outputs a light reception signal according to the amount of charge accumulated in the charge accumulation section 14a and the charge accumulation section 14b, respectively. That is, the pixel 12 outputs a light reception signal corresponding to the amount of charge accumulated in the charge storage section 14a and a light reception signal corresponding to the amount of charge accumulated in the charge storage section 14b. These two light reception signals are used in a subsequent stage to obtain distance information to the subject.

An example of the configuration of the first light receiving section 9 is shown in FIG. 3.

The first light receiving section 9 includes a pixel array section 13, a vertical drive section 15, a column processing section 16, a horizontal drive section 17, and a system control section 18. The pixel array section 13, the vertical drive section 15, the column processing section 16, the horizontal drive section 17, and the system control section 18 are formed on a semiconductor substrate (chip) not shown.

In the pixel array section 13, pixels 12 are arranged in a matrix, each having a PD (Photo Diode) 19 (not shown in FIG. 3) as a photoelectric conversion element that generates and internally accumulates photoelectric charges corresponding to the amount of incident light. are arranged two-dimensionally.

Further, in the pixel array section 13, pixel drive lines 20 are formed for each row of the pixel array in a matrix form along the arrangement direction of the pixels 12 in the pixel row (horizontal direction in FIG. 3), and vertical signal lines 20 are formed for each column. 21 are formed along the arrangement direction of the pixels 12 in the pixel column (vertical direction in FIG. 3). One end of the pixel drive line 20 is connected to an output end corresponding to each row of the vertical drive section 15.

The vertical drive unit 15 is a pixel drive unit that is composed of a shift register, an address decoder, etc., and drives each pixel 12 of the pixel array unit 13, either all pixels at the same time or row by row. The pixel signals output from each pixel 12 in a pixel row selected and scanned by the vertical drive unit 15 are supplied to the column processing unit 16 through each vertical signal line 21. The column processing unit 16 performs a predetermined signal processing on the pixel signals output from each pixel in the selected row through the vertical signal line 21 for each pixel column in the pixel array unit 13, and temporarily holds the pixel signals after signal processing.

Specifically, the column processing unit 16 performs at least noise removal processing, such as CDS (Correlated Double Sampling) processing, as signal processing. This correlated double sampling process by the column processing unit 16 removes fixed pattern noise specific to pixels, such as reset noise and threshold variation of amplification transistors. In addition to noise removal processing, the column processing section 16 may also be provided with, for example, an A/D (Analog/Digital) conversion function, and output the signal level as a digital signal.

The horizontal drive section 17 is composed of a shift register, an address decoder, etc., and sequentially selects unit circuits corresponding to the pixel columns of the column processing section 16. By this selective scanning by the horizontal driving section 17, pixel signals subjected to signal processing in the column processing section 16 are sequentially outputted to the first signal processing section 10 in FIG.

The system control unit 18 includes a timing generator that generates various timing signals, and operates the vertical drive unit 15, column processing unit 16, horizontal drive unit 17, etc. based on the various timing signals generated by the timing generator. Performs drive control.

In the pixel array section 13, for a matrix-like pixel arrangement, a pixel drive line 20 is wired along the row direction for each pixel row, and two vertical signal lines 21 are wired along the column direction for each pixel column. has been done. For example, the pixel drive line 20 transmits a drive signal for driving when reading a signal from the pixel 12. Note that although FIG. 3 shows an example in which one pixel drive line 20 is wired for one pixel 12, the number of pixel drive lines 20 is not limited to one. One end of the pixel drive line 20 is connected to an output end corresponding to each row of the vertical drive section 15.

Next, FIG. 4 shows an example of the configuration of each pixel 12 included in the pixel array section 13.

The pixel 12 includes a PD 19 as a photoelectric conversion element, and is configured so that charges generated by the PD 19 are distributed to taps 22a and 22b. Of the charges generated by the PD 19, the charges distributed to the tap 22a are read out from the vertical signal line 21a and output as the light reception signal S1. Furthermore, the charge distributed to the tap 22b is read out from the vertical signal line 21b and output as a light reception signal S2.

Tap 22a is composed of a transfer transistor 23a, a charge storage section 14a (Floating Diffusion), a reset transistor 24, an amplification transistor 25a, and a selection transistor 26a. Similarly, tap 22b is composed of a transfer transistor 23b, a charge storage section 14b, a reset transistor 24, an amplification transistor 25b, and a selection transistor 26b.

Note that, as shown in FIG. 4, the reset transistor 24 may be shared by the charge storage section 14a and the charge storage section 14b, or may be provided in each of the charge storage section 14a and the charge storage section 14b. good.

When the reset transistor 24 is provided in each of the charge storage section 14a and the charge storage section 14b, the reset timing can be controlled individually for the charge storage section 14a and the charge storage section 14b, so that fine control can be performed. becomes possible. When the charge storage section 14a and the charge storage section 14b are provided with a common reset transistor 24, the reset timing can be made the same for the charge storage section 14a and the charge storage section 14b, and control becomes simple. The circuit configuration can also be simplified.

With reference to FIG. 5, distribution of charges in the pixel 12 will be explained. Here, distribution refers to the fact that charges generated by photoelectric conversion in the PD 19 are transferred to different charge storage sections 14a and 14b by being transferred by either of the transfer transistors 23a and 23b depending on the timing of charge generation. means.

As shown in FIG. 5, the light emitting unit 8 outputs irradiation light that is modulated so that the irradiation is turned on and off once in one cycle Tp of the light emission signal, which is a periodic signal, and the distance to the subject is adjusted. The reflected light is received by the PD 19 after a corresponding delay time Td.

The transfer control signal TRTa is supplied by the pixel drive line 20a and controls on/off of the transfer transistor 23a. Further, the transfer control signal TRTb is supplied by the pixel drive line 20b and controls on/off of the transfer transistor 23b. As shown in the figure, the transfer control signal TRTa has the same phase as the irradiation light, while the transfer control signal TRTb has the inverted phase of the transfer control signal TRTa.

Therefore, charges generated when the PD 19 receives reflected light are transferred to the charge storage section 14a while the transfer transistor 23a is turned on according to the transfer control signal TRTa. Furthermore, while the transfer transistor 23b is on according to the transfer control signal TRTb, the charge is transferred to the charge storage section 14b. As a result, during a predetermined period in which the irradiation light of the irradiation time T is periodically performed, the charges transferred via the transfer transistor 23a are sequentially accumulated in the charge storage section 14a, and are transferred via the transfer transistor 23b. The accumulated charges are sequentially accumulated in the charge accumulation section 14b.

After the charge accumulation period ends, as shown in FIG. 4, when the selection transistor 26a is turned on according to the selection signal SELa, the charges accumulated in the charge storage section 14a are read out via the vertical signal line 21a. The first light receiving section 9 outputs a light receiving signal S1 corresponding to the amount of charge. Similarly, when the selection transistor 26b is turned on according to the selection signal SELb, the charge stored in the charge storage section 14b is read out via the vertical signal line 21b, and the light reception signal S2 corresponding to the amount of charge is sent to the first light reception. It is output from section 9.

The charges accumulated in the charge accumulation section 14a are discharged when the reset transistor 24 is turned on in accordance with the reset signal RST supplied by the pixel drive line 20c. Similarly, the charges accumulated in the charge accumulation section 14b are discharged when the reset transistor 24 is turned on according to the reset signal RST.

In this way, the pixel 12 can distribute the charge generated by the reflected light received by the PD 19 to the tap 22a and the tap 22b according to the delay time Td, and output the light reception signal S1 and the light reception signal S2. The delay time Td depends on the time it takes for the light emitted by the light emitting unit 8 to travel to the subject, reflect on the subject, and then fly to the first light receiving unit 9, that is, it depends on the distance to the subject. be. Therefore, the iToF sensor 3 can determine the distance (depth) to the subject according to the delay time Td based on the light reception signal S1 and the light reception signal S2.

Returning to the explanation of FIG. 2.
As described above, the first light receiving section 9 performs light reception control using the synchronization signal of the light emission signal. Therefore, the synchronization signal of the light emission signal is applied from the light emission control section 11 to the first light receiving section 9 .

The first signal processing section 10 includes a distance image data generation section 27 and a brightness image data generation section 28.

The distance image data generation unit 27 uses the light reception signals S1 and S2 output from the first light reception unit 9 to generate distance image data in which distance information representing the distance to the subject is associated with each pixel 12.

Note that the distance image data generated by the distance image data generation section 27 is based on distance information obtained by receiving reflected light of the IR light emitted by the light emitting section 8 of the AMR 1.
However, if there is another AMR (hereinafter referred to as the opposing AMR 1') approaching and facing itself (AMR 1), there is a possibility that a distance measurement error may be included due to the IR light emitted by the opposing AMR 1'. There is.

The distance image data output from the distance image data generation unit 27 and which may contain distance measurement errors is referred to as "primary distance image data DD1."

There are several possible ways to handle the primary distance image data DD1.
The first method is to use the primary distance image data DD1 at a subsequent stage without correcting it. This is suitable when the primary distance image data DD1 does not include a distance measurement error or when the included distance measurement error is small.

The second method is to correct the primary distance image data DD1 and use it at a later stage. This is suitable when the distance measurement error included in the primary distance image data DD1 is not small and it is difficult to handle it without correction, but it is possible to correct it.

The third method is to obtain distance image data anew using another method without correcting the primary distance image data DD1. This is suitable when the distance measurement error contained in the primary distance image data DD1 is large and cannot be appropriately corrected, or when the area containing the distance measurement error is large.

The luminance image data generating unit 28 performs processing to realize the third method of the three methods described above. Specifically, the luminance image data generating unit 28 generates luminance image data BD using the light receiving signals S1 and S2 output from the first light receiving unit 9.

The brightness image data BD is used together with the RGB image data CD output from the RGB sensor 5 to generate distance measurement image data using a distance measurement method using stereo images.
Here, distance image data generated by a distance measurement method using stereo images instead of the primary distance image data DD1 will be referred to as "alternative distance image data SD."

Generation of the brightness image data BD will be explained.
First, the primary distance image data DD1 can be obtained by calculating the distribution of the amount of charge accumulated in the charge storage section 14a and the charge storage section 14b shown in FIG. 4 using the light reception signals S1 and S2.

In contrast, the luminance image data BD can be obtained according to the total amount of charge accumulated in both the charge accumulation section 14a and the charge accumulation section 14b shown in FIG. 4. That is, by calculating the total amount of charge accumulated in both the charge accumulation section 14a and the charge accumulation section 14b using the light receiving signals S1 and S2, it is possible to obtain the luminance information of each pixel 12 in the luminance image data BD.

As understood from this explanation, the primary distance image data DD1 and the brightness image data BD are information that can be calculated by performing different calculations using the light reception signal S1 and the light reception signal S2.

As described above, the primary distance image data DD1 is used as is, or after being corrected, it is used as distance information in subsequent processing such as object detection and travel route determination.
The brightness image data BD is treated as one image data in the stereo image distance measuring method and is used to generate the alternative distance image data SD.

The generation of the alternative distance image data SD will be described later.

The light emission control section 11 receives an instruction from the first signal processing section 10 and supplies a light emission signal to the light emission section 8, and also supplies a synchronization signal of the light emission signal to the first light receiving section 9.

The iToF sensor 3, which includes the above-mentioned parts, outputs primary distance image data DD1 and brightness image data BD to the control unit 6.

Next, the RGB sensor 5 will be explained. The RGB sensor 5 in this example includes a second light receiving section 29 and a second signal processing section 30. The RGB sensor 5 is composed of a CMOS sensor, a CCD sensor, etc., as described above. The spatial resolution of the RGB sensor 5 is higher than that of the iToF sensor 3.

The second light receiving section 29 has a pixel array section in which each pixel is two-dimensionally arranged with R, G, or B color filters arranged in a Bayer array, etc., and the R, G, or B wavelength band in which each pixel receives light. A signal obtained by photoelectrically converting the visible light is supplied to the second signal processing section 30 as a light reception signal.

The second signal processing section 30 includes an RGB image data generation section 31.
The RGB image data generation section 31 performs CDS processing, AGC (Automatic Gain Control) processing, etc. on the R signal, G signal, or B signal supplied from the second light receiving section 29, and further performs A/D conversion processing. By doing this, digital data is generated. Further, the RGB image data generation unit 31 generates RGB image data CD consisting of an R signal, a G signal, and a B signal for each pixel by performing a synchronization process on the digital data, and is supplied to the control section 6.

Note that a polarizing filter that transmits light in a predetermined polarization direction may be provided on the incident surface of the image sensor of the RGB sensor 5. A polarized image signal is generated based on light polarized in a predetermined polarization direction by the polarizing filter. The polarizing filter has, for example, four polarization directions, in which case polarized image signals in four directions are generated. The generated polarized image signal is supplied to the control section 6.

The control unit 6 uses the primary distance image data DD1 and brightness image data BD output from the iToF sensor 3 and the RGB image data CD output from the RGB sensor 5 to generate final distance image data. This distance image data will be referred to as "final distance image data DD2."

<3. Functional configuration of control section>
The control unit 6 includes an AI (Artificial Intelligence) image processing unit 32, a correction processing unit 33, and an alternative distance image data generation unit 34 in order to generate the final distance image data DD2 (see FIG. 2).

The AI image processing unit 32 performs various types of processing using an artificial intelligence model (hereinafter referred to as the "AI model") obtained through machine learning.
Specifically, the AI image processing unit 32 performs a process of inferring the type of interference mode of the IR light (hereinafter referred to as "interfering light") irradiated by the above-mentioned opposed AMR 1'.

Furthermore, the AI image processing unit 32 performs a process of inferring an area including a distance measurement error caused by interference light in the primary distance image data DD1 as a correction target area ArC.

Learning of the AI model used for inference by the AI image processing unit 32 will be described later.

The correction processing unit 33 corrects the distance information for the correction target area ArC inferred by the AI image processing unit 32. This correction processing may be realized by performing inference using an AI model different from the AI model used by the AI image processing unit 32, or may be realized by rule-based processing.

The alternative distance image data generation unit 34 uses the brightness image data BD output from the brightness image data generation unit 28 of the first signal processing unit 10 and the RGB image data output from the RGB image data generation unit 31 of the second signal processing unit 30. Substitute distance image data SD is generated based on the image data CD.

Here, the brightness image data BD is an IR image, and the RGB image data CD is a color image. Therefore, both image data are of different types.
However, the alternative distance image data generation unit 34 can generate alternative distance image data SD using these different types of image data.

Specifically, the alternative distance image data generation unit 34 first identifies a pixel in the RGB image data CD that corresponds to an arbitrary pixel in the brightness image data BD as a corresponding point.
The corresponding points are identified for each pixel in the brightness image data BD.

Furthermore, after converting RGB image data to monochrome image data, it is conceivable to use a block matching method on the luminance image data BD as an IR image and the monochrome RGB image data.

An AI model may be used in the block matching method.

By identifying the corresponding points of each pixel in the luminance image data BD on the RGB image data CD, it is possible to obtain disparity information for each pixel of the luminance image data BD. This makes it possible to obtain distance information for each pixel of the luminance image data BD using the disparity information.

Note that the number of pixels in the luminance image data BD is said to be smaller than the number of pixels in the RGB image data CD, so by identifying the corresponding point of each pixel in the luminance image data BD on the RGB image data CD, It is possible to prevent a situation in which there are no suitable pixels to be used.
However, the corresponding point of each pixel of the RGB image data CD may be specified on the luminance image data BD. In this case, there may be cases where there is no appropriate corresponding point on the brightness image data BD, or cases where corresponding points of a plurality of pixels of the RGB image data CD are the same pixel on the brightness image data BD.

<4. Types of interference modes>
The above-mentioned AI image processing unit 32 uses the AI model to classify the degree of influence of the interference light emitted from the opposing AMR 1' on the primary distance image data DD1 as a "disturbance pattern PT."

FIG. 6 shows an example of the first pattern PT1 regarding the interference pattern PT. The first pattern PT1 is a pattern that does not cause distance measurement errors due to interference light. For example, the first pattern PT1 is a state in which the opposing AMR 1' is not emitting light or a state in which direct light and reflected light of the interference light from the opposing AMR 1' are not incident on the sensor surface of the iToF sensor 3 due to the angle of the interference light. It is said that

FIG. 7 shows an example of the second pattern PT2 regarding the interference pattern PT. The second pattern PT2 is a pattern with a large distance measurement error due to interference light. For example, there is a state in which direct interference light from the opposing AMR 1' is incident over substantially the entire sensor surface of the iToF sensor 3. Further, for example, there is a case where the interference light is light substantially perpendicular to the sensor surface of the iToF sensor 3.
That is, when the area including the distance measurement error is wide and the correction target area ArC is wide, the second pattern PT2 is inferred.

FIG. 8 shows an example of the third pattern PT3 regarding the interference pattern PT. The third pattern PT3 is a pattern in which there is a distance measurement error due to the interference light, and the influence of the distance measurement error due to the interference light is smaller than the second pattern PT2. For example, this is a state where the opposing AMR 1' is emitting interference light toward the ground or a wall, and the reflected light of the interference light is incident on a part of the sensor surface of the iToF sensor 3. Another case is that the interference light is non-parallel to the sensor surface of the iToF sensor 3.

FIG. 9 shows an example of the fourth pattern PT4 regarding the interference pattern PT. The fourth pattern PT4 is a state in which interference light is irradiated from the opposing AMR 1' in the same manner as the third pattern PT3, and there is an obstacle BO that blocks the direct light or a part of the reflected light of the interference light. It is a state of In other words, the oncoming AMR 1' is emitting interference light toward the ground, floor, or wall, and part of the reflected light of the interference light that is not blocked by the obstacle BO is iToF. This is a state in which the light is incident on a part of the sensor surface of the sensor 3.

Note that, as understood from the above description, the type of the interference pattern PT can also be expressed as a type depending on the direction of the interference light.

<5. Inference of correction target area>
As described above, the AI image processing unit 32 of the control unit 6 of the AMR 1 performs a process of inferring the correction target area ArC according to the disturbance pattern PT using the AI model.

Specifically, when it is inferred that the interference pattern PT of the primary distance image data DD1 is the first pattern PT1 shown in FIG. 6, the AI image processing unit 32 does not infer the correction target area ArC. That is, the AI image processing unit 32 determines that the correction target area ArC does not exist in the primary distance image data DD1, and avoids the inference process for the correction target area ArC.

Furthermore, if it is inferred that the interference pattern PT of the primary distance image data DD1 is the second pattern PT2 shown in FIG. 7, the AI image processing unit 32 does not infer the correction target area ArC. Since it can be estimated that the second pattern PT2 includes a distance measurement error over almost the entire sensor surface of the iToF sensor 3, the AI image processing unit 32 determines that the accuracy of the distance information after correction cannot be guaranteed. , avoiding estimation of the correction target area ArC.

In this case, the alternative distance image data generation unit 34 of the control unit 6 generates alternative distance image data SD. The generated alternative distance image data SD is treated as final distance image data DD2.

Furthermore, when it is inferred that the interference pattern PT of the primary distance image data DD1 is the third pattern PT3 shown in Figure 8, the AI image processing unit 32 performs a process of inferring the correction target area ArC on the primary distance image data DD1 by providing the primary distance image data DD1 as input data to the AI model.
In this case, in the example shown in Figure 8, the area Ar1 (the area shown by the dashed line in Figure 8) where the interfering light is reflected on the floor surface is obtained as the output data of the AI model as the correction target area ArC.

Further, when it is inferred that the interference pattern PT of the primary distance image data DD1 is the fourth pattern PT4 shown in FIG. By giving as follows, processing for inferring the correction target area ArC on the primary distance image data DD1 is performed.

At this time, in the example shown in FIG. 9, an area Ar2 (area indicated by a broken line in FIG. 9) on the floor surface where the interfering light is reflected is obtained as the correction target area ArC as output data of the AI model. The region Ar3 (region indicated by the dashed line in FIG. 9) in which the obstacle BO is imaged in the primary distance image data DD1 is not regarded as the correction target region ArC since normal distance information is obtained.

<6. Correction of correction target area>
As described above, the correction processing section 33 of the control section 6 corrects the distance information for the correction target area ArC specified on the primary distance image data DD1.
Although an AI model may be used to correct the distance information, rule-based processing may also be applied.

Here, an example will be described in which correction processing is performed on the third pattern PT3 by rule-based processing.

In the example shown in FIG. 8, an inference result has been obtained that a part of the area Ar1 on the floor surface is the correction target area ArC.

In the correction of the correction target area ArC, first, an arbitrary pixel in the correction target area ArC is selected as the correction target pixel GC (see FIG. 10).

Next, among the pixels included in the area that is recognized as the floor surface and is estimated not to be the correction target area ArC, a pixel that is estimated to be located at the same distance as the correction target pixel GC, for example, pixel G1 whose position in the vertical direction coincides with that of the correction target pixel GC, is selected, and the distance information for the selected pixel G1 is replaced with the distance information for the correction target pixel GC.

In other words, the correction processing unit 33 selects the pixel G1 outside the correction target area ArC in the area recognized as the floor surface, acquires its distance information, and selects the pixel G1 located outside the correction target area ArC in the area recognized as the floor surface, and selects the pixel G1 located in the horizontal direction of the pixel G1. Correction is performed by replacing it with distance information (see FIG. 11).

Note that the pixels G1 selected outside the correction target area ArC may be selected from the pixels closest to the correction target area ArC (see FIG. 12).

Corrected distance image data, which is distance image data obtained after distance information is corrected, is treated as final distance image data DD2.

<7. Learning data>
Learning data used to generate an AI model will be explained.

As described above, the AI image processing unit 32 performs a process of classifying the interference pattern PT regarding the primary distance image data DD1 using the AI model. Here, the AI model used for classifying the disturbance pattern PT is referred to as a first AI model M1.

The learning data used for machine learning to generate the first AI model M1 is each distance image data corresponding to the images of the first pattern PT1 to the fourth pattern PT4 as shown in each figure of FIGS. 6 to 9. , is the label information of the first pattern PT1 to the fourth pattern PT4 as correct data.

By providing such distance image data and label information as correct data as teacher data, it is possible to obtain the first AI model M1 through supervised learning.

Furthermore, RGB image data CD obtained from the RGB sensor 5 may be used as learning data for generating the first AI model M1.
Thereby, it is possible to learn feature amounts regarding the presence or absence of influence by IR light and the magnitude of influence, and it is possible to obtain the first AI model M1 with high inference accuracy.

The AI image processing unit 32 also performs a process of identifying the correction target area ArC in the primary distance image data DD1 using an AI model. Here, the AI model used to identify the correction target area ArC is referred to as the second AI model M2.

The learning data used for machine learning to generate the second AI model M2 are distance image data belonging to the first pattern PT1 (first normal image data) and distance image data belonging to the third pattern PT3 (first correction required). (image data), and information on the correction target area ArC for distance image data belonging to the third pattern PT3 as correct data.

In particular, the distance image data belonging to the first pattern PT1 and the distance image data belonging to the third pattern PT3 are distance image data obtained as a result of imaging the same subject at the same angle of view, and are By using distance image data that differs only in the presence or absence of light emission, it is possible to obtain the second AI model M2 that has suitably learned the range affected by the interference light from the opposing AMR 1'.

Furthermore, RGB image data CD (second normal image data) obtained from the RGB sensor 5 may be used as learning data for generating the second AI model M2.

The AI image processing unit 32 described above uses the primary distance image data DD1 output from the iToF sensor 3 and the RGB image data CD output from the RGB sensor 5 as input data for inference of the correction target area ArC to an AI model. It is something that can be given.
That is, the second AI model M2 calculates the primary distance by taking into account both the characteristics of the object obtained from the RGB image data CD, from which the influence of IR light as interference light has been eliminated, and the characteristics of the distance information in the primary distance image data DD1. A correction target area ArC in the image data DD1 is inferred.

Therefore, by using the RGB image data CD as learning data for generating the second AI model M2, it is possible to improve the inference accuracy of the correction target area ArC.

Note that since the second AI model M2 uses the distance image data belonging to the first pattern PT1 as learning data, the second AI model M2 uses the distance image data of the third pattern PT3 that includes a distance measurement error due to the influence of interference light. It is assumed that the difference between the distance image data and distance image data from which distance measurement errors due to the influence of interfering light have been removed, that is, the distance image data of the first pattern PT1 as corrected distance image data, has been learned.

Therefore, in the AI image processing unit 32, the second AI model M2 can be used not only to specify the correction target area ArC but also to correct the correction target area ArC.

<8. Processing flow related to correction>
An example of processing executed by the control unit 6 of the AMR 1 will be described with reference to FIG. 13.
The control unit 6 acquires the primary distance image data DD1 output from the iToF sensor 3 in step S101.

Subsequently, the control unit 6 acquires RGB image data CD acquired from the RGB sensor 5 in step S102.

In step S103, the control unit 6 performs inference processing using the first AI model M1 to identify the interference pattern PT. The first AI model M1 receives primary distance image data DD1 and RGB image data CD as input data.

In step S104 and step S105, the control unit 6 performs branch processing according to the identified disturbance pattern PT.

Specifically, in step S104, the control unit 6 determines whether the identified interference pattern PT is the first pattern PT1 or not.
If it is determined that it is the first pattern PT1, that is, if the first AI model M1 infers that appropriate distance information has been obtained for each pixel and no correction is necessary, the control unit 6 proceeds to step S110 and outputs the primary distance image data DD1 as the final distance image data DD2.

On the other hand, if it is determined in step S104 that the identified interference pattern PT is not the first pattern PT1, the control unit 6 determines in the subsequent step S105 whether or not the identified interference pattern PT is the second pattern PT2. do.

If it is determined that it is not the second pattern PT2, that is, if the first AI model M1 infers that the primary distance image data DD1 includes a distance measurement error and can be corrected, the control unit 6 Proceeding to step S106, inference processing using the second AI model M2 is performed.

The second AI model M2 receives, as input data, the primary distance image data DD1, the RGB image data CD, and information about the disturbance pattern PT which is the inference result of the first AI model M1.
Thereby, the correction target area ArC on the primary distance image data DD1 is specified.

Subsequently, in step S107, the control unit 6 performs a process of correcting the distance information regarding the correction target area ArC. In the correction process, rule-based processing or inference using an AI model may be performed.

In step S110, the control unit 6 outputs the corrected distance image data obtained by the correction process in step S107 as final distance image data DD2.

In step S105, if it is determined that the identified interference pattern PT is the second pattern PT2, that is, the primary distance image data DD1 includes a distance measurement error, and even if corrected, the appropriate distance information is If it is inferred by the first AI model M1 that it cannot be obtained, the control unit 6 proceeds to step S108 and obtains the brightness image data BD output from the iToF sensor 3.

In step S109, the control unit 6 uses the brightness image data BD and the RGB image data CD to generate alternative distance image data SD using a distance measurement method using stereo images.

In step S110, the control unit 6 outputs the alternative distance image data SD obtained in step S109 as final distance image data DD2.

9. Configuration of AMR
FIG. 14 is a block diagram showing an example of the internal configuration of the AMR 1. As shown in FIG.
The AMR 1 includes the above-mentioned iToF sensor 3, RGB sensor 5, control unit 6, imaging optical system 35, optical system driving unit 36, memory unit 37, and communication unit 38.

The iToF sensor 3, the RGB sensor 5, the control section 6, the memory section 37, and the communication section 38 are connected via a bus 39, and are capable of mutual data communication.

Duplicate explanations regarding the iToF sensor 3 and RGB sensor 5 will be avoided.

The control unit 6 includes a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The CPU performs overall control of the AMR 1 by executing various processes according to programs stored in the ROM or programs loaded into the RAM.
Since the control unit 6 has already been described, further explanation will be omitted.

The imaging optical system 35 includes the first optical system 2 and the second optical system 4 described above, and it guides light (IR light and visible light) from the subject to the first optical system of the iToF sensor 3. The light is focused on the light receiving section 9 and the light receiving surface of the second light receiving section 29 of the RGB sensor 5.

The optical system driving unit 36 collectively refers to the driving units for each optical component of the imaging optical system 35. Specifically, the optical system driving unit 36 has actuators for driving the zoom lens, focus lens, and aperture mechanism, and driving circuits for the actuators.

The memory unit 37 is a nonvolatile storage device such as an HDD (Hard Disk Drive) or a flash memory device, and is used as a storage destination (recording destination) for image data output from the iToF sensor 3 and the RGB sensor 5.

The communication unit 38 performs various data communications with external devices according to the communication control of the control unit 6. For example, the communication unit 38 of the AMR 1 may be able to perform data communication with the communication unit of the opposing AMR 1'.

<10. Second embodiment>
In the AMR 1 described above, an example was given in which inference processing using an AI model is performed outside the iToF sensor 3.

However, the implementation of the present technology is not limited to this, and inference processing using an AI model may be performed inside the iToF sensor 3.

The configuration of the iToF sensor 3A and RGB sensor 5A included in such an AMR 1 is shown in FIG. 15. Note that descriptions of configurations similar to those in FIG. 2 will be omitted as appropriate.

The iToF sensor 3A includes a control section 6 in addition to a light emitting section 8, a first light receiving section 9, a first signal processing section 10, and a light emission control section 11. That is, the iToF sensor 3A functions as the AI image processing section 32, correction processing section 33, and alternative distance image data generation section 34 included in the control section 6.

The RGB sensor 5A has a configuration similar to that shown in FIG. 2, including a second light receiving unit 29 and a second signal processing unit 30, but the RGB image data CD generated in the RGB image data generating unit 31 of the second signal processing unit 30 is supplied to the iToF sensor 3A.

FIG. 16 is a block diagram showing an example of the internal configuration of the AMR 1 according to the second embodiment.
The AMR 1 includes an iToF sensor 3A, an RGB sensor 5A, an imaging optical system 35, an optical system drive section 36, a memory section 37, a communication section 38, and an external control section 40. There is.

The iToF sensor 3A, the RGB sensor 5A, the memory section 37, the communication section 38, and the external control section 40 are connected via a bus 39, and are capable of mutual data communication.

Descriptions of the imaging optical system 35, optical system drive section 36, memory section 37, and communication section 38 are omitted to avoid duplication with FIG. 14.

The extra-sensor control unit 40 realizes, for example, some of the functions of the control unit 6 described above, and includes, for example, processing for driving the drive unit of the AMR 1 based on distance image data and communication with the opposing AMR 1'. Perform processing, etc.

The iToF sensor 3A includes a light emitting section 8, a first light receiving section 9, a first signal processing section 10, an in-sensor control section 41, an AI image processing section 42, a memory section 43, and a communication I/F 44. and are capable of data communication with each other via a bus 45.

The light emitting section 8, the first light receiving section 9, and the first signal processing section 10 will not be repeatedly explained.

The sensor internal control unit 41 realizes some of the functions of the control unit 6 and the light emission control unit 11 in FIG. 15, and specifically performs various processes that can be executed without using an AI model.
The sensor internal control unit 41 is realized by, for example, a CPU or an MPU (Micro Processor Unit).

For example, the in-sensor control unit 41 issues instructions to the light emission control unit 11 to realize light emission control. Further, the in-sensor control unit 41 issues instructions to the first light receiving unit 9 and controls execution of the imaging operation. Similarly, the first signal processing section 10 also controls the execution of processing.

The AI image processing unit 42 performs processing using an AI model in the control unit 6, and is realized by, for example, a DSP (Digital Signal Processor).

In the present embodiment, the processing executed by the AI image processing unit 42 includes processing for inferring the disturbance pattern PT and processing for inferring the correction target area ArC, as described above. Further, the correction process may be realized by inferring distance information after correction with respect to the correction target area ArC.

Switching of the AI model used by the AI image processing unit 42 is performed, for example, based on the processing of the extra-sensor control unit 40 and the intra-sensor control unit 41. Furthermore, switching between AI models is realized by, for example, storing a plurality of AI models in the memory unit 43 or the memory unit 37.

The memory unit 43 can be used as a so-called frame memory in which the primary distance image data DD1 and luminance image data BD obtained by the first signal processing unit 10 are stored. The memory unit 43 can also be used for temporary storage of data used in the process in which the AI image processing unit 42 executes processing using an AI model.

Additionally, the memory unit 43 stores information on programs, AI applications, and AI models used by the AI image processing unit 42. Note that the AI application refers to an application that uses an AI model.

For example, an application that generates primary distance image data DD1 using an AI model is an example of an AI application.

The communication I/F 44 is an interface that communicates with the external control section 40, the memory section 37, etc. located outside the iToF sensor 3A. The communication I/F 44 performs communication to acquire programs executed by the sensor internal control unit 41 and AI applications and AI models used by the AI image processing unit 42 from the outside, and stores them in the memory unit 43 included in the iToF sensor 3A. Make me remember.
As a result, the AI model is stored in the memory unit 43 included in the iToF sensor 3A, and can be used by the AI image processing unit 42.

Distance image data such as the final distance image data DD2 as a processing result of the processing using the AI model performed by the AI image processing unit 42 is output to the outside of the iToF sensor 3A via the communication I/F 44.

The RGB sensor 5A includes a second light receiving section 29, a second signal processing section 30, an internal sensor control section 46, a memory section 47, and a communication I/F 48, each of which exchanges data with each other via a bus 49. It is said that communication is possible.

The second light receiving section 29 and the second signal processing section 30 will not be repeatedly explained.

The in-sensor control unit 46 instructs the second light receiving unit 29 to control the execution of the imaging operation. Similarly, the second signal processing section 30 also controls the execution of processing.
Thereby, RGB image data CD is obtained in the RGB sensor 5A.

The memory section 47 can be used as a so-called frame memory in which the RGB image data CD obtained by the second signal processing section 30 is stored.

The communication I/F 48 is an interface that communicates with the external control unit 40, the memory unit 37, etc. located outside the RGB sensor 5A. The communication I/F 44 performs communication to acquire a program and the like to be executed by the in-sensor control unit 41 from the outside, and stores it in the memory unit 47 included in the RGB sensor 5A.

<11. Modified example>
The above-mentioned AMR 1 describes that the communication unit 38 may communicate with the opposing AMR 1'. In this communication by the communication unit 38, the AMR 1 may be configured to acquire manufacturer information, model information, etc. of the opposing AMR 1' from the opposing AMR 1'.

By acquiring information for specifying the opposing AMR 1', the AMR 1 can obtain information about the interfering light emitted from the opposing AMR 1', making it possible to improve correction accuracy.
Further, since processing can be performed using various AI models in consideration of the size, shape, etc. of the casing of the opposing AMR 1', it is possible to improve inference accuracy.

The AMR1 described above uses an AI model to perform a correction process to reduce the distance measurement error included in the correction target area ArC. At this time, the AMR 1 may determine the material of the subject using an AI model or the like, and perform correction on the correction target area ArC in consideration of the material of the subject in the correction process.

For example, in learning an AI model, a plurality of images with different materials for floors and walls are provided as learning data.
For example, if the AI model obtained through this learning is an AI model that infers the correction target area ArC, it can infer an appropriate area as the correction target area ArC depending on the material. It becomes possible to improve inference accuracy.
In addition, with an AI model that performs correction processing by inferring distance information after correction, it is possible to appropriately correct the correction target area ArC according to the material of the subject, so that the correction result can be It becomes possible to make the distance information of the final distance image data DD2 highly accurate.

In the above-mentioned AMR 1, an example was given in which the iToF sensor 3 (3A) is the first sensor and the RGB sensor 5 (5A) is the second sensor.
In implementing the present technology, the types of sensors included in the AMR 1 are not limited to these.

For example, the AMR 1 may be provided with a dToF (direct ToF) sensor instead of the iToF sensor 3 (3A) as the first sensor, or may be provided with a distance measurement sensor provided with phase difference detection pixels, or may be provided with any other distance measurement sensor.

Further, the AMR 1 may be provided with a sensor that outputs a monochrome image instead of the RGB sensor 5 (5A), may be provided with an IR sensor, or may be provided with a polarization sensor as the second sensor. .
In other words, the AMR 1 can use a sensor capable of detecting the external shape of the subject as the second sensor. This makes it possible to identify the corresponding point of any pixel in the distance image data output from the distance measurement sensor on the image data output from the second sensor, and replaces it with the stereo image distance measurement method. Distance image data SD can be generated.

<12. Summary>
As described in each of the above examples, the AMR1 as an information processing device is equipped with an AI image processing unit 32, 42 that performs processing to infer, as input data to a machine-learned artificial intelligence model (second AI model M2), a ranging error area that occurs in the first image data due to interfering light (e.g., interfering light of IR light) in the light receiving wavelength band of the first sensor emitted by an external object (opposing AMR1'), as a correction target area ArC, by providing the first image data (primary distance image data DD1) as distance image data obtained based on the light receiving signals S1, S2 of the first sensor ( iToF sensor 3, 3A), which is a light receiving sensor that performs light receiving operation for distance measurement, and the second image data (RGB image data CD), which is image data obtained based on the light receiving signals (R signal, G signal, B signal) of the second sensor ( RGB sensor 5, 5A), which is a light receiving sensor of a different type from the first sensor, as input data to the machine-learned artificial intelligence model (second AI model M2).
In other words, by using a distance measurement sensor in combination with other sensors, it is possible to identify a pixel area where the distance measurement results are affected by IR band laser light emitted by an external object as the correction target area ArC.
Therefore, by performing an appropriate correction process, the influence of disturbances can be reduced.
It is also conceivable to generate distance image data by using different frequencies or emission patterns for each individual AMR 1, but this would require intentionally using different frequencies and emission patterns between AMRs using ID authentication, etc. Given that the number of AMRs 1 performing distance measurement in this way will increase, it is highly likely that controlling AMRs from different manufacturers to use different frequencies and emission patterns will become extremely complicated and difficult.
According to this configuration, by identifying the correction target area ArC in which distance measurement errors occur and enabling correction, it is possible to eliminate the above-mentioned complicated processing and to increase the robustness of distance measurement accuracy against disturbances.
By obtaining appropriate distance measurement information about the subject, it is possible to avoid an emergency stop of the AMR 1 or a deadlock in which movement cannot be resumed.

As explained with reference to FIG. 2, etc., in the AMR1 as an information processing device, the distance information about the correction target area ArC in the first image data (primary distance image data DD1) is corrected, and the corrected distance image data is It may also include a correction processing section 33 that outputs.
By correcting the distance information, appropriate distance image data can be obtained. Therefore, it is possible to prevent inappropriate control from being performed based on incorrect distance information.

As described above, in the AMR 1 as the information processing device, the above-mentioned external object may be the AMR (opposing AMR 1').
If information processing devices such as AMRs that move while generating distance image data about objects located in front of them pass each other, there is a concern that the distance measurement accuracy will decrease due to the laser light emitted for distance measurement. Ru. By specifying the correction target area ArC in such an information processing device, it is possible to suppress a decrease in distance measurement accuracy.

As described above, the artificial intelligence model (second AI model M2) in AMR1 as an information processing device uses the first normal image data (primary distance image data DD1) in which the correction target area ArC is assumed not to exist. Image data (for example, distance image data regarding FIG. 6), first correction-required image data (for example, distance image data regarding FIG. 8), which is first image data having a correction target area ArC, and second image data (RGB The second normal image data based on the image data CD) may be obtained by machine learning using learning data set as a set.
The first normal image data and the first image data requiring correction are distance image data, and the second normal image data is, for example, an RGB image, a monochrome image, a polarized light image, or the like.
By using not only the distance image data but also the second normal image data as learning data, it becomes possible to recognize features such as the outline of the object to be measured, and improve the accuracy of inference of the correction target area ArC. Can be done.
Furthermore, by not using the distance image data of the second pattern PT2 shown in FIG. 7 and the distance image data of the fourth pattern PT4 shown in FIG. 9 as learning data, it is possible to reduce the learning data, and The time required can be shortened.
Note that by further using the distance image data of the fourth pattern PT4 shown in FIG. 9 as learning data, the correction accuracy in the case where an obstacle BO exists between the oncoming AMR 1' and the oncoming AMR 1' may be improved.
Moreover, by further using the distance image data of the second pattern PT2 shown in FIG. 7 as learning data, it is possible to determine whether or not to perform correction using the AI model.

As described above, in the AMR1 as an information processing device, the learning data for the second AI model M2 may further include corrective data for the correction target area ArC.
This makes it possible to improve the estimation accuracy of the correction target area ArC.

As explained with reference to each figure from FIG. 6 to FIG. 9, the AI image processing units 32 and 42 in the AMR 1 as an information processing device use an artificial intelligence model (first AI model M1) to prevent interference with interference light. Processing may be performed to infer a disturbance pattern PT which is a type of aspect.
Thereby, for example, the correction target area ArC can be inferred using the second AI model M2 learned for each disturbance pattern PT, so that the inference accuracy can be improved.

As described with reference to each of FIGS. 6 to 9, in the AMR 1 as an information processing device, the interference patterns PT may be classified into types according to the direction of interference light.
For example, if interference light is irradiated approximately orthogonally to the sensor surface of the first sensor ( iToF sensors 3, 3A), the interference light will affect the entire sensor surface and the accuracy of distance image data will be significantly reduced. It will drop. Further, when the reflected light of the interference light is incident on the sensor surface of the first sensor, the accuracy of the distance image data decreases in a part of the area that receives the reflected light.
By estimating such a pattern of interference light, it becomes possible to use an appropriate second AI model M2 for each interference pattern PT, and it is possible to improve inference accuracy.

As explained with reference to each figure from FIG. 6 to FIG. 9, in the AMR1 as an information processing device, the interference pattern PT is such that the direct light of the interference light hits the sensor surface of the first sensor ( iToF sensor 3, 3A). A first pattern PT1 in which no direct light is incident, a second pattern PT2 in which direct light is incident on the sensor surface, and a third pattern in which direct light is not incident on the sensor surface and there is no obstacle BO that blocks at least a portion of the interfering light. The pattern may include at least PT3 and a fourth pattern PT4 in which direct light is not incident on the sensor surface and an obstacle BO is present.
Thereby, it is possible not only to improve the inference accuracy of the correction target area ArC, but also to determine whether or not correction is possible in the first place. Therefore, while correcting distance image data that can be corrected with high precision and avoiding correction processing for distance image data that does not require correction, other distance image data can be newly calculated using alternative methods. It becomes possible to perform processing according to the situation, such as generating image data.

As described with reference to FIG. 9, for AMR1 as an information processing device, distance information about the correction target area ArC in the first image data (primary distance image data DD1) is corrected and corrected distance image data is output. The correction processing unit 33 may perform correction on the fourth pattern PT4 except for the area where the obstacle BO is imaged.
When there is an obstacle BO that blocks at least a portion of the direct light or reflected light of the interference light, the distance information obtained about the obstacle BO is considered to be correct.
By excluding such distance information from the objects of correction, it is possible to prevent unnecessary correction processing from being performed, and it is possible to reduce the processing load.
Furthermore, since the obstacle BO can be correctly recognized, appropriate control such as temporarily stopping in front of the obstacle BO can be performed.

As explained with reference to each figure from FIG. 6 to FIG. 9, in the AMR1 as an information processing device, the artificial intelligence model (first AI model M1) that infers the disturbance pattern PT is the one in which the correction target area ArC does not exist. The first normal image data (for example, the distance image data for FIG. 6) is the first image data (primary distance image data DD1) that has been determined to be the same, and the first correction-required image is the first image data having the correction target area ArC. Data (for example, the distance image data for FIG. 8), second normal image data based on the second image data (RGB image data CD), and correct data for the interference pattern PT are used. It may also be obtained through machine learning.
By performing supervised learning in which the interference pattern PT is given as correct data, the accuracy of inference of the interference pattern PT for distance image data can be improved, and subsequent correction processing can be performed appropriately.
Note that by further using the distance image data of the second pattern PT2 shown in FIG. 7 and the distance image data of the fourth pattern PT4 shown in FIG. 9 as learning data, the interference pattern PT for various situations is inferred. This makes it possible to perform appropriate correction processing and the like.

As explained with reference to FIG. 2 etc., in the AI image processing units 32 and 42 of the AMR1 as an information processing device, when the inference result that the interference pattern PT is the second pattern PT2 is obtained, the stereo It may also include an alternative distance image data generation unit 34 that generates alternative distance image data SD by a distance measurement method using images.
That is, for example, a distance measurement method using stereo images is adopted as an alternative to distance measurement using the iToF method. Therefore, it is possible to generate alternative distance image data SD that eliminates distance measurement errors caused by interfering light.

As described with reference to FIGS. 2, 13, etc., the first sensor ( iToF sensor 3, 3A) in the AMR 1 as an information processing device receives a light reception signal S1 from which distance image data and brightness image data BD are obtained, The alternative distance image data generation unit 34 generates brightness image data BD and second image data as two images used in the distance measurement method using stereo images. Image data (RGB image data CD) may also be used.
Thereby, the light reception signals S1 and S2 output from the first sensor are used both for generating distance image data using the iToF distance measurement method and for generating distance image data using the distance measurement method using stereo images.
Therefore, the two-lens sensors, the first sensor ( iToF sensors 3, 3A) and the second sensor ( RGB sensors 5, 5A), can handle the generation of distance image data and alternative distance image data SD. This makes it possible to reduce component costs and downsize the AMR1 compared to a total of 3-lens configuration, which has a single-lens sensor that generates distance image data and two-lens sensors that obtain stereo images. It becomes possible.

As described with reference to the figures of FIGS. 10 to 12, the correction processing unit 33 in the AMR 1 serving as the information processing apparatus may perform correction using information on peripheral pixels of the correction target area ArC.
Thereby, correction processing can be performed with simple processing.

As described with reference to FIG. 2 and the like, the first sensors ( iToF sensors 3, 3A) in the AMR 1 serving as the information processing device may be sensors having sensitivity to IR light.
In this case, the interfering light for the first sensor is IR light. Therefore, if the second sensor is, for example, the RGB sensor 5 or 5A, second image data (RGB image data CD) that is not affected by interference light, which is IR light, is output, so that the correction target area ArC can be inferred. The accuracy and accuracy of correction processing can be improved.

The information processing method executed by the AMR1 as an information processing device is to process distance image data obtained based on the light reception signals S1 and S2 of the first sensor ( iToF sensor 3, 3A), which is a light reception sensor that performs a light reception operation for distance measurement. as the first image data (primary distance image data DD1), and the light reception signals (R signal, G signal, B signal) of the second sensor ( RGB sensor 5, 5A) which is a light reception sensor of a different type from the first sensor. By giving the second image data (RGB image data CD), which is image data obtained based on , as input data to the machine-learned artificial intelligence model (second AI model M2), The information is used to infer a distance measurement error area that occurs in the first image data due to interference light in the wavelength band received by the first sensor emitted by an external object (opposing AMR 1') as a correction target area ArC. This is done by the processing device.

The program in this technology is based on first image data (primary This is image data obtained based on the distance image data DD1) and the light reception signals (R signal, G signal, B signal) of the second sensor ( RGB sensor 5, 5A) which is a light reception sensor of a different type from the first sensor. By giving the second image data (RGB image data CD) as input data to the machine-learned artificial intelligence model (second AI model M2), the external object ( The arithmetic processing unit executes a function of inferring a distance measurement error area that occurs in the first image data due to interference light in the wavelength band received by the first sensor emitted by the opposing AMR 1') as a correction target area ArC. .
With such a program, in the AMR 1 as the information processing device described above, it is possible to suppress a decrease in distance measurement accuracy due to disturbance.

These programs can be recorded in advance in an HDD (Hard Disk Drive) as a recording medium built into equipment such as a computer device, or in a ROM in a microcomputer having a CPU. The program can also be used on flexible disks, CD-ROMs (Compact Disk Read Only Memory), MO (Magneto Optical) disks, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, and semiconductor memory. can be stored (recorded) temporarily or permanently in a removable recording medium such as a memory card. Such a removable recording medium can be provided as so-called package software.
In addition to installing such a program into a personal computer or the like from a removable recording medium, it can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

Note that the effects described in this specification are merely examples and are not limiting, and other effects may also exist.

Furthermore, the above-mentioned examples may be combined in any manner, and even when various combinations are used, the above-mentioned various operational effects can be obtained.

<13. This technology＞
The present technology can also adopt the following configuration.
(1)
First image data as distance image data obtained based on a light reception signal of a first sensor that is a light reception sensor that performs a light reception operation for distance measurement, and a second sensor that is a light reception sensor of a different type from the first sensor. By providing the second image data, which is image data obtained based on the light reception signal of An information processing device comprising an AI (Artificial Intelligence) image processing unit that performs a process of inferring a distance measurement error area that occurs in the first image data due to interference light in a received light wavelength band as a correction target area.
(2)
The information processing device according to (1) above, further comprising a correction processing unit that corrects distance information regarding the correction target area in the first image data and outputs corrected distance image data.
(3)
The information processing device according to any one of (1) to (2) above, wherein the external object is an AMR (Autonomous Mobile Robot).
(4)
The artificial intelligence model includes first normal image data that is the first image data in which the correction target area does not exist, and first correction-required image data that is the first image data that has the correction target area. and second normal image data based on the second image data. Information processing device.
(5)
The information processing device according to (4) above, wherein the learning data further includes correct data regarding the correction target area.
(6)
The information processing device according to any one of (1) to (5) above, wherein the AI image processing unit performs a process of inferring a disturbance pattern that is a type of disturbance mode of the disturbance light using an artificial intelligence model. .
(7)
The information processing device according to (6) above, wherein the interference pattern is classified into types according to the direction of the interference light.
(8)
The interference pattern includes a first pattern in which the direct light of the interference light does not enter the sensor surface of the first sensor, a second pattern in which the direct light does not enter the sensor surface, and a second pattern in which the direct light does not enter the sensor surface. at least a third pattern in which the direct light is not incident on the sensor surface and there is no obstacle that blocks at least a part of the interfering light; and a fourth pattern in which the direct light is not incident on the sensor surface and the obstacle is present. The information processing device according to (7) above.
(9)
comprising a correction processing unit that corrects distance information regarding the correction target area in the first image data and outputs corrected distance image data;
The information processing device according to (8), wherein the correction processing unit performs the correction on the fourth pattern except for an area where the obstacle is imaged.
(10)
The artificial intelligence model that infers the disturbance pattern uses first normal image data that is the first image data in which the correction target area does not exist, and first normal image data that is the first image data that has the correction target area. The above-mentioned image data is obtained by machine learning using learning data that is a set of first correction image data, second normal image data based on the second image data, and correct data for the interference pattern. The information processing device according to any one of (7) to (9) above.
(11)
an alternative distance image data generation unit that generates alternative distance image data by a distance measurement method using stereo images when the AI image processing unit obtains an inference result that the interference pattern is the second pattern; The information processing device according to any one of (8) to (9) above.
(12)
The first sensor is a light-receiving sensor that performs a light-receiving operation using an iToF (indirect time of flight) ranging method that outputs a light-receiving signal from which the distance image data and brightness image data are obtained;
The information processing device according to (11), wherein the alternative distance image data generation unit uses the luminance image data and the second image data as two images used in the distance measurement method using the stereo images.
(13)
The information processing device according to (2) above, wherein the correction processing unit performs the correction using information on peripheral pixels of the correction target area.
(14)
The information processing device according to any one of (1) to (13) above, wherein the first sensor is a sensor having sensitivity to IR (Infrared) light.
(15)
First image data as distance image data obtained based on a light reception signal of a first sensor that is a light reception sensor that performs a light reception operation for distance measurement, and a second sensor that is a light reception sensor of a different type from the first sensor. By providing the second image data, which is image data obtained based on the light reception signal of An information processing method, wherein an information processing device performs a process of inferring a distance measurement error area that occurs in the first image data due to interference light in a received light wavelength band as a correction target area.
(16)
First image data as distance image data obtained based on a light reception signal of a first sensor that is a light reception sensor that performs a light reception operation for distance measurement, and a second sensor that is a light reception sensor of a different type from the first sensor. By providing the second image data, which is image data obtained based on the light reception signal of A program that causes an arithmetic processing unit to execute a function of inferring a distance measurement error area that occurs in the first image data due to interference light in a received light wavelength band as a correction target area.

1 AMR (information processing equipment)
1' Opposing AMR (external object)
3, 3A iToF sensor (first sensor)
5, 5A RGB sensor (second sensor)
32 AI image processing section 33 Correction processing section 34 Alternative distance image data generation section S1, S2 Light reception signal DD1 Primary distance image data (first image data, distance image data)
BD Brightness image data CD RGB image data (second image data)
SD Alternative distance image data ArC Correction target area PT Obstruction pattern PT1 1st pattern PT2 2nd pattern PT3 3rd pattern PT4 4th pattern BO Obstacle M1 1st AI model (artificial intelligence model)
M2 2nd AI model (artificial intelligence model)

Claims (16)

1. First image data as distance image data obtained based on a light reception signal of a first sensor that is a light reception sensor that performs a light reception operation for distance measurement, and a second sensor that is a light reception sensor of a different type from the first sensor. By providing the second image data, which is image data obtained based on the light reception signal of An information processing device comprising an AI (Artificial Intelligence) image processing unit that performs a process of inferring a distance measurement error area that occurs in the first image data due to interference light in a received light wavelength band as a correction target area.
2. The information processing apparatus according to claim 1, further comprising a correction processing unit that corrects distance information regarding the correction target area in the first image data and outputs corrected distance image data.
3. The information processing device according to claim 1, wherein the external object is an AMR (Autonomous Mobile Robot).
4. The artificial intelligence model includes first normal image data that is the first image data in which the correction target area does not exist, and first correction-required image data that is the first image data that has the correction target area. and second normal image data based on the second image data, the information processing apparatus according to claim 1, is obtained by machine learning using learning data that is a set of the following: and second normal image data based on the second image data.
5. The information processing device according to claim 4, wherein the learning data further includes correct data regarding the correction target area.
6. The information processing device according to claim 1, wherein the AI image processing unit performs a process of inferring a disturbance pattern that is a type of disturbance mode of the disturbance light using an artificial intelligence model.
7. The information processing device according to claim 6, wherein the interference pattern is classified into types according to the direction of the interference light.
8. The interference pattern includes a first pattern in which direct light of the interference light does not enter the sensor surface of the first sensor, a second pattern in which the direct light does not enter the sensor surface, and a second pattern in which the direct light does not enter the sensor surface. at least a third pattern in which the direct light is not incident on the sensor surface and there is no obstacle that blocks at least a portion of the interfering light; and a fourth pattern in which the direct light is not incident on the sensor surface and the obstacle is present. The information processing device according to claim 7.
9. comprising a correction processing unit that corrects distance information regarding the correction target area in the first image data and outputs corrected distance image data;
   The information processing device according to claim 8, wherein the correction processing unit performs the correction on the fourth pattern except for an area where the obstacle is imaged.
10. The artificial intelligence model that infers the disturbance pattern uses first normal image data that is the first image data in which the correction target area does not exist, and first normal image data that is the first image data that has the correction target area. The claim is that the image data is obtained by machine learning using learning data that is a set of first correction image data, second normal image data based on the second image data, and correct data for the interference pattern. The information processing device according to item 7.
11. an alternative distance image data generation unit that generates alternative distance image data by a distance measurement method using stereo images when the AI image processing unit obtains an inference result that the interference pattern is the second pattern; The information processing device according to claim 8.
12. The first sensor is a light-receiving sensor that performs a light-receiving operation using an iToF (indirect time of flight) ranging method that outputs a light-receiving signal from which the distance image data and brightness image data are obtained;
    The information processing device according to claim 11, wherein the alternative distance image data generation unit uses the luminance image data and the second image data as two images used in the distance measurement method using the stereo images.
13. The information processing device according to claim 2, wherein the correction processing unit performs the correction using information on peripheral pixels of the correction target area.
14. The information processing device according to claim 1, wherein the first sensor is a sensor having sensitivity to IR (Infrared) light.
15. First image data as distance image data obtained based on a light reception signal of a first sensor that is a light reception sensor that performs a light reception operation for distance measurement, and a second sensor that is a light reception sensor of a different type from the first sensor. By providing the second image data, which is image data obtained based on the light reception signal of An information processing method, wherein an information processing device performs a process of inferring a distance measurement error area that occurs in the first image data due to interference light in a received light wavelength band as a correction target area.
16. First image data as distance image data obtained based on a light reception signal of a first sensor that is a light reception sensor that performs a light reception operation for distance measurement, and a second sensor that is a light reception sensor of a different type from the first sensor. By providing the second image data, which is image data obtained based on the light reception signal of A program that causes an arithmetic processing unit to execute a function of inferring a distance measurement error area that occurs in the first image data due to interference light in a received light wavelength band as a correction target area.

Images