WO2022201848A1

WO2022201848A1 - Information processing device, information processing method, and program

Info

Publication number: WO2022201848A1
Application number: PCT/JP2022/003234
Authority: WO
Inventors: 英之市橋; 和憲神尾
Original assignee: ソニーグループ株式会社
Priority date: 2021-03-22
Filing date: 2022-01-28
Publication date: 2022-09-29

Abstract

An information processing device (30) has a depth estimation unit (32) and a depth correction unit (33). The depth estimation unit (32) estimates, on the basis of the result of time-of-flight measurement of a reference light (PL), the depth of a projection position at which the reference light (PL) is projected and a non-projection position that is offset from the projection position. The depth correction unit (33) extracts, as an error component, the estimated value of the depth of a position to be measured that is obtained when the reference light (PL) is projected to a position offset from the position to be measured of a subject (SU). The depth correction unit (33) removes the error component from the estimated value of the depth of the position to be measured that is obtained when the reference light (PL) is projected to the position to be measured.

Description

Information processing device, information processing method and program

The present invention relates to an information processing device, an information processing method, and a program.

Three-dimensional measurement technology using the ToF (Time of Flight) method is known. In this method, the depth of the object is detected based on information on the time from when the reference light is projected toward the object until the reflected light is received.

Japanese Patent Publication No. 2016-510396 JP 2014-138691 A JP 2019-153863 A

The ToF method includes the dToF (direct Time of Flight) method and the iToF (indirect Time of Flight) method. Each of these methods has advantages and disadvantages. For example, in the dToF method, the circuit is complicated, so the resolution is largely limited. Although the iToF method has a high resolution, there is a possibility that the depth is erroneously measured due to multipath interference.

Therefore, this disclosure proposes a depth measurement method that can reduce errors due to multipath interference.

According to the present disclosure, a depth estimating unit that estimates the depth of a projection position where the reference light is projected and a non-projection position shifted from the projection position based on a measurement result of the time of flight of the reference light; is projected to a position deviated from the measurement target position of the subject, extracting an estimated value of the depth of the measurement target position as an error component, and obtaining when the reference light is projected to the measurement target position and a depth correction unit that removes the error component from the estimated value of the depth of the measurement target position. Further, according to the present disclosure, there are provided an information processing method in which the information processing of the information processing device is executed by a computer, and a program for causing the computer to implement the information processing of the information processing device.

It is a figure explaining the influence of multipath in depth measurement. It is a figure explaining the influence of multipath in depth measurement. It is a figure explaining the influence of multipath in depth measurement. It is a figure explaining the influence of multipath in depth measurement. It is a figure explaining the influence of multipath in depth measurement. 1 is a diagram showing a schematic configuration of a depth measurement system; FIG. 4 is a functional block diagram of a processing unit; FIG. FIG. 10 is a diagram showing a process in which a synthetic depth map is densified; It is a figure explaining the removal principle of a multipath component. It is a figure explaining the removal principle of a multipath component. It is a figure explaining the removal principle of a multipath component. It is a figure explaining the removal principle of a multipath component. It is a figure which shows an example of termination determination. FIG. 4 is a diagram showing an example of a method of calculating a normal vector N; FIG. 5 is a diagram showing an example of information processing related to depth estimation; FIG. 5 is a diagram showing an example of information processing related to depth estimation;

Below, embodiments of the present disclosure will be described in detail based on the drawings. In each of the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

The description will be given in the following order.
[1. Influence of multipath on depth measurement]
[2. Measurement of depth using scanning of reference light]
[2-1. System overview]
[2-2. Densification of synthetic depth map]
[2-3. Elimination Principle of Multipath Components]
[2-4. Abort Judgment]
[3. Information processing method]
[4. effect]

[1. Influence of multipath on depth measurement]
1 to 5 are diagrams for explaining the influence of multipath on depth measurement.

The received light waveform of the reference light PL includes not only direct reflection components from the subject SU, but also multipath components caused by scattering within the subject SU and indirect reflection from other objects. If the depth is calculated based on received light data containing multipath components, an error occurs in the estimated value of the depth. This problem is known as the multipath problem.

For example, FIGS. 1 and 2 are examples of errors in depth estimates due to indirect reflections from other objects. As shown in FIG. 1, a highly reflective object HR such as a whiteboard exists around the subject SU. The ToF camera SE projects reference light PL toward the subject SU and receives its reflected light. When the distance between the subject SU and the highly reflective object HR is short, a mixture of the direct reflected light DR that does not pass through the highly reflective object HR and the multipath light MR that passes through the highly reflective object HR enters the ToF camera SE. . As a result, as shown in FIG. 2, the depth estimate DO is calculated as a larger value than the ground truth GT. In the example of FIG. 2, there is an error of about 100 mm.

FIGS. 3 to 5 are examples in which an error occurs in the estimated value of the depth due to scattering by the object SU. As shown in FIGS. 3 and 4, the subject SU includes an opaque object area OPA and a translucent object area TRA. Multiple reflections of the reference beam PL occur inside the translucent object area TRA. Multipath light MR generated by multiple reflection is detected by ToF camera SE together with direct reflected light DR. As a result, as shown in FIG. 5, the estimated value DO of the depth of the translucent object area TRA is calculated as a larger value than the ground truth GT. In the example of FIG. 5, there is an error of about 30 mm.

[2. Measurement of depth using scanning of reference light]
The above-described error in the estimated value of depth is caused by measuring the route information of the multipath light MR mixed with the route information of the directly reflected light DR. Therefore, in the present disclosure, the route information of the directly reflected light DR and the multipath light MR are separated by performing multiple measurements with different projection positions (direct positions) of the reference light PL. The depth measurement method of the present disclosure will be described below.

[2-1. System configuration]
FIG. 6 is a diagram showing a schematic configuration of the depth measuring system 1. As shown in FIG. FIG. 7 is a functional block diagram of the processing section 30. As shown in FIG.

The measurement system 1 has a light projecting section 10 , a light receiving section 20 , a processing section 30 and a storage section 50 .

The light projecting section 10 includes, for example, an actuator 11 and a light source 12 (see FIG. 15). Light source 12 is a laser or a projector that outputs reference light PL. The actuator 11 is a scanner that scans the subject SU with the reference light PL emitted from the light source 12 . The reference light PL is, for example, infrared light modulated with a sine wave.

The light receiving section 20 is, for example, an image sensor having a plurality of pixels PX (see FIG. 14) arranged in the x and y directions. The z-direction orthogonal to the arrangement direction of the pixels PX is the depth direction. A light receiving element that detects the reference light PL is provided in the pixel PX. The light receiving unit 20 receives the reference light PL (reflected light RL) reflected by the subject SU.

The light projecting unit 10 projects, as the reference light PL, a dot pattern PAT in which a plurality of dots DT are two-dimensionally arranged at regular intervals. The light projecting unit 10 scans the entire subject SU with the reference light PL at high density while shifting the projection position IP of the dot pattern PAT for each preset measurement period (one frame). The projection position IP is the measurement target position TA where depth measurement is performed using the directly reflected light DR.

The light receiving unit 20 receives the reference light PL in accordance with the switching of the projection position IP. The light receiving unit 20 outputs the received light data for one measurement period to the processing unit 30 as frame data indicating the measurement result of the flight time of the reference light PL. The light receiving unit 20 continues to generate frame data until the scanning of the reference beam PL is completed, and sequentially outputs the generated frame data to the processing unit 30 .

The frame data is time-series luminance data of each pixel PX measured in one measurement period. The frame data includes brightness data of the projection position IP and non-projection position NIP of the dot pattern PAT. The projection position IP is a position on the subject SU where the dot pattern PAT is directly projected. The non-projection position NIP is a position on the subject SU where the dot pattern PAT is not directly projected. The brightness data of the reference light PL at the projection position IP includes brightness data of both the direct reflected light DR and the multipath light MR. The brightness data of the reference light PL at the non-projection position NIP includes only the brightness data of the multipath light MR and does not include the brightness data of the directly reflected light DR.

The processing unit 30 is an information processing device that processes various types of information. The processing unit 30 synchronously drives the light projecting unit 10 and the light receiving unit 20 and generates a depth map DM (see FIG. 8) based on the frame data acquired from the light receiving unit 20 .

The processing unit 30 has a data acquisition unit 31 , depth estimation unit 32 , depth correction unit 33 , projection control unit 34 and discontinuation determination unit 35 .

The projection control unit 34 controls the operations of the light source 12 and the actuator 11 based on the control plan CP (see FIG. 15). The control plan CP includes information such as the pattern shape of the reference beam PL, projection position IP and projection timing. Based on the control plan CP, the projection control unit 34 shifts the projection position IP of the dot pattern PAT for each measurement period while scanning the entire subject SU with the reference light PL at high density.

The data acquisition unit 31 sequentially acquires frame data from the light receiving unit 20 every measurement period in accordance with the switching of the projection position IP. The depth estimation unit 32 sequentially generates a plurality of depth maps DM based on a plurality of frame data sequentially acquired from the data acquisition unit 31 .

For example, the depth estimation unit 32 extracts the luminance data of the reference light PL of each pixel PX from the frame data. The depth estimating unit 32 calculates the flight time of the reference light PL (the time from when the reference light PL is projected until it is reflected by the subject SU and received by the light receiving unit 20) from the arrival time delay obtained from the phase difference of the reflected light RL. time) is detected indirectly (iToF method). The depth estimation unit 32 estimates the depth of the subject SU for each pixel PX based on the flight time.

The frame data includes luminance data of the reference light PL for the entire subject SU including the projection position IP and the non-projection position NIP. The depth estimation unit 32 estimates the depth of each pixel PX based on the frame data. Thereby, the depths of both the projection position IP where the reference light PL is projected and the non-projection position NIP shifted from the projection position IP are estimated.

The depth correction unit 33 calculates the estimated value of the depth of the measurement target position TA obtained when the reference light PL is projected onto the measurement target position TA by combining the true value of the depth, the error component caused by the multipath light MR, Estimate a mixed depth value containing The depth correction unit 33 estimates the estimated value of the depth of the measurement target position TA obtained when the reference light PL is projected at a position shifted from the measurement target position TA as the error component. The depth correction unit 33 extracts, as error component information, depth value information estimated to be an error component from the plurality of depth maps DM sequentially generated by the depth estimation unit 32 . The depth correction unit 33 corrects the mixed depth value based on the error component by removing the error component from the mixed depth value.

The depth correction unit 33 combines true value information extracted from a plurality of depth maps DM based on error component information to generate a combined depth map ADM. For example, the depth correction unit 33 corrects the latest depth map DM generated by the depth estimation unit 32 based on the error component information. The depth correction unit 33 synthesizes the latest corrected depth map DM with the latest synthesized depth map ADM. The most recent synthetic depth map ADM is a synthetic depth map ADM generated based on frame data of a plurality of measurement periods up to the last time. The depth correction unit 33 repeats correction and synthesis of the depth map DM every measurement period until the scanning of the reference light PL is completed. As a result, the density of the synthetic depth map ADM is gradually increased.

[2-2. Densification of synthetic depth map]
FIG. 8 is a diagram showing the process of densifying the composite depth map ADM.

The first depth map DM1 generated in the first measurement period is a sparse depth map DM corresponding to the dot pattern PAT. The measurement system 1 sequentially generates the depth map DM while shifting the projection position IP of the dot pattern PAT. The latest depth map DM is combined with the latest synthesized depth map ADM after being corrected based on the error component information. As a result, a new synthesized depth map ADM is generated by adding information on the depth of the latest measurement target position TA to the latest synthesized depth map ADM.

If the number of dots DT included in the dot pattern PAT is q, depth information of the subject SU at q locations can be obtained by one measurement. In the Q-th measurement, a synthetic depth map ADM including information on the depth of the subject SU at q×Q locations is generated. Even if a sparse dot pattern PAT is used, repeated measurements generate a high-density synthetic depth map ADM in which depth information is accumulated.

[2-3. Elimination Principle of Multipath Components]
9 to 12 are diagrams for explaining the principle of removing multipath components.

The projection control unit 34 sets a plurality of measurement target positions TA whose depth is to be measured on the subject SU. The projection control unit 34 divides the projection of the reference light PL onto all the measurement target positions TA into a plurality of times by shifting the projection position IP of the dot pattern PAT for each measurement period.

For example, the projection control unit 34 classifies all measurement target positions TA into a plurality of phase groups based on the projection timing of the reference light PL. The reference beam PL is projected at the same timing on a plurality of measurement target positions TA belonging to the same phase group. The projection control unit 34 sequentially projects the reference light PL for each phase group.

In the example of FIG. 9, a plurality of measurement target positions TA are classified into two phase groups of odd-numbered phases and even-numbered phases. In the first measurement, a plurality of measurement target positions TA classified as odd phases (odd phase positions TA _odd ) are projection positions IP. A plurality of measurement target positions TA classified as even phases (even phase positions TA _even ) are non-projection positions NIP where the dot pattern PAT is not projected (directly). In the second measurement, the even phase position TA _even becomes the projected position IP and the odd phase position TA _odd becomes the non-projected position NIP.

As shown in FIGS. 10 and 11 , the plurality of pixels PX of the light receiving unit 20 includes a plurality of odd phase pixels PX odd for measuring a plurality of odd phase positions TA _odd and a plurality of pixels PX _odd for measuring a plurality of even phase positions TA _even . , even phase pixels PX _even . Directly reflected light DR from the odd phase position TA _odd enters the odd phase pixel PX _odd when the odd phase position TA _odd becomes the projection position IP. Directly reflected light DR from the even phase position TA _even enters the even phase pixel PX _even when the even phase position TA _even becomes the projection position IP.

In the first measurement, a plurality of odd-numbered phase positions TA _odd are projected positions IP. Therefore, as shown in FIG. 10, the odd-numbered phase pixel PX _odd receives the direct reflected light DR reflected at the odd-numbered phase position TA _odd , which is the projection position IP, and the multipath light MR reflected by the highly reflective object HR. Incident. On the other hand, as shown in FIG. 11, only the multipath light MR reflected by the highly reflective object HR is incident on the even phase pixel PX _even . Since the projection position IP is not the even phase position TA _even (the even phase position TA _even is the non-projection position NIP), the direct reflected light DR reflected at the projection position IP does not enter the even phase pixels PX _even .

As described above, the multipath light MR, which is an error component, is measured by both the odd phase pixels PX _odd and the even phase pixels PX _even . The directly reflected light DR indicating the true depth value is measured only by the odd-numbered phase pixels PX _odd . The measurement system 1 estimates the depth of each measurement target position TA based on the relationship between the projection timing and the measurement data of each phase group.

For example, as shown in FIG. 12, the projection control unit 34 projects the dot pattern PAT to a plurality of odd-numbered phase positions TA _odd to perform the first measurement. The depth estimation unit 32 estimates the depth of each pixel PX based on the frame data obtained by the first measurement. The depth estimation unit 32 generates a first depth map DM1 based on the estimated value of the depth of each pixel PX. The depth correction unit 33 treats the depth value d _even,k,l (k,l: x and y coordinates of the pixel) of each even-numbered phase pixel PX _even (k,l) as an error component caused by the multipath light MR. presume. The depth correction unit 33 converts the depth value d _odd,i,j (i, j: x and y coordinates of the pixel) of each odd-numbered phase pixel PX _odd (i, j) into a mixed depth value including a true value and an error component. estimated as

Next, the projection control unit 34 projects the dot pattern PAT to a plurality of even-numbered phase positions TA _even to perform the second measurement. The depth estimation unit 32 estimates the depth of each pixel PX based on the frame data obtained by the second measurement. The depth estimation unit 32 generates a second depth map DM2 based on the depth estimation value of each pixel PX. The depth correction unit 33 estimates the depth value d _odd,i,j of each odd-numbered phase pixel PX _odd (i,j) as an error component caused by the multipath light MR. The depth correction unit 33 estimates the depth value d _even,k,l of each even-numbered phase pixel PX _even (k,l) as a mixed depth value including a true value and an error component.

The depth correction unit 33 extracts, as error component information, depth value information that is estimated to be an error component from the first depth map DM1 and the second depth map DM2. The depth correction unit 33 corrects the first depth map DM1 and the second depth map DM2 based on the error component information.

For example, the depth correction unit 33, from the depth value d _odd,i,j of the odd-numbered phase pixel PX _odd (i,j) included in the first depth map DM1, based on the following equation (1), the error component , the depth value d _odd,i,j of the same odd-phase pixel PX _odd (i,j) contained in the second depth map DM2.

The depth _correction unit 33 calculates _the error component, the first Subtract the depth values d _even,k,l of the same even-phase pixel PX _even (k,l) included in one depth map DM1.

The depth correction unit 33 synthesizes the corrected first depth map DM1 and the corrected second depth map DM2 to generate a high-density synthetic depth map ADM from which error components are removed.

[2-4. Abort Judgment]
In the above example, the number of phase groups (number of phases) is two, odd phases and even phases. However, the number of phases should be two or more, and may be three or more. The number of phases indicates the number of measurements required to scan all the measurement target positions TA. As the number of phases increases, the density of the depth map DM increases, but the measurement time increases. When the shape of the subject SU is simple, even if the density of the depth map DM is small, the shape of the subject SU may be accurately estimated by interpolation processing or the like. In such a case, the measurement can be terminated after a number of times less than the number of phases.

For example, the discontinuation determination unit 35 sequentially acquires the depth map DM generated for each measurement period from the depth correction unit 33 . The discontinuation determination unit 35 determines discontinuation of measurement based on transition of the estimated value of depth updated for each measurement period. The discontinuation determination unit 35 determines to discontinue the measurement when the transition of the estimated depth value satisfies a preset saturation condition. When the termination is determined, the termination determination section 35 outputs termination information to the projection control section 34 and the data acquisition section 31 . The projection control unit 34 and the data acquisition unit 31 stop projecting the reference beam PL and acquiring the received light data based on the discontinuation information.

FIG. 13 is a diagram showing an example of termination determination.

The discontinuation determination unit 35 extracts the geometry information GI of the subject SU from the depth map DM. The geometry information GI is calculated using the estimated value of the depth of the measurement target position TA that has already been measured. The discontinuation determination unit 35 updates the geometry information GI for each measurement period based on the latest measurement result. The discontinuation determination unit 35 detects changes in the geometry information GI before and after updating. The discontinuation determination unit 35 determines whether or not the transition of the estimated depth value satisfies the saturation condition based on the change in the geometry information GI. If the saturation condition is met, it is decided to truncate the measurement.

For example, the discontinuation determination unit 35 extracts, as the geometry information GI, information on the normal vector N of the subject SU at each measurement target position TA that has already been measured. If the change in the normal vector N before and after the update (magnitude of the difference in the normal vector) is equal to or less than a preset threshold at all the measurement target positions TA that have already been measured, the discontinuation determination unit 35 (|N _t −N _t+1 |≦threshold), it is determined that the saturation condition is satisfied. Note that the symbol t indicates the number of measurements. "t=n" means that the projection of the reference beam PL has been performed up to the nth phase group.

The upper part of FIG. 13 shows an example in which the shape of the subject SU is a simple shape without irregular unevenness, such as a curved surface (scene A). In scene A, even if the number of measurements increases to n, (n+1), and (n+2), the orientation of the updated normal vector N does not change significantly. The discontinuation determination unit 35 determines discontinuation of the measurement with emphasis on efficiency of the measurement. The depth correction unit 33 interpolates the depth of the measurement target position TA for which measurement has not been completed, based on the estimated value of the depth of another measurement target position TA that has already been measured.

The lower part of FIG. 13 shows an example in which the shape of the subject SU is a complex shape with unevenness such as steps (scene B). In scene B, the direction of the normal vector N changes greatly for each measurement. The discontinuation determination unit 35 determines that the shape of the subject SU cannot be accurately grasped only from the measured data, and determines to continue the measurement.

FIG. 14 is a diagram showing an example of a method for calculating the normal vector N.

The discontinuation determination unit 35 determines the normal vector N of the subject SU at the position i on the subject SU from the three-dimensional coordinates of the position i and the positions j ₀ , j ₁ , j ₂ , and j ₃ adjacent to the position i. Calculated using dimensional coordinates. The _three -dimensional coordinates of the positions _i , _j0 , _j1 , _j2 , and _j3 are the xy coordinates of the pixels _PXi , _PXj0 , PXj1, PXj2, and _PXj3 corresponding to each position, and the pixels _PXi , Using depth estimates D(i), D(j ₀ ), D(j ₁ ), D(j ₂ ), D(j ₃ ) detected at PX _j0 , PX _j1 , PX _j2 , PX _j3 Calculated.

For example, the discontinuation determination unit 35 calculates four vectors S ₀ , S ₁ , S ₂ , S ₃ directed from the positions j ₀ , j ₁ , j ₂ , j ₃ to the position i. The discontinuation determining unit 35 divides the rectangular area surrounded by the positions j ₀ , j ₁ , j ₂ , and j ₃ into four triangular areas with the position i as the vertex. The discontinuation determining unit 35 calculates normal vectors N ₁₀ , N ₀₂ , N ₂₃ and N ₃₁ of each triangular region using the vectors S ₀ , S ₁ , S ₂ and S ₃ . The discontinuation determination unit 35 calculates the average vector of the normal vectors N ₁₀ , N ₀₂ , N ₂₃ and N ₃₁ of the four triangular regions as the normal vector N at the position i.

The setting information 51 includes the above-described various conditions and information on criteria such as thresholds. The setting information 51 and the program 59 used for the above process are stored in the storage unit 50 . The program 59 is a program that causes a computer to execute information processing according to this embodiment. The processing section 30 performs various processes according to the program 59 stored in the storage section 50 . The storage unit 50 may be used as a work area (buffer) that temporarily stores the processing results of the processing unit 30 . Storage unit 50 includes, for example, any non-transitory storage medium such as a semiconductor storage medium and a magnetic storage medium. The storage unit 50 includes, for example, an optical disc, a magneto-optical disc, or a flash memory. The program 59 is stored, for example, in a non-transitory computer-readable storage medium.

The processing unit 30 is, for example, a computer configured with a processor and memory. The memory of the processing unit 30 includes RAM (Random Access Memory) and ROM (Read Only Memory). The processing unit 30 functions as a data acquisition unit 31 , a depth estimation unit 32 , a depth correction unit 33 , a projection control unit 34 and an end determination unit 35 by executing the program 59 .

[3. Information processing method]
15 and 16 are diagrams illustrating an example of information processing related to depth estimation.

In step S1, the projection control unit 34 sets the phase group number (group number) n to 1. In step S2, the projection control section 34 determines whether or not the group number n is smaller than the number of phases. If it is determined in step S2 that the group number n is the same as the number of phases (step S2: No), the process ends. If it is determined in step S2 that the group number n is smaller than the number of phases (step S2: No), the process proceeds to step S3. In step S3, the projection control unit 34 controls the light projecting unit 10 so that the plurality of measurement target positions TA corresponding to the group number n become the projection positions IP.

In step S4, the light projecting unit 10 projects the reference light PL onto a plurality of measurement target positions TA corresponding to the group number n at the timing controlled by the projection control unit 34. The light receiving unit 20 receives the reference light PL reflected by the subject SU, and outputs luminance data of the reference light PL measured at each pixel PX as frame data. The depth estimation unit 32 generates a depth map DM (first depth map DM1) based on the frame data.

In step S5, the projection control unit 34 controls the light projecting unit 10 so that the plurality of measurement target positions TA corresponding to the next group number n become the projection positions IP. In the example of FIG. 15, all the measurement target positions TA are divided into odd-numbered phases and even-numbered phases. Therefore, the projection position IP for the second time is in phase opposite to the projection position IP for the first time.

In step S6, the light projecting unit 10 projects the reference light PL onto a plurality of measurement target positions TA corresponding to the next group number n at the timing controlled by the projection control unit 34. The light receiving unit 20 receives the reference light PL reflected by the subject SU, and outputs luminance data of the reference light PL measured at each pixel PX as frame data. The depth estimation unit 32 generates a depth map DM (second depth map DM2) based on the frame data.

In step S7, the depth correction unit 33 extracts error component information from the first depth map DM1 and the second depth map DM2. The depth correction unit 33 corrects the first depth map DM1 and the second depth map DM2 based on the error component information. In step S8, the depth correction unit 33 combines the corrected first depth map DM1 and the corrected second depth map DM2 to generate a combined depth map ADM.

In the example of FIG. 15, the measurement target position TA is classified into two phase groups, so the measurement is completed twice. However, when the measurement target position TA is classified into three or more phase groups, discontinuation determination is performed in step S9. The discontinuation determination unit 35 extracts the geometry information GI of the subject SU from both the corrected first depth map DM1 and the combined depth map ADM. The discontinuation determination unit 35 applies changes in the geometry information GI updated every measurement period based on the latest measurement result to the saturation condition.
Based on, it is determined whether or not to terminate the measurement.

In step S10, the discontinuation determination unit 35 determines whether or not to discontinue measurement based on changes in the geometry information GI. In step S10, when the termination is determined (step S10: Yes), the processing section 30 ends the processing. Due to the discontinuation of measurement, the depth of the measurement target position TA for which the measurement has not been completed is interpolated based on the estimated values of the depths of the other measurement target positions TA that have already been measured.

In step S10, if it is determined to continue the measurement (step S10: No), the process proceeds to step S11. In step S11, the processing unit 30 advances the group number n by one, and repeats the above-described processing until termination is determined.

[4. effect]
The processing unit 30 has a depth estimation unit 32 and a depth correction unit 33 . The depth estimator 32 estimates the depth of the projection position IP where the reference light PL is projected and the depth of the non-projection position NIP shifted from the projection position IP, based on the measurement result of the flight time of the reference light PL. The depth correction unit 33 extracts, as an error component, an estimated value of the depth of the measurement target position TA obtained when the reference light PL is projected at a position shifted from the measurement target position TA of the subject SU. The depth correction unit 33 removes the error component from the estimated value of the depth of the measurement target position TA obtained when the reference light PL is projected onto the measurement target position TA. In the information processing method of this embodiment, the processing of the processing unit 30 described above is executed by a computer. The program of the present embodiment causes a computer to implement the processing of the processing unit 30 described above.

According to this configuration, the error component caused by the multipath light MR is removed satisfactorily. Therefore, highly accurate depth information can be obtained.

The processing unit 30 has a projection control unit 34. The projection control unit 34 scans the subject SU with the reference light PL while shifting the projection position IP of the reference light PL for each measurement period.

According to this configuration, the subject SU can be scanned with the reference light PL at high density.

The projection control unit 34 projects the reference light PL as a dot pattern PAT in which a plurality of dots DT are two-dimensionally arranged.

According to this configuration, the depth measurement of a plurality of measurement target positions TA is performed at once.

The depth estimation unit 32 generates a plurality of depth maps DM based on a plurality of received light data of the reference light PL acquired for each measurement period in accordance with the switching of the projection position IP. The depth correction unit 33 extracts, as error component information, information on depth values estimated as error components from the plurality of depth maps DM. The depth correction unit 33 extracts true value information from the plurality of depth maps DM based on the error component information. The depth correction unit 33 combines the extracted true value information to generate a combined depth map ADM.

According to this configuration, a high-density synthetic depth map ADM in which depth information is accumulated for each measurement is generated.

The processing unit 30 has an discontinuation determination unit 35 . The discontinuation determination unit 35 extracts the geometry information GI of the subject SU from the synthetic depth map ADM. The discontinuation determination unit 35 determines whether or not to discontinue measurement based on changes in the geometry information GI updated every measurement period based on the latest measurement results.

According to this configuration, it is possible to prevent the measurement time from becoming longer than necessary.

The depth correction unit 33 interpolates the depth of the measurement target position TA for which measurement has not been completed due to the discontinuation of measurement, based on the estimated value of the depth of another measurement target position TA that has already been measured.

According to this configuration, the density of depth information does not decrease even if measurement is discontinued.

It should be noted that the effects described in this specification are only examples and are not limited, and other effects may also occur.

[Appendix]
Note that the present technology can also take the following configuration.
(1)
a depth estimating unit that estimates the depth of a projection position where the reference light is projected and a non-projection position shifted from the projection position based on the measurement result of the time of flight of the reference light;
When the reference light is projected onto the measurement target position, an estimated value of the depth of the measurement target position obtained when the reference light is projected to the measurement target position of the subject is extracted as an error component, and the reference light is projected onto the measurement target position. a depth correction unit that removes the error component from the estimated value of the depth of the measurement target position obtained in
Information processing device having
(2)
a projection control unit that scans the subject with the reference light while shifting the projection position of the reference light for each measurement period;
The information processing apparatus according to (1) above.
(3)
The projection control unit projects the reference light as a dot pattern in which a plurality of dots are arranged two-dimensionally.
The information processing apparatus according to (2) above.
(4)
The depth estimating unit generates a plurality of depth maps based on a plurality of received light data of the reference light acquired for each measurement period in accordance with the switching of the projection position,
The depth correction unit extracts information of the depth value estimated as the error component from the plurality of depth maps as error component information, and extracts a true value extracted from the plurality of depth maps based on the error component information. generate a synthetic depth map by synthesizing information from
The information processing apparatus according to (2) or (3) above.
(5)
A discontinuation determination unit that extracts geometry information of the subject from the synthetic depth map and determines whether or not to discontinue measurement based on changes in the geometry information that is updated every measurement period based on the latest measurement results. having
The information processing apparatus according to (4) above.
(6)
The depth correction unit interpolates the depth of the measurement target position for which measurement has not been completed due to discontinuation of measurement, based on an estimated value of the depth of the other measurement target position that has already been measured.
The information processing apparatus according to (5) above.
(7)
estimating the depth of a projection position where the reference light is projected and a non-projection position shifted from the projection position based on the measurement result of the flight time of the reference light;
extracting, as an error component, an estimated value of the depth of the measurement target position obtained when the reference light is projected at a position shifted from the measurement target position of the object;
removing the error component from an estimated value of the depth of the measurement target position obtained when the reference light is projected onto the measurement target position;
A computer-implemented information processing method, comprising:
(8)
estimating the depth of a projection position where the reference light is projected and a non-projection position shifted from the projection position based on the measurement result of the flight time of the reference light;
extracting, as an error component, an estimated value of the depth of the measurement target position obtained when the reference light is projected at a position shifted from the measurement target position of the object;
removing the error component from an estimated value of the depth of the measurement target position obtained when the reference light is projected onto the measurement target position;
A program that makes a computer do something.

30 processing unit (information processing device)
32 Depth estimation unit 33 Depth correction unit 34 Projection control unit 35 Abort determination unit 59 Program ADM Combined depth map DM Depth map DT Dot GI Geometry information IP Projection position NIP Non-projection position PAT Dot pattern PL Reference light SU Subject TA Measurement target position

Claims

a depth estimating unit that estimates the depth of a projection position where the reference light is projected and a non-projection position shifted from the projection position based on the measurement result of the time of flight of the reference light;
When the reference light is projected onto the measurement target position, an estimated value of the depth of the measurement target position obtained when the reference light is projected to the measurement target position of the subject is extracted as an error component, and the reference light is projected onto the measurement target position. a depth correction unit that removes the error component from the estimated value of the depth of the measurement target position obtained in
Information processing device having
a projection control unit that scans the subject with the reference light while shifting the projection position of the reference light for each measurement period;
The information processing device according to claim 1 .
The projection control unit projects the reference light as a dot pattern in which a plurality of dots are arranged two-dimensionally.
The information processing apparatus according to claim 2.
The depth estimating unit generates a plurality of depth maps based on a plurality of received light data of the reference light acquired for each measurement period in accordance with the switching of the projection position,
The depth correction unit extracts information of the depth value estimated as the error component from the plurality of depth maps as error component information, and extracts a true value extracted from the plurality of depth maps based on the error component information. generate a synthetic depth map by synthesizing information from
The information processing apparatus according to claim 2.
A discontinuation determination unit that extracts geometry information of the subject from the synthetic depth map and determines whether or not to discontinue measurement based on changes in the geometry information that is updated every measurement period based on the latest measurement results. having
The information processing apparatus according to claim 4.
The depth correction unit interpolates the depth of the measurement target position for which measurement has not been completed due to discontinuation of measurement, based on an estimated value of the depth of the other measurement target position that has already been measured.
The information processing device according to claim 5 .
estimating the depth of a projection position where the reference light is projected and a non-projection position shifted from the projection position based on the measurement result of the flight time of the reference light;
extracting, as an error component, an estimated value of the depth of the measurement target position obtained when the reference light is projected at a position shifted from the measurement target position of the object;
removing the error component from an estimated value of the depth of the measurement target position obtained when the reference light is projected onto the measurement target position;
A computer-implemented information processing method, comprising:
estimating the depth of a projection position where the reference light is projected and a non-projection position shifted from the projection position based on the measurement result of the flight time of the reference light;
extracting, as an error component, an estimated value of the depth of the measurement target position obtained when the reference light is projected at a position shifted from the measurement target position of the object;
removing the error component from an estimated value of the depth of the measurement target position obtained when the reference light is projected onto the measurement target position;
A program that makes a computer do something.