WO2025041215A1 - Information processing device, information processing method, and program - Google Patents

Abstract

This information processing device comprises: an acquiring means that acquires input data; a point cloud data generating means that generates three-dimensional point cloud data from the input data; a thinned point cloud data generating means that generates three-dimensional thinned point cloud data by using thinning processing on the input data and/or the three-dimensional point cloud data; and a training means that refers to the three-dimensional point point cloud data and the three-dimensional thinned point cloud data to train an estimation model that uses point cloud data as input and uses completed point cloud data and a segmentation label pertaining to the completed point cloud data as output.

Description

Information processing device, information processing method, and program

The present invention relates to an information processing device, an information processing method, and a program.

A technique is known for segmenting target data, such as point cloud data, by attaching a label to each data point in the target data indicating which of multiple candidate regions the data point belongs to (Non-Patent Document 1, etc.).

Angela Dai et. al., ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans, 2018 IEEE /CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City,UT, USA, 2018 pp. 4578-4587.

The technology described in Non-Patent Document 1 aims to improve classification accuracy by using missing synthetic data to learn both the completion of missing or occluded areas and semantic segmentation.

However, the technology described in Non-Patent Document 1 requires the preparation of synthetic data in advance, which increases the time and effort required to create and pre-process training data (i.e., the cost of training).

This disclosure has been made in consideration of the above problems, and its purpose is to provide a highly accurate segmentation technique while suppressing increases in costs.

An information processing device according to one aspect of the present disclosure includes an acquisition means for acquiring input data, a point cloud data generation means for generating three-dimensional point cloud data from the input data, a thinned point cloud data generation means for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data, and a learning means for training an estimation model that receives point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

An information processing device according to one aspect of the present disclosure includes an acquisition means for acquiring input data, a point cloud data generation means for generating three-dimensional point cloud data from the input data, and an estimation means for estimating the complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means and the segmentation label for the complemented point cloud data using an estimation model that receives point cloud data as an input and outputs complemented point cloud data and a segmentation label for the complemented point cloud data, the estimation model being machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

An information processing method according to one aspect of the present disclosure includes acquiring input data, generating three-dimensional point cloud data from the input data, generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data, and training an estimation model that receives point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

An information processing method according to one aspect of the present disclosure includes acquiring input data, generating three-dimensional point cloud data from the input data, and estimating the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the generating step and the segmentation label related to the complemented point cloud data using an estimation model that takes the point cloud data as input and outputs complemented point cloud data and a segmentation label related to the complemented point cloud data, the estimation model being machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

A program according to one aspect of the present disclosure causes a computer to execute an acquisition process for acquiring input data, a point cloud data generation process for generating three-dimensional point cloud data from the input data, a thinned point cloud data generation process for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data, and a learning process for training an estimation model that takes point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

A program according to one aspect of the present disclosure causes a computer to execute an acquisition process for acquiring input data, a point cloud data generation process for generating three-dimensional point cloud data from the input data, and an estimation process for estimating the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and the segmentation label for the complemented point cloud data using an estimation model that takes point cloud data as input and outputs complemented point cloud data and segmentation labels for the complemented point cloud data, the estimation model being machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

This disclosure makes it possible to provide highly accurate segmentation technology while suppressing increases in costs.

1 is a block diagram showing a configuration of an information processing device according to the present disclosure. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. 1 is a diagram for explaining a data flow in an information processing device according to the present disclosure. 1 is a diagram for explaining a data flow in an information processing device according to the present disclosure. 1 is a diagram for explaining a data flow in an information processing device according to the present disclosure. 1 is a diagram for explaining a data flow in an information processing device according to the present disclosure. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. 1 is a diagram for explaining a data flow in an information processing device according to the present disclosure. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 2 is a diagram for explaining processing executed by an information processing device according to the present disclosure. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. 1 is a block diagram showing a hardware configuration of an information processing device according to the present disclosure.

Below are examples of embodiments of the present invention. However, the present invention is not limited to the embodiments exemplified below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the embodiments exemplified below may also be included in the scope of the present invention. Alternatively, embodiments obtained by appropriately omitting some of the technical means employed in the embodiments exemplified below may also be included in the scope of the present invention. Furthermore, the effects mentioned in each of the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in each of the exemplary embodiments shown below may also be included in the scope of the present invention.

First Embodiment
A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form of each exemplary embodiment described later. The scope of application of each technical means adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means adopted in this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure to the extent that no particular technical obstacle occurs. In addition, each technical means shown in the drawings referred to for explaining this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure to the extent that no particular technical obstacle occurs.

(Configuration of information processing device 1)
The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1. As shown in Fig. 1, the information processing device 1 includes an acquisition unit 11, a point cloud data generation unit 12, a thinned point cloud data generation unit 13, and a learning unit 14.

(Acquisition unit 11)
The acquisition unit 11 acquires input data. Here, the input data is, for example, input data for the learning phase. In addition, although a specific example of the input data acquired by the acquisition unit 11 does not limit the present exemplary embodiment, for example,
The image data may be configured to include at least one of: RGB data in which each pixel (data point) represents an RGB value; depth data in which each pixel (data point) represents a depth value; and three-dimensional point cloud data in which each data point represents a three-dimensional coordinate. In addition, the three-dimensional point cloud data may be point cloud data acquired by a LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) device, as an example, but this example does not limit the present exemplary embodiment.

(Point cloud data generator 12)
The point cloud data generating unit 12 generates three-dimensional point cloud data from the input data acquired by the acquiring unit 11. As an example, the point cloud data generating unit 12 generates three-dimensional point cloud data from the input data acquired by the acquiring unit 11, the input data including at least one of RGB data and depth data. As an example, the point cloud data generating unit 12 generates three-dimensional point cloud data from the input data acquired by the acquiring unit 11,
Identifying the three-dimensional coordinates of each pixel included in the RGB data by referencing the depth data of each pixel in the depth data;
- The specified three-dimensional coordinates may be assigned to each pixel (each data point) to generate three-dimensional point cloud data. Here, the three-dimensional point cloud data may be configured to include a feature value of each data point in addition to the three-dimensional coordinates assigned to each data point. Here, each feature value may be configured to include at least one of the RGB value and the normal value (normal vector) of each data point. Alternatively, the point cloud data generating unit 12 may be configured to generate attribute data including the feature value of each data point (for example, at least one of the RGB value and the normal value (normal vector)) in association with the three-dimensional point cloud data including the three-dimensional coordinates of each data point.

The point cloud data generator 12 may also generate the above-mentioned three-dimensional point cloud data using algorithms such as Structure from Motion (SfM) and Simultaneous Localization and Mapping (SLAM).

The point cloud data generator 12, with the above-mentioned configuration, can generate the above-mentioned three-dimensional point cloud data from one or more frames (one or more data sets) included in the input data acquired by the acquisition unit 11.

Furthermore, when the acquisition unit 11 acquires input data including three-dimensional point cloud data, the point cloud data generation unit 12 may be configured to output the three-dimensional point cloud data as is. Alternatively, the three-dimensional point cloud data may be configured to include the feature amounts of each of the above-mentioned data points (for example, at least one of RGB values and normal values (normal vectors)) before outputting the data. Alternatively, the attribute data including the feature amounts of each of the above-mentioned data points may be output in association with the three-dimensional point cloud data including the three-dimensional coordinates of each data point.

Note that if the acquisition unit 11 acquires input data including three-dimensional point cloud data and the point cloud data generation unit 12 outputs the three-dimensional point cloud data as is, the information processing device 1 may be configured not to include the point cloud data generation unit 12. Such a configuration is also included in this exemplary embodiment.

(Thinned point cloud data generating unit 13)
The thinned point cloud data generating unit 13 generates three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data acquired by the acquiring unit 11 and the three-dimensional point cloud data generated by the point cloud data generating unit 12. Here, the thinning process may be, for example,
The thinning process includes a process of generating the three-dimensional thinned point cloud data by using only a part of the frames among the plurality of frames included in the input data. Alternatively, the thinning process includes
The configuration may include a process of generating the three-dimensional thinned point cloud data using only a portion of a plurality of data points included in at least one of the input data and the three-dimensional point cloud data.

(Learning Unit 14)
The learning unit 14 trains an estimation model to which point cloud data is input, and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data. As an example, the learning unit 14 trains the estimation model by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. Note that the expression "completion" in this exemplary embodiment is derived from "point cloud completion" as an example, but this wording does not limit this exemplary embodiment.

More specifically, as an example, the learning unit 14
Refer to training data including the three-dimensional point cloud data and the three-dimensional thinned point cloud data,
The three-dimensional thinned point cloud data included in the training data is input to the estimation model to generate three-dimensional point cloud data in the estimation model;
The estimation model is subjected to machine learning so that the difference between the 3D point cloud data generated by the estimation model and the 3D point cloud data included in the training data is reduced.

Here, the three-dimensional point cloud data included in the teacher data may include a correct answer label (also called a true value label) for segmentation, or the three-dimensional point cloud data included in the teacher data may be associated with (associated with) a correct answer label for segmentation. The learning unit 14 may be configured to train the estimation model so that the difference between the segmentation label output (estimated) by the estimation model and the correct answer label for segmentation is small. Here, the correct answer label may be, as one example, included in the input data or associated with the input data, and the acquisition unit 11 may acquire the correct answer label.

The learning unit 14 configured as described above can effectively learn the estimation model that receives point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data. Note that the term "complement" in the above description includes the complementation of so-called missing areas and the complementation of occlusion areas, but this term does not limit this exemplary embodiment.

(Effects of information processing device 1)
As described above, in the information processing device 1,
- Obtain input data,
Generate three-dimensional point cloud data from the input data;
applying a thinning process to at least one of the input data and the three-dimensional point cloud data to generate three-dimensional thinned point cloud data;
A configuration is adopted in which an estimation model that uses point cloud data as input, and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data is trained by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. According to the information processing device 1 configured as above, a thinning process is applied to at least one of the input data and the three-dimensional point cloud data to generate three-dimensional thinned point cloud data, and the estimation model is trained by referring to the three-dimensional thinned point cloud data, so that a highly accurate estimation model can be generated while suppressing an increase in cost. Therefore, according to the above configuration, a highly accurate segmentation technology can be provided while suppressing an increase in cost.

(Additional Notes Regarding Information Processing Device 1)
As described above, in the case where the acquisition unit 11 is configured to acquire three-dimensional point cloud data, the information processing device 1 may be configured not to include the point cloud data generation unit 12.
An acquisition unit 11 that acquires input data including three-dimensional point cloud data;
a thinned point cloud data generating unit 13 for generating three-dimensional thinned point cloud data by applying a thinning process to the input data;
The information processing device 1 may be configured to include a learning unit 14 that learns an estimation model that receives point cloud data as an input, and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. The information processing device 1 configured in this manner can also achieve the above-mentioned effects.

(Flow of information processing method S1)
Next, the flow of the information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1. As shown in Fig. 2, the information processing method S1 includes a step (process) S11 of acquiring input data, a step (process) S12 of generating three-dimensional point cloud data, a step (process) S13 of generating three-dimensional thinned point cloud data, and a step (process) S14 of training an estimation model.

(Step S11)
In step S11, the acquisition unit 11 acquires input data. The specific process performed by the acquisition unit 11 has been described above, and therefore will not be described here.

(Step S12)
Next, in step S12, the point cloud data generating unit 12 generates three-dimensional point cloud data from the input data acquired by the acquiring unit 11. The specific processing by the point cloud data generating unit 12 has been described above, and therefore will not be described here.

(Step S13)
Next, in step S13, the thinned point cloud data generator 13 generates three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data acquired by the acquisition unit 11 and the three-dimensional point cloud data generated by the point cloud data generator 12. The specific process by the thinned point cloud data generator 13 has been described above, and therefore will not be described here.

(Step S14)
Next, in step S14, the learning unit 14 trains an estimation model to which point cloud data is input and which outputs the complemented point cloud data and a segmentation label related to the complemented point cloud data. As an example, the learning unit 14 trains the estimation model by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. The specific processing by the learning unit 14 has been described above, and therefore will not be described here.

(Effects of information processing method S1)
As described above, in the information processing method S1,
- Obtain input data,
Generate three-dimensional point cloud data from the input data;
applying a thinning process to at least one of the input data and the three-dimensional point cloud data to generate three-dimensional thinned point cloud data;
A configuration is adopted in which an estimation model that uses point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data is trained by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. According to the information processing method S1 configured as above, a thinning process is applied to at least one of the input data and the three-dimensional point cloud data to generate three-dimensional thinned point cloud data, and the estimation model is trained by referring to the three-dimensional thinned point cloud data, so that a highly accurate estimation model can be generated while suppressing an increase in cost. Therefore, according to the above configuration, a highly accurate segmentation technology can be provided while suppressing an increase in cost.

(Additional Notes Regarding Information Processing Method S1)
In the case where the acquisition unit 11 acquires three-dimensional point cloud data in step S11, the information processing method S1 may be configured not to include step S12.
An acquisition step S11 of acquiring input data including three-dimensional point cloud data;
a thinned point cloud data generating step S13 of generating three-dimensional thinned point cloud data by applying a thinning process to the input data;
The information processing method S1 may include a learning step S14 in which an estimation model having point cloud data as input and having complemented point cloud data and segmentation labels related to the complemented point cloud data as output is learned by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. The information processing method S1 configured in this manner can also achieve the above-mentioned effects.

(Configuration of information processing device 2)
Next, the configuration of the information processing device 2 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing device 2. As shown in Fig. 3, the information processing device 2 includes an acquisition unit 21, a point cloud data generation unit 22, and an estimation unit 23.

(Acquisition unit 21)
The acquisition unit 11 acquires input data. Here, the input data is, for example, input data for an inference phase (estimation phase, test phase). In addition, a specific example of the input data acquired by the acquisition unit 21 does not limit the present exemplary embodiment, but may be, for example,
The image data may be configured to include at least one of: RGB data in which each pixel (data point) represents an RGB value; depth data in which each pixel (data point) represents a depth value; and three-dimensional point cloud data in which each data point represents a three-dimensional coordinate. In addition, the three-dimensional point cloud data may be point cloud data acquired by a LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) device, as an example, but this example does not limit the present exemplary embodiment.

In this way, the acquisition unit 21 can be configured to acquire data in the same format as the input data acquired by the acquisition unit 11 provided in the information processing device 1 described above, but the acquisition unit 21 does not need to acquire the correct answer label regarding the segmentation described in the information processing device 1.

(Point cloud data generator 22)
The point cloud data generator 12 generates three-dimensional point cloud data from the input data acquired by the acquisition unit 21 .

As an example, the point cloud data generating unit 22 generates three-dimensional point cloud data from the input data acquired by the acquiring unit 21, the input data including at least one of RGB data and depth data.
Identifying the three-dimensional coordinates of each pixel included in the RGB data by referencing the depth data of each pixel in the depth data;
- A configuration may be adopted in which three-dimensional point cloud data is generated in which the specified three-dimensional coordinates are assigned to each pixel (each data point). Here, the three-dimensional point cloud data may be configured to include at least one of the RGB values and normal values (normal vectors) of each data point in addition to the three-dimensional coordinates assigned to each data point. Alternatively, a configuration may be adopted in which attribute data including at least one of the RGB values and normal values (normal vectors) of each data point is generated in association with the three-dimensional point cloud data including the three-dimensional coordinates of each data point.

Furthermore, when the acquisition unit 21 acquires input data including three-dimensional point cloud data, the point cloud data generation unit 22 may be configured to output the three-dimensional point cloud data as is. Alternatively, the three-dimensional point cloud data may be configured to include at least one of the RGB values and normal values (normal vectors) of each of the above-mentioned data points before outputting the data. Alternatively, the point cloud data generation unit 22 may be configured to output attribute data including at least one of the RGB values and normal values (normal vectors) of each of the above-mentioned data points, in association with the three-dimensional point cloud data including the three-dimensional coordinates of each data point.

Note that if the acquisition unit 21 acquires input data including three-dimensional point cloud data and the point cloud data generation unit 22 outputs the three-dimensional point cloud data as is, the information processing device 2 may be configured not to include the point cloud data generation unit 22. Such a configuration is also included in this exemplary embodiment.

In this way, the point cloud data generator 22 can be configured to perform processing similar to that of the point cloud data generator 12 included in the information processing device 1, as an example, but this is not a limitation of this exemplary embodiment.

(Estimation unit 23)
The estimation unit 23 uses a machine-learned estimation model to estimate the complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation unit 22 and the segmentation label related to the complemented point cloud data. Here, as an example, the machine-learned estimation model may be an estimation model trained by the learning unit 14 included in the information processing device 1 described above. In other words, as an example, the machine-learned estimation model is an estimation model that receives point cloud data as an input and outputs complemented point cloud data and a segmentation label related to the complemented point cloud data, and is machine-learned with reference to the three-dimensional point cloud data and the three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

(Effects of information processing device 2)
As described above, in the information processing device 2,
- Obtain input data,
Generate three-dimensional point cloud data from the input data;
- An estimation model that uses point cloud data as input and outputs complemented point cloud data and a segmentation label related to the complemented point cloud data, and uses an estimation model that has been machine-learned with reference to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data to estimate the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and the segmentation label related to the complemented point cloud data. According to the information processing device 2 configured as above, the estimation model that has been machine-learned with reference to three-dimensional thinned point cloud data generated by applying a thinning process to the three-dimensional point cloud data is used to estimate the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and the segmentation label related to the complemented point cloud data, so that it is possible to perform an estimation process using a highly accurate estimation model while suppressing an increase in cost. Therefore, according to the above configuration, it is possible to provide a highly accurate segmentation technology while suppressing an increase in cost.

(Additional Notes Regarding Information Processing Device 2)
As described above, in the case where the acquisition unit 21 is configured to acquire three-dimensional point cloud data, the information processing device 2 may be configured not to include the point cloud data generation unit 22.
An acquisition unit 21 for acquiring input data including three-dimensional point cloud data;
The information processing device 2 may be configured to include an estimation unit 23 that uses an estimation model that is machine-learned by referring to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data, to estimate the complemented point cloud data corresponding to the three-dimensional point cloud data acquired by the acquisition unit 21 and the segmentation label related to the complemented point cloud data, using an estimation model that receives point cloud data as an input and outputs complemented point cloud data and a segmentation label related to the complemented point cloud data. The information processing device 2 configured in this manner can also achieve the above-mentioned effects.

(Flow of information processing method S2)
Next, the flow of the information processing method S2 according to this exemplary embodiment will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of the information processing method S2. As shown in Fig. 4, the information processing method S2 includes a step (process) S21 of acquiring input data, a step (process) S22 of generating three-dimensional point cloud data, and a step (process) S23 of performing estimation using an estimation model.

(Step S21)
In step S21, the acquiring unit 21 acquires input data. The specific process performed by the acquiring unit 21 has been described above, and therefore will not be described here.

(Step S22)
Next, in step S22, the point cloud data generating unit 22 generates three-dimensional point cloud data from the input data acquired by the acquiring unit 21. The specific processing by the point cloud data generating unit 22 has been described above, and therefore will not be described here.

(Step S23)
Next, in step S23, the estimation unit 23 estimates the complemented point cloud data corresponding to the three-dimensional point cloud data generated in step S22 and the segmentation label related to the complemented point cloud data by using the machine-learned estimation model. Here, as an example, the machine-learned estimation model can be an estimation model trained by the above-mentioned information processing method S1. In other words, as an example, the machine-learned estimation model is an estimation model that inputs point cloud data and outputs complemented point cloud data and a segmentation label related to the complemented point cloud data, and is machine-learned with reference to the three-dimensional point cloud data and the three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

(Effects of information processing method S2)
As described above, in the information processing method S2,
- Obtain input data,
Generate three-dimensional point cloud data from the input data;
- An estimation model that uses point cloud data as input and outputs complemented point cloud data and a segmentation label related to the complemented point cloud data, and uses an estimation model machine-learned with reference to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data to estimate the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and the segmentation label related to the complemented point cloud data. According to the information processing method S2 configured as above, the estimation model machine-learned with reference to three-dimensional thinned point cloud data generated by applying a thinning process to the three-dimensional point cloud data is used to estimate the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and the segmentation label related to the complemented point cloud data, so that it is possible to perform an estimation process using a highly accurate estimation model while suppressing an increase in cost. Therefore, according to the above configuration, it is possible to provide a highly accurate segmentation technology while suppressing an increase in cost.

(Additional Notes Regarding Information Processing Method S2)
In the case where the three-dimensional point cloud data is acquired in step S21, the information processing method S2 may be configured not to include step S22.
An acquisition step S21 of acquiring input data including three-dimensional point cloud data;
The information processing method S2 may include an estimation step S23 in which an estimation model having point cloud data as input and having complemented point cloud data and segmentation labels related to the complemented point cloud data as output is used to estimate the complemented point cloud data corresponding to the three-dimensional point cloud data acquired in the acquisition step S21 and the segmentation labels related to the complemented point cloud data, using an estimation model machine-learned with reference to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data. The information processing method S2 configured in this manner can also achieve the above-mentioned effects.

Second Embodiment
A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be given the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means adopted in this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure, as long as no particular technical hindrance occurs. In addition, each technical means shown in each drawing referred to for explaining this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure, as long as no particular technical hindrance occurs.

(Configuration of information processing device 1A)
The configuration of the information processing device 1A according to this exemplary embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 1A. As shown in Fig. 5, the information processing device 1A includes a control unit 10A, a storage unit 20A, a communication unit 30, and an input/output unit 40.

(Communication unit 30)
The communication unit 30 communicates with devices external to the information processing device 1A via a network. As an example, the communication unit 30 transmits data supplied from the control unit 10A to the external device, and supplies data received from the external device to the control unit 10A. Note that the specific configuration of the network does not limit the present exemplary embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

(Input/Output Unit 40)
The input/output unit 40 is configured to include at least one of input/output devices such as a keyboard, a mouse, a display, a printer, and a touch panel. Alternatively, the input/output unit 40 may be configured to be connected to input/output devices such as a keyboard, a mouse, a display, a printer, and a touch panel. In this configuration, the input/output unit 40 accepts input of various information from the connected input device to the information processing device 1A. Moreover, the input/output unit 40 outputs various information to the connected output device under the control of the control unit 10A. An example of the input/output unit 40 is an interface such as a Universal Serial Bus (USB).

(Memory unit 20A)
The storage unit 20A stores various data referenced by the control unit 10A and various data generated by the control unit 10A.
Input data IND
・3D point cloud data PCD
・Three-dimensional thinned point cloud data TPCD
First feature amount F1
Second feature amount F2
・Estimation model PM
・Segmentation data SG
Here, the input data IND is data acquired by an acquisition unit 11 (21) described later. Specific examples of the input data IND will be described later. The three-dimensional point cloud data PCD is data generated by a point cloud data generation unit 12 (22) described later. Specific examples of the three-dimensional point cloud data PCD will be described later. The three-dimensional thinned point cloud data TPCD is data generated by a thinned point cloud data generation unit 13 described later. Specific examples of the three-dimensional thinned point cloud data TPCD will be described later.

The first feature F1 is data referenced by the first feature selection unit 151, which will be described later. A specific example of the first feature F1 will be described later. The second feature F2 is data referenced by the second feature selection unit 152, which will be described later. A specific example of the second feature F2 will be described later.

The estimation model PM is a model for inference that is machine-learned by the learning unit 14, which will be described later, and is referenced by the estimation unit 23. The memory unit 20A stores various programs and parameters that constitute the estimation model PM. Specific examples of the estimation model PM will be described later.

(Control unit 10A)
5, the control unit 10A includes the acquisition unit 11, the point cloud data generation unit 12, the thinned point cloud data generation unit 13, the learning unit 14, and the estimation unit 23 described in the exemplary embodiment 1. Here, the acquisition unit 11 can be expressed as having the same configuration as the acquisition unit 21 described in the exemplary embodiment 1, so the acquisition unit 11 may also be referred to as the acquisition unit 11 (21). In addition, the point cloud data generation unit 12 can be expressed as having the same configuration as the point cloud data generation unit 22 described in the exemplary embodiment 1, so the point cloud data generation unit 12 may also be referred to as the point cloud data generation unit 12 (22).

The control unit 10A also includes a feature quantity selection unit 15, as shown in FIG. 5. Here, the feature quantity selection unit 15 includes a first feature quantity selection unit 151 and a second feature quantity selection unit 152, as shown in FIG. 5.

(Acquisition unit 11 (21))
The acquisition unit 11 (21) acquires input data IND. Here, the acquisition unit 11 acquires input data for learning in the learning phase, and acquires input data for inference in the inference phase (estimation phase, test phase). Although specific examples of the input data acquired by the acquisition unit 11 (21) do not limit this exemplary embodiment, as in the exemplary embodiment, as an example,
The image data may be configured to include at least one of: RGB data in which each pixel (data point) represents an RGB value; depth data in which each pixel (data point) represents a depth value; and three-dimensional point cloud data in which each data point represents a three-dimensional coordinate. In addition, the three-dimensional point cloud data may be point cloud data acquired by a LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) device, as an example, but this example does not limit the present exemplary embodiment.

In addition, as an example, the acquisition unit 11 (21) can be configured to acquire input data including correct answer labels for segmentation (input data in which correct answer labels for segmentation are attached to each data point) in the learning phase, and to acquire input data that does not include the correct answer labels in the inference phase.

The input data IND acquired by the acquisition unit 11 (21) is stored in the memory unit 20A, for example, and is referenced by the point cloud data generation unit 12 (22), the thinned point cloud data generation unit 13, the learning unit 14, the estimation unit 23, etc.

(Point cloud data generator 12 (22))
The point cloud data generating unit 12 (22) generates three-dimensional point cloud data PCD from the input data IND acquired by the acquiring unit 11 (21). As an example, the point cloud data generating unit 12 (22) generates three-dimensional point cloud data PCD from the input data IND acquired by the acquiring unit 11 (21), the input data IND including at least one of RGB data and depth data. As an example, the point cloud data generating unit 12 (22)
Identifying the three-dimensional coordinates of each pixel included in the RGB data by referencing the depth data of each pixel in the depth data;
- A configuration may be adopted in which three-dimensional point cloud data PCD is generated in which the specified three-dimensional coordinates are assigned to each pixel (each data point). Here, the three-dimensional point cloud data PCD may be configured to include at least one of the RGB values and normal values (normal vectors) of each data point in addition to the three-dimensional coordinates assigned to each data point. Alternatively, a configuration may be adopted in which attribute data including at least one of the RGB values and normal values (normal vectors) of each data point is generated in association with the three-dimensional point cloud data including the three-dimensional coordinates of each data point.

Furthermore, when the acquisition unit 11 (21) acquires input data including three-dimensional point cloud data, the point cloud data generation unit 12 (22) may be configured to output the three-dimensional point cloud data as it is as three-dimensional point cloud data PCD. Alternatively, the three-dimensional point cloud data acquired by the acquisition unit 11 (21) may be configured to include at least one of the RGB values and normal values (normal vectors) of each of the above-mentioned data points and output the data as three-dimensional point cloud data PCD. Alternatively, the attribute data including at least one of the RGB values and normal values (normal vectors) of each of the above-mentioned data points may be output in association with the three-dimensional point cloud data PCD including the three-dimensional coordinates of each data point.

In addition, when the acquisition unit 11 (21) acquires multiple frames of input data IND including at least one of RGB data, RGB depth data, and three-dimensional point cloud data, the point cloud data generation unit 12 (22) can be configured to, as an example, generate three-dimensional point cloud data PCD by referring to all frames included in the input data IND.

In addition, if the acquisition unit 11 (21) acquires input data including three-dimensional point cloud data, and the point cloud data generation unit 12 (22) outputs the three-dimensional point cloud data included in the input data as three-dimensional point cloud data PCD, the information processing device 1A may be configured not to include the point cloud data generation unit 12 (22). Such a configuration is also included in this exemplary embodiment.

(Thinned point cloud data generating unit 13)
Similarly to the first exemplary embodiment, the thinned point cloud data generating unit 13 applies a thinning process to at least one of the input data IND acquired by the acquiring unit 11 (21) and the three-dimensional point cloud data PCD generated by the point cloud data generating unit 12 (22) to generate three-dimensional thinned point cloud data TPCD. Here, the thinning process may be, for example,
The process includes a process of generating the three-dimensional thinned point cloud data TPCD using only some of the frames included in the input data IND. More specifically, the thinning process may include, as an example, a process of generating the three-dimensional thinned point cloud data TPCD using only N % (N is a real number less than 100) of the frames included in the input data IND.
Here, the thinned point cloud data generating unit 13 performs the process of selecting the N % of frames as follows:
Randomly select the frames to be removed. A process may be performed in which a group of consecutive frames are removed in such a way that a specific area is missing from the image or object represented by the input data IND.

Alternatively, the thinning process may include:
The configuration may include a process of generating the three-dimensional thinned point cloud data TPCD by using only some of the data points included in at least one of the input data IND and the three-dimensional point cloud data PCD. More specifically, the thinning process may include:
The configuration may include at least one of the following: a process for reducing the resolution of a depth image contained in the input data IND; and a process for applying a thinning method to the three-dimensional point cloud data contained in the input data IND so as to reduce the density of the light receiving sensors of the LiDAR device.

(Learning Unit 14)
The learning unit 14, similarly to the exemplary embodiment 1, trains an estimation model PM to which the point cloud data IND is input and which outputs complemented point cloud data and a segmentation label related to the complemented point cloud data. As an example, the learning unit 14 trains the estimation model PM by referring to the three-dimensional point cloud data PCD and the three-dimensional thinned point cloud data TPCD.

More specifically, as an example, the learning unit 14
Refer to teacher data TD including the three-dimensional point cloud data PCD1 and three-dimensional thinned point cloud data TPCD1 obtained by thinning the three-dimensional point cloud data PCD1,
The three-dimensional thinned point cloud data TPCD1 included in the teacher data TD is input to the estimation model PM, thereby causing the estimation model PM to generate three-dimensional point cloud data PCD2 (which may also be referred to as complemented three-dimensional point cloud data IPCD1);
- The estimation model PM is subjected to machine learning so that the difference between the three-dimensional point cloud data PCD2 generated by the estimation model PM and the three-dimensional point cloud data PCD1 included in the training data TD is reduced.

Here, the three-dimensional point cloud data PCD1 included in the teacher data TD may include a correct answer label GTL1 for segmentation, or the three-dimensional point cloud data PCD1 included in the teacher data TD may be configured to accompany (associate) the correct answer label GTL1 for segmentation. The learning unit 14 may be configured to train the estimation model PM so that the difference between the segmentation label PL1 output (estimated) by the estimation model PM and the correct answer label GTL1 for segmentation is small. Here, the correct answer label GTL1 may be, as an example, included in the input data IND or may be configured to accompany (associate) with the input data IND, and the acquisition unit 11 may be configured to acquire the correct answer label GTL1.

The learning unit 14 may also be configured to learn the estimation model PM by further referring to a first feature value F1 associated with at least one of the input data IND, the three-dimensional point cloud data PCD, and the three-dimensional thinned point cloud data TPCD.

Here, a specific example of the first feature amount F1 does not limit the present exemplary embodiment. However, as an example, the first feature amount F1 may be
At least one of the RGB values and normal values (normal vectors) of each data point included in (or associated with) the input data IND or the three-dimensional point cloud data PCD can be used. A specific method of training the estimation model PM with reference to the first feature amount F1 will be described later.

In the present exemplary embodiment, the estimation model PM is, for example,
A first estimation model PM1 that receives point cloud data and outputs complemented point cloud data;
and a second estimation model PM2 that receives at least the interpolated point cloud data and outputs a segmentation label related to the interpolated point cloud data. A more specific configuration example of the estimation model PM will be described later with reference to the drawings.

The learning unit 14 configured as described above can effectively train the estimation model PM, which receives point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data.

Furthermore, with the learning unit 14 configured as described above, the estimation model PM is learned by referring to the three-dimensional thinned point cloud data TPCD generated by applying a thinning process to at least one of the input data IND and the three-dimensional point cloud data PCD, so that it is possible to generate a highly accurate estimation model while suppressing increases in cost. Therefore, with the above configuration, it is possible to provide a highly accurate segmentation technique while suppressing increases in cost.

(Estimation unit 23)
The estimation unit 23, similarly to the exemplary embodiment 2, estimates the complemented point cloud data corresponding to the three-dimensional point cloud data PCD generated by the point cloud data generation unit 12 (22) and the segmentation label related to the complemented point cloud data by using a machine-learned estimation model. Here, as an example, the machine-learned estimation model can use the estimation model PM trained by the learning unit 14 described above. In other words, as an example, the machine-learned estimation model is an estimation model that inputs point cloud data and outputs complemented point cloud data and a segmentation label related to the complemented point cloud data, and is the estimation model PM trained by machine learning with reference to the three-dimensional point cloud data and the three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

The estimation unit 23 may also be configured to further refer to a second feature F2 associated with at least one of the input data IND and the three-dimensional point cloud data PCD generated by the point cloud data generation unit 12 (22) to estimate the complemented point cloud data corresponding to the three-dimensional point cloud data PCD generated by the point cloud data generation unit 12 (22) and the segmentation label related to the complemented point cloud data.

Here, a specific example of the second feature amount F2 does not limit the present exemplary embodiment, but as an example, the second feature amount F2 may be
At least one of the RGB values and normal values (normal vectors) of each data point included in (or associated with) the input data IND or the three-dimensional point cloud data PCD can be used. The type of the second feature F2 (e.g., RGB value or normal value) can be the same as the type of the first feature F1 described above, for example. A specific method of estimation with reference to the second feature F2 will be described later.

Furthermore, data including the "complemented point cloud data" and "segmentation labels related to the complemented point cloud data" estimated (generated) by the estimation unit 23 is stored in the storage unit 20A as the segmentation data SD shown in FIG. 5, for example, and is provided to the outside of the information processing device 100A via the communication unit 30 and the input/output unit 40, etc. In this exemplary embodiment, the segmentation data SD may also be expressed as a segmentation image, a segmentation result, etc.

As an example, the estimation unit 23 also functions as a presentation means that presents a segmentation image including objects having a display color and a display texture corresponding to the segmentation label estimated by the estimation unit 23 to the user via a display provided in the input/output unit 40.

The estimation unit 23 configured as described above performs estimation processing using the estimation model PM that has been trained with reference to the three-dimensional thinned point cloud data TPCD that has been generated by applying a thinning process to at least one of the input data IND and the three-dimensional point cloud data PCD, making it possible to perform highly accurate estimation while suppressing increases in cost. Therefore, with the above configuration, it is possible to provide a highly accurate segmentation technique while suppressing increases in cost.

(Feature quantity selection unit 15)
5, the feature quantity selection unit 15 includes a first feature quantity selection unit 151 and a second feature quantity selection unit 152. As an example, the first feature quantity selection unit 151 selects a first feature quantity F1 to be referred to when the learning unit 14 learns the estimation model PM. On the other hand, as an example, the second feature quantity selection unit 152 selects a second feature quantity F2 to be referred to when the estimation unit 23 performs estimation processing using the estimation model PM.

Specific examples of the first feature amount F1 and the second feature amount F2 include:
At least one of the RGB values and normal values (normal vectors) of each data point included in (or associated with) the input data IND or the three-dimensional point cloud data PCD. Specific examples of the processes performed by the first feature amount selection unit 151 and the second feature amount selection unit 152 will be described later with reference to different drawings.

(Data flow during the learning phase)
Next, the flow of data in the learning phase of the information processing device 1A will be described with reference to FIGS.

FIG. 6 is a diagram for explaining an example of the flow of data in the learning phase of the information processing device 1A. As shown in FIG. 6, first, the acquisition unit 11 acquires input data IND. Here, the input data IND is, as an example, input data for learning, and each data point included in the input data is assigned, as an example, a correct answer label GTL1 for segmentation. Then, the point cloud data generation unit 12 generates three-dimensional point cloud data PCD by referring to the input data IND. Here, the generated three-dimensional point cloud data PCD is supplied to the thinned point cloud data generation unit 13 and the first estimation model learning unit 141 provided in the learning unit 14.

On the other hand, as an example, the thinned point cloud data generation unit 13 generates three-dimensional thinned point cloud data TPCD from the three-dimensional point cloud data PCD, and supplies the generated three-dimensional thinned point cloud data TPCD to the first estimation model learning unit 141. Here, the first estimation model learning unit 141 is configured to learn the first estimation model PM1 included in the estimation model PM.

The thinned point cloud data generator 13 also associates at least one of the RGB values and normal values (normal vectors) of each data point included in (or associated with) the input data IND or the three-dimensional point cloud data PCD with the coordinates of the data point and stores them in the memory unit 20A as a first feature F1.

The first estimation model learning unit 141 inputs the three-dimensional thinned point cloud data TPCD1 generated by the thinned point cloud data generation unit 13 to the first estimation model PM1, and obtains the complemented three-dimensional point cloud data IPCD1 output by the first estimation model PM1. The first estimation model learning unit 141 then supplies the complemented three-dimensional point cloud data IPCD1 to the second estimation model learning unit 142.

The first estimation model learning unit 141 also calculates a first loss value (first loss function) (also denoted as BCE_Loss) indicating the difference between the above-mentioned complemented three-dimensional point cloud data IPCD1 and the three-dimensional point cloud data PCD generated by the point cloud data generation unit 12.

In addition, the first estimation model learning unit 141 supplies the coordinates of each data point included in the complemented three-dimensional point cloud data IPCD1 to the first feature quantity selection unit 151. Then, the first feature quantity selection unit 151
Compare the coordinates associated with the first feature amount F1 stored in the storage unit 20A with the coordinates of each data point included in the interpolated three-dimensional point cloud data IPCD1;
Identifying the first feature values F1 corresponding to each data point included in the interpolated three-dimensional point cloud data IPCD1 from among the first feature values F1 stored in the storage unit 20A;
The identified first feature amount F1 is supplied to the second estimation model learning unit 142.

The second estimation model learning unit 142
- the complemented three-dimensional point cloud data IPCD1 calculated by the first estimation model learning unit 141 using the first estimation model PM1; and
The first feature F1 supplied from the first feature selection unit 151 is input to the second estimation model PM2, and an estimated label PL1 relating to the segmentation output by the second estimation model PM2 is obtained.

The second estimation model learning unit 142 also calculates a second loss value (second loss function) (also indicated as CE_Loss) indicating the difference between the estimated label PL1 and, as an example, the correct label GTL1 for the segmentation acquired by the acquisition unit 11.

Then, the learning unit 14 linearly calculates the sum of the first loss value BCE_Loss and the second loss value CE_Loss as follows: Loss_total = CE_Loss + α × BCE_Loss
The total loss value Loss_total is calculated by the above formula, and the parameters of the first estimation model PM1 and the second estimation model PM2 are updated so that the loss value Loss_total becomes smaller, where α denotes a weighting coefficient that can be appropriately set.

In this way, the learning unit 14 refers to the integrated loss value Loss_total and updates (trains) the first estimation model PM1 and the second estimation model PM2, thereby suitably improving the accuracy of the complementation process and the accuracy of the segmentation process. Note that updating (training) the first estimation model PM1 and the second estimation model PM2 in this way by referring to the integrated loss value Loss_total can also be expressed as "simultaneously training the first estimation model PM1 and the second estimation model PM2."

In this way, the learning unit 14 trains a first estimation model PM that receives point cloud data as input and outputs complemented point cloud data, and a second estimation model PM2 that receives at least the complemented point cloud data as input and outputs segmentation labels related to the complemented point cloud data.

FIG. 7 is a diagram for explaining an example of the flow of data regarding the network in the learning phase. As shown in FIG. 7, at least one of the acquisition unit 11 and the point cloud data generation unit 12 acquires teacher data TD including three-dimensional point cloud data PCD1 and ground truth labels GTL1 (also indicated as true value labels GTL1 in FIG. 7) assigned to each data point of the three-dimensional point cloud data PCD1.

Next, the three-dimensional point cloud data BPCD1 from which attribute data such as RGB values and normal values have been removed is input to the thinned point cloud data generator 13, and the thinned point cloud data generator 13 generates three-dimensional thinned point cloud data BTPCD1. The three-dimensional thinned point cloud data BTPCD1 is then input to the first estimation model PM1 (also referred to as the complementation network in FIG. 7).

Then, the first estimation model PM1 outputs the complemented three-dimensional point cloud data IPCD1. The complemented three-dimensional point cloud data IPCD1 is input to the second estimation model PM2 (also referred to as semantic segmentation network in FIG. 7) together with the first feature F1 (referred to as first feature data F1 in FIG. 7). Here, the first feature F1 is the first feature F1 selected by the first feature selection unit 151 described above. The first feature F1 is input to the second estimation model PM2 in the form of a mask feature, for example. Here, the mask feature refers to a data group in which the data points at which the first feature F1 selected by the first feature selection unit 151 exists have the value of the feature, and the other data points are assigned a null feature (a predetermined value such as 0, 255, -1, etc.).

In addition, the difference between the complemented three-dimensional point cloud data IPCD1 and the above-mentioned three-dimensional point cloud data BPCD1 is calculated as a first loss value (BCE_Loss).

Then, the second estimation model PM2 outputs an estimated label PL1 for the segmentation of each data point of the complemented 3D point cloud data IPCD1. In addition, the difference between the estimated label PL1 and the ground truth label GTL1 is calculated as a second loss value (CE_Loss).

As described above, the learning unit 14 updates the parameters of the first estimation model PM1 and the second estimation model PM2 so that the total loss value represented by the first loss value and the second loss value becomes smaller.

(Data flow in the estimation phase)
Next, the flow of data in the estimation phase of the information processing device 1A will be described with reference to FIGS.

FIG. 8 is a diagram for explaining an example of the flow of data in the estimation phase of the information processing device 1A. As shown in FIG. 8, first, the acquisition unit 21 acquires input data IND. Here, the input data IND is, as an example, input data for estimation, and unlike the input data for learning described above, each data point included in the input data is not assigned a correct answer label GTL1 for segmentation. Then, the point cloud data generation unit 22 generates three-dimensional point cloud data PCD by referring to the input data IND. Here, the generated three-dimensional point cloud data PCD is supplied to a first estimation model PM1 provided in the estimation unit 23.

The point cloud data generator 22 also associates at least one of the RGB values and normal values (normal vectors) of each data point included in (or associated with) the input data IND or the three-dimensional point cloud data PCD with the coordinates of the data point, and stores them in the memory unit 20A as a second feature F2.

The estimation unit 23 inputs the three-dimensional point cloud data PCD generated by the point cloud data generation unit 22 into the first estimation model PM1. Here, the first estimation model PM1 is, as an example, the first estimation model PM1 that has been machine-learned by the first estimation model learning unit 141 described above in the learning phase.

The first estimation model PM1 to which the above-mentioned three-dimensional point cloud data PCD is input outputs the complemented three-dimensional point cloud data IPCD2. The estimation unit 23 also supplies the coordinates of each data point included in the complemented three-dimensional point cloud data IPCD2 to the second feature quantity selection unit 152. The second feature quantity selection unit 152 then:
Compare the coordinates associated with the second feature amount F2 stored in the storage unit 20A with the coordinates of each data point included in the interpolated three-dimensional point cloud data IPCD2,
Identifying the second feature amounts F2 corresponding to each data point included in the interpolated three-dimensional point cloud data IPCD2 from among the second feature amounts F2 stored in the storage unit 20A;
The identified second feature F2 is supplied to a second estimation model PM2.

The estimation unit 23
- Interpolated three-dimensional point cloud data IPCD2 calculated using the first estimation model PM1; and
- The second feature F2 supplied from the second feature selection unit 152 is input into the second estimation model PM2, and an estimated label PL2 regarding the segmentation output by the second estimation model PM2 (in other words, a segmentation label regarding the complemented 3D point cloud data IPCD2) is obtained.

Then, data including the "complemented three-dimensional point cloud data IPCD2" and "segmentation labels related to the complemented three-dimensional point cloud data IPCD2" estimated (generated) by the estimation unit 23 is stored in the storage unit 20A as the segmentation data SD shown in FIG. 5, for example, and is provided to the outside of the information processing device 100A via the communication unit 30 and the input/output unit 40, etc.

FIG. 9 is a diagram for explaining an example of the flow of data related to the network in the estimation phase. As shown in FIG. 9, three-dimensional point cloud data PCD2 is acquired by at least one of the acquisition unit 21 and the point cloud data generation unit 22.

Next, the three-dimensional point cloud data BPCD2 from which attribute data such as RGB values and normal values have been removed is input to the first estimation model PM1 (also referred to as the complementation network in FIG. 9).

Then, the first estimation model PM1 outputs the complemented three-dimensional point cloud data IPCD2. The complemented three-dimensional point cloud data IPCD2 is input to the second estimation model PM2 (also referred to as the semantic segmentation network in FIG. 7) together with the second feature F2 (referred to as the second feature data F2 in FIG. 9). Here, the second feature F2 is the second feature F2 selected by the second feature selection unit 152 described above. The second feature F2 is input to the second estimation model PM2 in the form of a mask feature, for example. Here, the mask feature refers to a data group in which the data points at which the second feature F2 selected by the second feature selection unit 152 exists have the value of the feature, and the other data points are assigned a null feature (a predetermined value such as 0, 255, -1, etc.).

Then, the second estimation model PM2 outputs estimated labels PL2 for the segmentation of each data point of the interpolated 3D point cloud data IPCD2.

(Effects of information processing device 1A)
As described above, in the learning phase, in the information processing device 1A,
- Obtain input data,
Generate three-dimensional point cloud data from the input data;
applying a thinning process to at least one of the input data and the three-dimensional point cloud data to generate three-dimensional thinned point cloud data;
A configuration is adopted in which an estimation model that takes point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data is trained by referring to the 3D point cloud data and the 3D thinned point cloud data.

In addition, in the information processing device 1A, in the estimation phase,
- Obtain input data,
Generate three-dimensional point cloud data from the input data;
An estimation model that takes point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, and is configured to estimate complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and segmentation labels related to the complemented point cloud data using an estimation model that has been machine-learned by referring to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

In the information processing device 1A configured as described above, in the learning phase, the estimation model is trained by referring to three-dimensional thinned point cloud data generated by applying a thinning process to three-dimensional point cloud data. Then, in the estimation phase, the machine-learned estimation model is used to estimate complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and segmentation labels related to the complemented point cloud data. Therefore, with the above configuration, it is possible to perform estimation processing using a highly accurate estimation model while suppressing increases in costs.

Furthermore, with the information processing device 1A configured as described above, in the learning phase, the estimation model PM is trained by further referring to a first feature associated with at least one of the input data, the three-dimensional point cloud data, and the three-dimensional thinned point cloud data, and in the estimation phase, estimation processing is performed using the estimation model PM by further referring to a second feature associated with at least one of the input data and the three-dimensional point cloud data. Therefore, with the above configuration, it is possible to perform estimation processing using a more accurate estimation model while suppressing increases in costs.

(Additional Notes Regarding Exemplary Embodiment 2)
In the above description, a configuration in which the first estimation model PM1 and the second estimation model PM2 constituting the estimation model PM are in series has been taken as an example, but this example does not limit the present exemplary embodiment. For example, a configuration in which the first estimation model PM1 and the second estimation model PM2 are in parallel or partially in parallel may be adopted, and the first estimation model PM1 and the second estimation model PM2 may be simultaneously trained. As an example, the estimation model PM is
- a first network;
A configuration may be adopted in which the first network is branched off from a second network, the above-mentioned loss value CE_Loss (or BCE_Loss) is calculated by referring to the output of the first network, the above-mentioned loss value BCE_Loss (or CE_Loss) is calculated by referring to the output of the second network, and the parameters of the first network and the second network are updated by referring to a total loss value including these loss values. With such a configuration, various effects according to this exemplary embodiment can be obtained. The same applies to other exemplary embodiments described in this specification.

Third Embodiment
A third exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be given the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means adopted in this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure to the extent that no particular technical hindrance occurs. In addition, each technical means shown in each drawing referred to for explaining this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure to the extent that no particular technical hindrance occurs.

(Configuration of information processing device 1B)
The configuration of the information processing device 1B according to this exemplary embodiment will be described with reference to Fig. 10. Fig. 10 is a block diagram showing the configuration of the information processing device 1B. As shown in Fig. 10, the information processing device 1B includes a projection unit 16 in addition to the units included in the information processing device 1A according to the second exemplary embodiment.

In addition, the storage unit 20B of the information processing device 1B stores projected point cloud data PPCD (also referred to as post-projection point cloud data PPCD) in addition to the data stored in the storage unit 20A of the information processing device 1A according to the second exemplary embodiment. The following explanation will focus on the differences from the information processing device 1A.

(Projection unit 16)
The projection unit 16 generates projected point cloud data PPCD by projecting the complemented point cloud data generated by the estimation model PM onto a two-dimensional plane. Here, the two-dimensional plane is, for example, a two-dimensional plane corresponding to a projection onto a camera position (viewpoint) that was not used when the thinned point cloud data generation unit 13 generated the three-dimensional thinned point cloud data TPCD in the learning phase.

In other words, the projection unit 16 may be expressed as generating the projected point cloud data PPCD by projecting the complemented point cloud data generated by the estimation model PM from a camera position (viewpoint) that was not used when the thinned point cloud data generation unit 13 generated the three-dimensional thinned point cloud data TPCD in the learning phase.

However, the above example does not limit the present exemplary embodiment, and the projection unit 16 projects the complemented point cloud data generated by the estimation model PM as follows:
A position (viewpoint) different from the position (viewpoint) of an imaging device (including LiDAR) when the imaging device acquires at least one of the RGB values, depth values, and 3D point cloud data included in the input data IND.
Alternatively, the projection unit 16 may be configured to project the interpolated point cloud data generated by the estimation model PM onto the
It may also be expressed as a configuration in which the projected point cloud data PPCD is generated by projecting to a viewpoint different from the viewpoint for at least any of the RGB values, depth values, and three-dimensional point cloud data contained in the input data IND that the point cloud data generation unit 12 (22) refers to when generating the three-dimensional point cloud data PCD.

(Learning Unit 14)
The learning unit 14 according to this exemplary embodiment learns the estimation model by further referring to the projected point cloud data PPCD and the segmentation ground truth label GTL corresponding to the projected point cloud data.

As an example, the learning unit 14 calculates a third loss value (third loss function) CE_2D_Loss indicating the difference between the estimated label estimated by the estimation model PM for the projected point cloud data PPCD and the segmentation correct label GTL acquired by the acquisition unit 11. Then, the learning unit 14 calculates a total loss value (Loss_total) using a linear sum of the first loss value BCE_Loss, the second loss value CE_Loss, and the third loss value CE_2D_Loss described in the exemplary embodiment 2 as follows: Loss_total = CE_Loss + α × BCE_Loss + β × CE_2D_Loss
The learning unit 14 then updates the parameters of the first estimation model PM1 and the second estimation model PM2 so that the total loss value (Loss_total) is smaller. Here, α and β indicate weighting coefficients that can be appropriately set. The third loss value CE_2D_Loss may be calculated by the projection unit 16.

(Data flow during the learning phase)
Next, the flow of data in the learning phase of the information processing device 1B will be described with reference to FIG.

11 is a diagram for explaining an example of the data flow in the learning phase of the information processing device 1B. As shown in FIG. 11, in the learning phase of the information processing device 1B, in addition to the data flow in the learning phase of the information processing device 1A described with reference to FIG.
The segmentation correct label GTL is supplied from the acquisition unit 11 to the projection unit 16,
The first estimation model learning unit 141 supplies the complemented three-dimensional point cloud data IPCD generated by the first estimation model PM1,
The second estimation model learning unit 142 supplies the estimated labels PL, which are estimated by the second estimation model PM2 and relate to each data point of the three-dimensional point cloud data PPCD after projection by the projection unit 16.

Then, the projection unit 16 calculates a third loss value CE_2D_Loss indicating the difference between the segmentation ground truth label GTL and the estimated label PL, and supplies it to the learning unit 14. The learning unit 14 then calculates a total loss value (Loss_total) using the linear sum of the first loss value BCE_Loss, the second loss value CE_Loss, and the third loss value CE_2D_Loss, as described above. The learning unit 14 then updates the parameters of the first estimation model PM1 and the second estimation model PM2 so that the value of the total loss value (Loss_total) becomes smaller.

(Effects of information processing device 1B)
According to the information processing device 1B configured as above,
The interpolated point cloud data IPCD generated by the estimation model PM is projected onto a two-dimensional plane to generate projected point cloud data PPCD;
The learning unit 14 is configured to learn the estimation model PM by further referring to the projected point cloud data PPCD and the segmentation ground truth label GTL corresponding to the projected point cloud data PPCD. According to this configuration, a more suitable loss function can be configured by referring to the projected point cloud data PPCD, so that an estimation model PM with higher estimation accuracy can be generated in the learning phase. Also, in the estimation phase, highly accurate estimation can be performed using such an estimation model PM.

Fourth embodiment
A fourth exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be given the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means adopted in this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure to the extent that no particular technical hindrance occurs. In addition, each technical means shown in each drawing referred to for explaining this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure to the extent that no particular technical hindrance occurs.

(Configuration of information processing device 1C)
The configuration of the information processing device 1C according to this exemplary embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing the configuration of the information processing device 1C. As shown in Fig. 12, a control unit 10C included in the information processing device 1C has the same configuration as the control unit 10A included in the information processing device 1A. Meanwhile, a storage unit 20C included in the information processing device 1C stores thinning information TI in addition to the data stored in the storage unit 20A included in the information processing device 1A according to the second exemplary embodiment.

The thinning information TI here is information that indicates what type of thinning process was performed when the thinned point cloud data generation unit 13 executed the thinning process. The following explanation will focus on the points that are different from the information processing device 1A.

(Thinning-out information TI)
The thinning information TI is information indicating what type of thinning processing has been performed when the thinned point cloud data generating unit 13 executes the thinning processing. Fig. 13 is a diagram for explaining an example of the thinning information TI.

The upper part of FIG. 13 shows a schematic representation of multiple frames included in the input data IND acquired by the acquisition unit 11 during the learning phase. In the example shown in the upper part of FIG. 13, the frame numbers increase as you move to the right, for example.

The lower left side of Fig. 13 shows thinning pattern 1 in the thinning process by the thinned point cloud data generation unit 13. Here, the shaded frames are frames that are referenced by the thinned point cloud data generation unit 13 when generating the three-dimensional thinned point cloud data TPCD, and the non-shaded frames (white frames in the lower left side of Fig. 13) are frames that are not referenced when generating the three-dimensional thinned point cloud data TPCD. In other words, the non-shaded frames are frames that are thinned out when generating the three-dimensional thinned point cloud data TPCD. In thinning pattern 1, in certain 10 consecutive frames,
- Refer to two consecutive frames,
- The next two consecutive frames are thinned out. - The next two consecutive frames are referenced.
The next two consecutive frames are thinned out, and the next two consecutive frames are referenced.

On the other hand, the lower right side of Fig. 13 shows thinning pattern 2 in the thinning process by the thinned point cloud data generating unit 13. In thinning pattern 2, in certain consecutive 10 frames,
- Refer to three consecutive frames,
- The next two consecutive frames are thinned out. - The next two consecutive frames are referenced.
The next two consecutive frames are then thinned out, and the next frame is referenced.

Then, the thinned point cloud data generation unit 13 stores, as an example, a frame index indicating the thinned frame, as thinning information TI in the storage unit 20C. By referencing such thinning information TI, the thinned point cloud data generation unit 13 generates multiple mutually different three-dimensional thinned point cloud data TPCD using multiple thinning patterns for the same input data IND or the same three-dimensional point cloud data PCD.

In this way, the information processing device 1C according to this exemplary embodiment stores the thinned frame index when generating the three-dimensional thinned point cloud data TPCD in the thinned point cloud data generating unit 13, thereby creating multiple three-dimensional thinned point cloud data TPCD with different thinning methods from a single data input, thereby strengthening the data used for learning in the learning phase.

(Effects of information processing device 1C)
According to the information processing device 1B configured as above, since the estimation model PM can be trained using a larger amount of data in the learning phase, it is possible to generate a more accurate estimation model PM. Furthermore, it is possible to perform more accurate estimation using such an estimation model PM.

Fifth embodiment
A fifth exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be given the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means adopted in this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure, as long as no particular technical hindrance occurs. In addition, each technical means shown in each drawing referred to for explaining this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure, as long as no particular technical hindrance occurs.

(Configuration of information processing device 1D)
The configuration of the information processing device 1D according to this exemplary embodiment will be described with reference to Fig. 14. Fig. 14 is a block diagram showing the configuration of the information processing device 1D. As shown in Fig. 14, the information processing device 1D has the following configurations (or data) among the units included in the information processing devices 1A, 1B, and 1C according to exemplary embodiments 2 to 4, and other configurations are not essential.
- In the control unit 10D, an acquisition unit 21, a point cloud data generation unit 22, an estimation unit 23, a second feature selection unit 152, and - In the memory unit 20D, input data IND, three-dimensional point cloud data PCD, a second feature F2, an estimation model PM, and segmentation data SG.

In other words, the information processing device 1D has a configuration for performing processing of the estimation phase using the estimation model PM, but does not have a configuration for learning the estimation model PM. As an example, the estimation model PM used by the information processing device 1D for the estimation processing may be the estimation model PM learned by the information processing devices 1A, 1B, and 1C according to exemplary embodiments 2 to 4.

The information processing device 1D configured as described above uses the estimation model PM, which has been machine-learned with reference to three-dimensional thinned point cloud data generated by applying a thinning process to three-dimensional point cloud data, to estimate complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and segmentation labels related to the complemented point cloud data. This makes it possible to perform estimation processing using a highly accurate estimation model while suppressing increases in costs. Therefore, with the above configuration, it is possible to provide a highly accurate segmentation technology while suppressing increases in costs.

(Examples of application)
In the following, application examples of the information processing devices 1, 1A, 1B, 1C, and 1D according to the above-described exemplary embodiments will be described.

The information processing devices 1, 1A, 1B, 1C, and 1D (hereinafter simply referred to as information processing device 1, etc.) according to the exemplary embodiments described above can also be applied to the medical and healthcare fields, for example. In this field, the technology of the present application can be used for medical applications by performing segmentation based on medical images of patients.

When the information processing device 1 or the like is applied to the medical/healthcare field, for example, processing may be performed according to the processing flow shown below.

(Step S101: Scan step)
A medical professional such as a doctor or medical staff uses an imaging device (endoscope, fMRI, etc.) to capture images of a patient's organs (stomach, intestines, etc.) and generate a medical image. Then, the medical image is input to an information processing device 1, etc. as input data IND.

(Step S102: 3D modeling step)
Next, the point cloud data generating unit 12 (22) included in the information processing device 1 or the like refers to the input data IND and generates three-dimensional point cloud data PCD corresponding to the input data IND. Here, the three-dimensional point cloud data PCD generated by the point cloud data generating unit 12 (22) may be configured to be presented to medical professionals, medical staff, patients, etc. via a display or the like included in the input/output unit 40, as an example. Note that this step can be applied to both the learning phase and the estimation phase.

(Step S103A: decision making step)
Next, the estimation unit 23 included in the information processing device 1 or the like performs semantic segmentation on the medical image using the machine-learned estimation model PM, and outputs the segmentation result. Here, the semantic segmentation performs segmentation (area classification) into, for example, lesion areas (inflammation, ulcer, polyp), normal areas, sites, and the like. This allows medical personnel to, for example, make a treatment plan. Therefore, the present device can support medical personnel in making diagnostic decisions.

(Step S103B: decision making step)
Instead of the process of step S103A, or together with step S103A, the information processing device 1 or the like may perform the following process. That is, the estimation unit 23 of the information processing device 1 or the like outputs the three-dimensional point cloud data IPCD (or a 3D model constructed based on the three-dimensional point cloud data IPCD whose defects have been completed) complemented by the first estimation model PM1 (for example, whose defects have been completed). The output process may be performed, for example, in the form of presentation to a medical professional via a display or the like provided in the input/output unit 40. By referring to the complemented three-dimensional point cloud data (3D model), the medical professional can understand, for example, the state of the patient's organs. Therefore, the present device can support the medical professional in making a diagnosis.

In this manner, in this application example, the acquisition unit 11 (21) of the information processing device 1 or the like acquires medical images as the input data, and the estimation unit 23 functions as a presentation means for presenting the segmentation results to assist medical professionals in making decisions.

[Software implementation example]
Some or all of the functions of the information processing devices 1, 1A, 1B, 1C, and 1D (hereinafter also referred to as "each of the above devices") may be realized by hardware such as an integrated circuit (IC chip), or by software.

In the latter case, each of the above devices is realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 15. FIG. 15 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

Computer C has at least one processor C1 and at least one memory C2. Memory C2 stores a program P for operating computer C as each of the above devices. In computer C, processor C1 reads and executes program P from memory C2, thereby realizing each function of each of the above devices.

The processor C1 may be, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating point number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination of these.

Computer C may further include a RAM (Random Access Memory) for expanding program P during execution and for temporarily storing various data. Computer C may further include a communications interface for sending and receiving data to and from other devices. Computer C may further include an input/output interface for connecting input/output devices such as a keyboard, mouse, display, and printer.

The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communications network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.

[Appendix A]
This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

(Appendix A1)
An acquisition means for acquiring input data;
a point cloud data generating means for generating three-dimensional point cloud data from the input data;
a thinned point cloud data generating means for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data;
and a learning means for learning an estimation model that receives point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

(Appendix A2)
The thinning process by the thinned point cloud data generating means includes the following steps:
The information processing device according to claim A1, further comprising a process of generating the three-dimensional thinned point cloud data using only a portion of a plurality of frames included in the input data.

(Appendix A3)
The thinning process by the thinned point cloud data generating means includes the following steps:
The information processing device described in Appendix A1 or A2 includes a process of generating the three-dimensional thinned point cloud data using only a portion of the multiple data points included in at least one of the input data and the three-dimensional point cloud data.

(Appendix A4)
The information processing device according to claim 1, wherein the learning means learns the estimation model by further referring to a feature amount associated with the three-dimensional thinned point cloud data.

(Appendix A5)
The estimation model is
a first estimation model that receives point cloud data and outputs interpolated point cloud data;
and a second estimation model that receives at least the complemented point cloud data and outputs a segmentation label related to the complemented point cloud data.

(Appendix A6)
a projection unit for projecting the interpolated point cloud data generated by the estimation model onto a two-dimensional surface to generate projected point cloud data;
The information processing device according to any one of Appendices A1 to A5, wherein the learning means further refers to the projected point cloud data and a segmentation answer label corresponding to the projected point cloud data to learn the estimation model.

(Appendix A7)
The information processing device according to any one of appendices A1 to A6, wherein the thinned point cloud data generation means generates a plurality of mutually different three-dimensional thinned point cloud data for the same input data or the same three-dimensional point cloud data.

(Appendix A8)
An acquisition means for acquiring input data;
a point cloud data generating means for generating three-dimensional point cloud data from the input data;
An information processing device comprising: an estimation means for estimating, using an estimation model trained by machine learning with reference to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data, the complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means and the segmentation label related to the complemented point cloud data, the estimation model having point cloud data as input and complemented point cloud data and segmentation labels related to the complemented point cloud data as output.

(Appendix A9)
The information processing device according to claim A8, further comprising a presentation means for presenting a segmentation result indicated by the segmentation label estimated by the estimation means.

(Appendix A10)
The acquisition means acquires a medical image as the input data,
The information processing device according to claim 9, wherein the presentation means presents the segmentation result to assist a medical professional in making a decision.

(Appendix A11)
The information processing device according to any one of Appendices A8 to A10, wherein the estimation means further refers to features accompanying the three-dimensional point cloud data generated by the point cloud data generation means, and estimates complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means and a segmentation label related to the complemented point cloud data.

(Appendix A12)
The estimation model is
A first estimation model that receives point cloud data as input and outputs interpolated point cloud data;
and a second estimation model that receives at least the complemented point cloud data and outputs a segmentation label related to the complemented point cloud data.

[Appendix B]
This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

(Appendix B1)
An acquisition step of acquiring input data;
a point cloud data generation step of generating three-dimensional point cloud data from the input data;
a thinned point cloud data generating step of generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data;
an information processing method including a learning process for learning an estimation model that receives point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

(Appendix B2)
The thinning process in the thinned point cloud data generation step includes the following steps:
The information processing method according to claim B1, further comprising a process of generating the three-dimensional thinned point cloud data using only a portion of a plurality of frames included in the input data.

(Appendix B3)
The thinning process in the thinned point cloud data generation step includes the following steps:
The information processing method described in Appendix B1 or B2 includes a process of generating the three-dimensional thinned point cloud data using only a portion of the multiple data points included in at least one of the input data and the three-dimensional point cloud data.

(Appendix B4)
The information processing method according to claim 1, wherein the learning step further refers to feature amounts associated with the three-dimensional thinned point cloud data to learn the estimation model.

(Appendix B5)
The estimation model is
a first estimation model that receives point cloud data and outputs interpolated point cloud data;
The information processing method according to claim 1 or 2, further comprising: a second estimation model that receives at least the complemented point cloud data and outputs a segmentation label related to the complemented point cloud data.

(Appendix B6)
The method further includes a projection step of generating projected point cloud data by projecting the interpolated point cloud data generated by the estimation model onto a two-dimensional surface,
The information processing method according to any one of Appendices B1 to B5, wherein the learning step further refers to the projected point cloud data and a segmentation answer label corresponding to the projected point cloud data to learn the estimation model.

(Appendix B7)
The information processing method according to any one of appendices B1 to B6, wherein the thinned point cloud data generation step generates a plurality of mutually different three-dimensional thinned point cloud data for the same input data or the same three-dimensional point cloud data.

(Appendix B8)
An acquisition step of acquiring input data;
a point cloud data generation step of generating three-dimensional point cloud data from the input data;
An information processing method comprising: an estimation step of estimating the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation step and the segmentation label related to the complemented point cloud data, using an estimation model that has been machine-learned by referring to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data, and the segmentation label related to the complemented point cloud data, the estimation model having point cloud data as input and complemented point cloud data and segmentation labels related to the complemented point cloud data as output.

(Appendix B9)
The information processing method according to appended claim B8, further comprising a presentation step of presenting a segmentation result indicated by the segmentation label estimated by the estimation step.

(Appendix B10)
The acquiring step acquires a medical image as the input data;
The information processing method according to claim 9, wherein the presentation step presents the segmentation results to assist a medical professional in making a decision.

(Appendix B11)
The information processing method according to any one of Appendices B8 to B10, wherein the estimation step further refers to features associated with the three-dimensional point cloud data generated by the point cloud data generation step, and estimates complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation step and a segmentation label related to the complemented point cloud data.

(Appendix B12)
The estimation model is
A first estimation model that receives point cloud data as input and outputs interpolated point cloud data;
and a second estimation model that receives at least the complemented point cloud data and outputs a segmentation label related to the complemented point cloud data.

[Appendix C]
This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

(Appendix C1)
A program for causing a computer to function as an information processing device,
The computer,
An acquisition means for acquiring input data;
a point cloud data generating means for generating three-dimensional point cloud data from the input data;
a thinned point cloud data generating means for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data;
An information processing program that functions as a learning means that receives point cloud data as input, and learns an estimation model that outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

(Appendix C2)
The thinning process by the thinned point cloud data generating means includes the following steps:
The information processing program according to claim C1, further comprising a process of generating the three-dimensional thinned point cloud data using only a portion of a plurality of frames included in the input data.

(Appendix C3)
The thinning process by the thinned point cloud data generating means includes the following steps:
The information processing program described in Appendix C1 or C2 includes a process of generating the three-dimensional thinned point cloud data using only a portion of the multiple data points included in at least one of the input data and the three-dimensional point cloud data.

(Appendix C4)
The information processing program according to claim 1, wherein the learning means further refers to features associated with the three-dimensional thinned point cloud data to learn the estimation model.

(Appendix C5)
The estimation model is
a first estimation model that receives point cloud data and outputs interpolated point cloud data;
and a second estimation model that receives at least the complemented point cloud data and outputs a segmentation label related to the complemented point cloud data.

(Appendix C6)
The computer,
The projection unit further functions as a projection unit that generates projected point cloud data by projecting the interpolated point cloud data generated by the estimation model onto a two-dimensional surface;
The information processing program according to any one of appendices C1 to C5, wherein the learning means further refers to the projected point cloud data and a segmentation answer label corresponding to the projected point cloud data to train the estimation model.

(Appendix C7)
The information processing program according to any one of appendices C1 to C6, wherein the thinned point cloud data generation means generates multiple different three-dimensional thinned point cloud data for the same input data or the same three-dimensional point cloud data.

(Appendix C8)
The computer,
An acquisition means for acquiring input data;
a point cloud data generating means for generating three-dimensional point cloud data from the input data;
An information processing program that functions as an estimation means for estimating complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means and a segmentation label related to the complemented point cloud data, using an estimation model that has been machine-learned by referring to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data, and an estimation model that has been machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

(Appendix C9)
The computer,
The information processing program according to claim C8, which functions as a presentation means for presenting a segmentation result indicated by the segmentation label estimated by the estimation means.

(Appendix C10)
The acquisition means acquires a medical image as the input data,
The information processing program according to claim C9, wherein the presentation means presents the segmentation results to assist medical personnel in making decisions.

(Appendix C11)
The information processing program according to any one of Appendices C8 to C10, wherein the estimation means further refers to features associated with the three-dimensional point cloud data generated by the point cloud data generation means, and estimates complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means and a segmentation label related to the complemented point cloud data.

(Appendix C12)
The estimation model is
A first estimation model that receives point cloud data as input and outputs interpolated point cloud data;
and a second estimation model that receives at least the complemented point cloud data and outputs a segmentation label related to the complemented point cloud data.

[Appendix D]
This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

(Appendix D1)
At least one processor, the at least one processor comprising:
An acquisition process for acquiring input data;
A point cloud data generation process for generating three-dimensional point cloud data from the input data;
a thinned point cloud data generation process for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data;
An information processing device that executes a learning process in which point cloud data is input, and an estimation model that outputs complemented point cloud data and segmentation labels related to the complemented point cloud data is trained by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

(Appendix D2)
The thinning process by the thinned point cloud data generation process includes the following steps:
The information processing device according to appendix D1, further comprising a process of generating the three-dimensional thinned point cloud data using only a portion of a plurality of frames included in the input data.

(Appendix D3)
The thinning process by the thinned point cloud data generation process includes the following steps:
The information processing device described in Appendix D1 or D2 includes a process of generating the three-dimensional thinned point cloud data using only a portion of the multiple data points included in at least one of the input data and the three-dimensional point cloud data.

(Appendix D4)
The information processing device according to claim 1, wherein in the learning process, the at least one processor further refers to features associated with the three-dimensional thinned point cloud data to learn the estimation model.

(Appendix D5)
The estimation model is
a first estimation model that receives point cloud data and outputs interpolated point cloud data;
The information processing device according to claim 1 or 2, further comprising: a second estimation model that receives at least the complemented point cloud data and outputs a segmentation label related to the complemented point cloud data.

(Appendix D6)
The at least one processor:
A projection process is further performed to generate projected point cloud data by projecting the interpolated point cloud data generated by the estimation model onto a two-dimensional surface;
The information processing device according to any one of appendices D1 to D5, wherein in the learning process, the at least one processor further refers to the projected point cloud data and a segmentation answer label corresponding to the projected point cloud data to learn the estimation model.

(Appendix D7)
The information processing device according to any one of appendices D1 to D6, wherein the thinned point cloud data generation process generates multiple pieces of three-dimensional thinned point cloud data that are different from each other for the same input data or the same three-dimensional point cloud data.

(Appendix D8)
The at least one processor
An acquisition process for acquiring input data;
A point cloud data generation process for generating three-dimensional point cloud data from the input data;
An information processing device that executes an estimation process to estimate complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation process and a segmentation label related to the complemented point cloud data, using an estimation model that has been machine-learned by referring to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data, and a segmentation label related to the complemented point cloud data.

(Appendix D9)
The at least one processor
The information processing device according to claim D8, which executes a presentation process of presenting a segmentation result indicated by the segmentation label estimated by the estimation process.

(Appendix D10)
In the acquisition process, the at least one processor acquires a medical image as the input data;
The information processing device of claim D9, wherein in the presentation process, the at least one processor presents the segmentation results to assist a medical professional in making decisions.

(Appendix D11)
The information processing device of any one of Appendices D8 to D10, wherein, in the estimation process, the at least one processor further refers to features associated with the three-dimensional point cloud data generated by the point cloud data generation process to estimate complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation process and a segmentation label related to the complemented point cloud data.

(Appendix D12)
The estimation model is
A first estimation model that receives point cloud data as input and outputs interpolated point cloud data;
The information processing device according to any one of appendices D8 to D11, further including a second estimation model that receives at least the complemented point cloud data and outputs a segmentation label related to the complemented point cloud data.

[Appendix E]
This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

(Appendix E1)
A program for causing a computer to function as an information processing device,
The computer includes:
An acquisition process for acquiring input data;
A point cloud data generation process for generating three-dimensional point cloud data from the input data;
a thinned point cloud data generation process for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data;
A non-transient recording medium having recorded thereon an information processing program that executes a learning process in which an estimation model is trained by inputting point cloud data and outputting complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

1, 2, 1A, 1B, 1C, 1D ... Information processing device 11, 21 ... Acquisition unit (acquisition means)
12, 22 ... Point cloud data generation unit (point cloud data generation means)
13 ... Thinned point cloud data generation unit (thinned point cloud data generation means)
14 ... learning unit (learning means)
23... Estimation unit (estimation means)
16... Projection unit (projection means)

Claims (16)

An acquisition means for acquiring input data;
a point cloud data generating means for generating three-dimensional point cloud data from the input data;
a thinned point cloud data generating means for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data;
and a learning means for learning an estimation model that receives point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. The thinning process by the thinned point cloud data generating means includes the following steps:
The information processing apparatus according to claim 1 , further comprising a process of generating the three-dimensional thinned point cloud data by using only a portion of a plurality of frames included in the input data. The thinning process by the thinned point cloud data generating means includes the following steps:
The information processing apparatus according to claim 1 or 2, further comprising a process of generating the three-dimensional thinned point cloud data using only a portion of a plurality of data points included in at least one of the input data and the three-dimensional point cloud data. The information processing apparatus according to claim 1 , wherein the learning means learns the estimation model by further referring to feature amounts accompanying the three-dimensional thinned point cloud data. The estimation model is
a first estimation model that receives point cloud data and outputs interpolated point cloud data;
The information processing apparatus according to claim 1 , further comprising: a second estimation model that receives at least the interpolated point cloud data and outputs a segmentation label related to the interpolated point cloud data. a projection unit for projecting the interpolated point cloud data generated by the estimation model onto a two-dimensional surface to generate projected point cloud data;
The information processing device according to claim 1 , wherein the learning means learns the estimation model by further referring to the projected point cloud data and a segmentation correct answer label corresponding to the projected point cloud data. The information processing apparatus according to claim 1 , wherein the thinned point cloud data generating means generates a plurality of mutually different three-dimensional thinned point cloud data for the same input data or the same three-dimensional point cloud data. An acquisition means for acquiring input data;
a point cloud data generating means for generating three-dimensional point cloud data from the input data;
An information processing device comprising: an estimation means for estimating, using an estimation model trained by machine learning with reference to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data, the complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means and the segmentation label related to the complemented point cloud data, the estimation model having point cloud data as input and complemented point cloud data and segmentation labels related to the complemented point cloud data as output. The information processing apparatus according to claim 8 , further comprising: a presentation unit that presents a segmentation result indicated by the segmentation label estimated by the estimation unit. The acquisition means acquires a medical image as the input data,
The information processing apparatus according to claim 9 , wherein the presentation means presents the segmentation result to assist a medical professional in making a decision. The information processing device according to any one of claims 8 to 10, wherein the estimation means further refers to features accompanying the three-dimensional point cloud data generated by the point cloud data generation means, and estimates complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means and a segmentation label related to the complemented point cloud data. The estimation model is
A first estimation model that receives point cloud data as input and outputs interpolated point cloud data;
and a second estimation model that receives at least the interpolated point cloud data and outputs a segmentation label related to the interpolated point cloud data. Obtaining input data;
generating three-dimensional point cloud data from the input data;
applying a thinning process to at least one of the input data and the three-dimensional point cloud data to generate three-dimensional thinned point cloud data;
An information processing method comprising: training an estimation model that receives point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. Obtaining input data;
generating three-dimensional point cloud data from the input data;
An information processing method comprising: estimating, using an estimation model that has point cloud data as input and outputs complemented point cloud data and a segmentation label related to the complemented point cloud data, the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the generating step, and the segmentation label related to the complemented point cloud data, the estimation model being machine-learned by referring to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data. On the computer,
An acquisition process for acquiring input data;
A point cloud data generation process for generating three-dimensional point cloud data from the input data;
a thinned point cloud data generation process for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data;
and a learning process for learning an estimation model that receives point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. On the computer,
An acquisition process for acquiring input data;
A point cloud data generation process for generating three-dimensional point cloud data from the input data;
A program that executes an estimation process to estimate complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and a segmentation label related to the complemented point cloud data, using an estimation model that has been machine-learned by referring to three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data, and the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process. The program executes an estimation process to estimate a segmentation label related to the complemented point cloud data, the ...

Images