JP2020204575A - Size data calculation device, program, method, product manufacturing device, and product manufacturing system - Google Patents

Abstract

To provide precise size data of each part of a target object.SOLUTION: A size data calculation device 1020 includes an acquisition unit 1024A, an extraction unit 1024B, a conversion unit 1024C, and a calculation unit 1024D. The acquisition unit 1024A acquires image data of a target object and total length data of the object. The extraction unit 1024B extracts shape data showing the shape of the object from the image data. The conversion unit 1024C converts the shape data on the basis of the total length data into a silhouette. The calculation unit 1024D calculates the size data of each part of the object by using the shape data converted by the conversion unit 1024C.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to dimensional data calculation devices, programs, methods, product manufacturing devices, and product manufacturing systems.

Conventionally, an apparatus for manufacturing a product based on the shape of an object has been studied. For example, in Patent Document 1 (Japanese Unexamined Patent Publication No. 2017-018158), a fingernail is photographed to obtain a nail image, and based on the obtained nail image, the shape of the nail, the position of the nail, the curvature of the nail, etc. Disclosed is a technique for obtaining nail information necessary for creating a nail, storing the information, and creating a nail part based on the information.

However, with the prior art, it has been difficult to calculate a large number of shapes in an object with high accuracy.

The dimension data calculation device of the first aspect includes an acquisition unit, an extraction unit, a conversion unit, and a calculation unit. The acquisition unit acquires the image data in which the object is photographed and the total length data of the object. The extraction unit extracts shape data indicating the shape of the object from the image data. The conversion unit converts the shape data based on the total length data and creates a silhouette. The calculation unit calculates the dimensional data of each part of the object by using the shape data converted by the conversion unit. With such a configuration, the dimensional data calculation device can provide highly accurate dimensional data of each part of the object.

The dimension data calculation device of the second aspect includes an acquisition unit and a calculation unit. The acquisition unit acquires attribute data including at least one of the total length data, the weight data, and the time-dependent data of the object. The calculation unit calculates the dimensional data of each part of the object by using the attribute data. By using these attribute data, the dimensional data calculation device can provide highly accurate dimensional data of each part of the object.

The information processing device of the third aspect uses a reception unit that receives a silhouette image of an object, and an object engine that associates the silhouette image of the sample object with the values of a predetermined number of shape parameters associated with the sample object. Then, an estimation unit that estimates the value of the shape parameter of the object from the received silhouette image is provided, and the estimated value of the shape parameter of the object is converted into dimensional data related to an arbitrary part of the object. Be associated. With such a configuration, it is possible to provide dimensional data calculated for an object with high accuracy.

The information processing device of the fourth aspect uses a reception unit that receives the attribute data of the object, and an object engine that associates the attribute data of the sample object with the values of a predetermined number of shape parameters associated with the sample object. Then, an estimation unit that estimates the value of the shape parameter of the object from the received attribute data is provided, and the estimated value of the shape parameter of the object is converted into dimensional data related to an arbitrary part of the object. Be associated. With such a configuration, it is possible to provide dimensional data calculated for an object with high accuracy.

The dimensional data calculation device of the fifth viewpoint has an acquisition unit that acquires image data of the object and full length data of the object, and an extraction unit and shape data that extracts shape data indicating the shape of the object from the image data. A silhouette image using a converter that converts the image into a silhouette image based on full-length data, and an object engine that associates the silhouette image of the sample object with the values of a predetermined number of shape parameters associated with the sample object. It is provided with an estimation unit that estimates the values of a predetermined number of shape parameters from the above, and a calculation unit that calculates dimensional data of an object based on the estimated values of a predetermined number of shape parameters. With such a configuration, it is possible to provide dimensional data calculated for an object with high accuracy.

It is a schematic diagram which shows the structure of the dimension data calculation apparatus 1020 which concerns on 1st Embodiment. It is a flowchart for demonstrating the operation of the dimension data calculation apparatus 1020 which concerns on this embodiment. It is a schematic diagram which shows the concept of the shape data which concerns on the same embodiment. It is a schematic diagram which shows the concept of the product manufacturing system 1001 which concerns on the same embodiment. It is a sequence diagram for demonstrating the operation of the product manufacturing system 1001 which concerns on this embodiment. It is a schematic diagram which shows an example of the screen displayed on the terminal apparatus 1010 which concerns on the same embodiment. It is a schematic diagram which shows an example of the screen displayed on the terminal apparatus 1010 which concerns on the same embodiment. It is a schematic diagram which shows the structure of the dimension data calculation apparatus 2120 which concerns on 2nd Embodiment. It is a schematic diagram which shows the concept of the data used in the quadratic regression which concerns on the same embodiment. It is a schematic diagram which shows the concept of the product manufacturing system 2001S which concerns on the same embodiment. It is a schematic diagram of the dimension data calculation system 3100 which concerns on 3rd Embodiment. It is a flowchart which shows the operation of the learning apparatus 3025 of FIG. It is a flowchart which shows the operation of the dimension data calculation apparatus 3020 of FIG. It is a schematic explanatory drawing which shows the characteristic of a shape parameter. It is a schematic graph which shows the characteristic of a shape parameter. It is a schematic diagram which shows the concept of the product manufacturing system 3001 which concerns on 3rd Embodiment. It is a sequence diagram which shows the operation of the product manufacturing system 3001 of FIG. It is a schematic diagram which shows an example of the screen displayed on the terminal apparatus 3010 of FIG. It is a schematic diagram which shows an example of the screen displayed on the terminal apparatus 3010 of FIG. It is a schematic diagram of the dimension data calculation system 4100 which concerns on 4th Embodiment. It is a flowchart which shows the operation of the learning apparatus 4125 of FIG. It is a flowchart which shows the operation of the dimension data calculation apparatus 4020 of FIG. It is a schematic diagram which shows the concept of the product manufacturing system 4001S which concerns on 4th Embodiment.

<First Embodiment>
(1-1) Configuration of Dimension Data Calculation Device FIG. 1 is a schematic view showing the configuration of the dimension data calculation device 1020 according to the present embodiment.

The dimensional data calculation device 1020 can be realized by an arbitrary computer, and includes a storage unit 1021, an input / output unit 1022, a communication unit 1023, and a processing unit 1024. The dimension data calculation device 1020 may be realized as hardware by using LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or the like.

The storage unit 1021 stores various types of information, and is realized by an arbitrary storage device such as a memory and a hard disk. For example, the storage unit 1021 stores a weighting coefficient necessary for executing information processing described later in relation to the length and weight of the object. The weighting coefficient is acquired in advance by executing machine learning from teacher data consisting of attribute data, image data, and dimension data described later.

The input / output unit 1022 is realized by a keyboard, a mouse, a touch panel, and the like, and inputs various information to the computer and outputs various information from the computer.
The communication unit 1023 is realized by an arbitrary network card or the like, and enables communication with a communication device on the network by wire or wirelessly.

The processing unit 1024 executes various types of information processing, and is realized by a processor such as a CPU or GPU and a memory. Here, the processing unit 1024 functions as the acquisition unit 1024A, the extraction unit 1024B, the conversion unit 1024C, and the calculation unit 1024D by reading the program stored in the storage unit 1021 into the CPU, GPU, or the like of the computer.

The acquisition unit 1024A acquires image data obtained by photographing the object, total length data, weight data, and the like of the object. Here, the acquisition unit 1024A acquires a plurality of image data obtained by photographing the object from different directions.

The extraction unit 1024B extracts shape data indicating the shape of the object from the image data. Specifically, the extraction unit 1024B uses a semantic segmentation algorithm (Mask R-CNN, etc.) prepared for each type of object to extract the object region included in the image data to extract the object. Extract the shape data of. Here, the semantic segmentation algorithm is constructed using teacher data in which the shape of the object is not specified.

When the semantic segmentation algorithm is constructed using the teacher data of an object whose shape is unspecified, it may not always be possible to extract the shape of the object with high accuracy. In such a case, the extraction unit 1024B extracts the shape data of the object from the object region by the GrabCut algorithm. This makes it possible to extract the shape of the object with high accuracy. Further, the extraction unit 1024B may correct the image of the object specified by the grab cut algorithm based on the color image of the specific portion of the object. This makes it possible to generate shape data of the object with higher accuracy.

The conversion unit 1024C converts the shape data based on the total length data and creates a silhouette. As a result, the shape data is standardized.

The calculation unit 1024D calculates the dimensional data of each part of the object by using the shape data converted by the conversion unit 1024C. Specifically, the calculation unit 1024D reduces the dimension of the shape data converted by the conversion unit 1024C. The dimension reduction referred to here is realized by a method such as principal component analysis, particularly kernel principal component analysis (KernelPCA), and linear discriminant analysis.

Then, the calculation unit 1024D calculates the dimensional data of each part of the object by using the reduced value of each dimension and the weighting coefficient optimized for each part of the object.

More specifically, the calculation unit 1024D linearly combines the value of each dimension reduced at the first time and the weighting coefficient W1pi optimized for each part of the object to obtain a predetermined value Zpi. The symbol p is the number of dimensions obtained by reduction, and is a value of 10 or more. Then, the calculation unit 1024D performs the second dimension reduction using the predetermined value Zpi and the attribute data including at least the attributes of the length and weight of the object, and each dimension obtained by the second dimension reduction. Dimensional data of each part of the object is calculated based on the value of. The number of weighting coefficients W1pi is prepared in the same number as the reduced dimension for each dimension (i) of the object.

In the above description, the calculation unit 1024D obtains a predetermined value Zpi using a linear combination, but the calculation unit 1024D may obtain these values by a method other than the linear combination. Specifically, the calculation unit 1024D generates a quadratic feature amount from the value of each dimension obtained by the dimension reduction, and sets the quadratic feature amount and the weighting coefficient optimized for each part of the object. May be obtained by combining the above.

(1-2) Operation of Dimension Data Calculator FIG. 2 is a flowchart for explaining the operation of the dimensional data calculation device 1020 according to the present embodiment.

First, the dimension data calculation device 1020 acquires a plurality of image data obtained by photographing the entire object from different directions via an external terminal device or the like together with the total length data indicating the total length of the object (S1001).
Next, the dimension data calculation device 1020 extracts shape data indicating the shape of each part of the object from each image data (S1002).
Subsequently, the dimension data calculation device 1020 executes a rescale process for converting each shape data into a predetermined size based on the total length data (S1003).

Next, the dimension data calculation device 1020 combines a plurality of converted shape data to generate new shape data (hereinafter, also referred to as shape data for calculation). Specifically, as shown in FIG. 3, the shape data of h rows and w columns are combined to form a data string of m × h × w. The symbol m is the number of shape data (S1004).

After that, the dimension data calculation device 1020 uses the newly generated shape data and the weighting coefficient W1pi optimized for each part of the object to make the i-th (i = 1 to j) in the object. The dimensional data of each part of the above is calculated (S1005 to S1008). The symbol j is the total number of dimensional locations for which dimensional data is to be calculated.

(1-3) Features of Dimension Data Calculator (1-3-1)
As described above, the dimension data calculation device 1020 according to the present embodiment includes an acquisition unit 1024A, an extraction unit 1024B, a conversion unit 1024C, and a calculation unit 1024D. The acquisition unit 1024A acquires the image data in which the object is photographed and the total length data of the object. The extraction unit 1024B extracts shape data indicating the shape of the object from the image data. The conversion unit 1024C converts the shape data based on the total length data and creates a silhouette. The calculation unit 1024D calculates the dimensional data of each part of the object by using the shape data converted by the conversion unit 1024C.

Therefore, since the dimension data calculation device 1020 calculates the dimension data of each part of the object using the image data and the total length data, it is possible to provide highly accurate dimension data. Further, since the dimensional data calculation device 1020 can process a large number of image data and the total length data at one time, it is possible to provide a large number of dimensional data with high accuracy.

Then, by using such a dimensional data calculation device 1020, for example, dimensional data of each part of an organism as an object can be calculated with high accuracy. In addition, it is possible to calculate the dimensional data of each part of an arbitrary object such as a car or various luggage as an object with high accuracy.
Further, by incorporating the dimensional data calculation device into the product manufacturing device that manufactures various products, it becomes possible to manufacture a product that matches the shape of the object.

(1-3-2)
Further, in the dimension data calculation device 1020, the acquisition unit 1024A acquires a plurality of image data obtained by photographing the object from different directions. With such a configuration, the accuracy of the dimensional data can be improved.

(1-3-3)
Further, in the dimension data calculation device 1020, the calculation unit 1024D reduces the dimension of the shape data converted by the conversion unit 1024C. Then, the calculation unit 1024D calculates the dimensional data of each part of the object by using the reduced value of each dimension and the weighting coefficient W1pi optimized for each part of the object. With such a configuration, the accuracy of the dimensional data can be improved while suppressing the calculation load.

Specifically, the calculation unit 1024D linearly combines the reduced values of each dimension with the weighting coefficient W1pi optimized for the i-th part of the object to obtain a predetermined value Zi. Further, the calculation unit 1024D executes the second dimension reduction using the predetermined value Zi and the attribute data including at least the attributes of the length and weight of the object, and executes the second dimension reduction to the i-th dimension of the object. Calculate the data. With such a configuration, the accuracy of the dimensional data can be further improved while suppressing the calculation load. In the above description, the calculation unit 1024D generates a quadratic feature amount from the value of each dimension obtained by the dimension reduction instead of the linear combination, and for each of the quadratic feature amount and the object part. A predetermined value may be obtained by combining with the optimized weighting coefficient.

(1-3-4)
Further, in the dimension data calculation device 1020, the extraction unit 1024B extracts the object area included in the image data by using the semantic segmentation algorithm constructed by using the teacher data prepared for each type of the object. By doing so, the shape data of the object is extracted. With such a configuration, the accuracy of the dimensional data can be improved.

Some algorithms for semantic segmentation are open to the public, but those that are open to the public are usually constructed using teacher data in which the shape of the object is not specified. Therefore, depending on the purpose, the accuracy of extracting the object region included in the image data may not always be sufficient.

Therefore, in such a case, the extraction unit 1024B extracts the shape data of the object from the object region by the grab cut algorithm. With such a configuration, the accuracy of the dimensional data can be further improved.

Further, the extraction unit 1024B may generate new shape data by correcting the image of the object extracted by the grab cut algorithm based on the color image of the specific portion in the image data. With such a configuration, the accuracy of the dimensional data can be further improved. For example, when the object is a person, the shape data of the person who is the object can be obtained with high accuracy by setting the hands and back as specific parts and correcting them based on the color image of these specific parts. it can.

(1-4) Modification example (1-4-1)
In the above description, it is assumed that the acquisition unit 1024A acquires a plurality of image data obtained by photographing the object from different directions, but a plurality of image data are not always required. It is possible to calculate the dimensional data of each part even if the image data of the object is one.

(1-4-2)
In the above description, the calculation unit 1024D executes the dimension reduction twice, but such processing is not always necessary. The calculation unit 1024D may calculate the dimensional data of each part of the object from the value of each dimension obtained by executing the dimension reduction once. Further, depending on the purpose, the dimension data calculation device 1020 may calculate the dimension data without reducing the dimension of the shape data.

(1-4-3)
In the above description, it is assumed that the extraction unit 1024B extracts the object region included in the image data by using the semantic segmentation algorithm constructed by using the teacher data in which the shape of the object is not specified. It is not necessary to use such teacher data. For example, a semantic segmentation algorithm constructed using teacher data in which the shape of the object is specified may be used. By using the teacher data in which the shape of the object is specified, it is possible to improve the calculation accuracy of the dimensional data and suppress the calculation load according to the purpose.

(1-5) Application to Product Manufacturing System Hereinafter, an example in which the above-mentioned dimension data calculation device 1020 is applied to the product manufacturing system 1001 will be described.
(1-5-1) Configuration of Product Manufacturing System FIG. 4 is a schematic diagram showing the concept of the product manufacturing system 1001 according to the present embodiment.
The product manufacturing system 1001 includes a dimension data calculation device 1020 capable of communicating with the terminal device 1010 owned by the user 1005, and a product manufacturing device 1030, and is a system for manufacturing a desired product 1006. In FIG. 4, as an example, the concept when the object 1007 is a person and the product 1006 is a chair is shown. However, the object 1007 and the product 1006 of the product manufacturing system according to the present embodiment are not limited thereto.

The terminal device 1010 can be realized by a so-called smart device. Here, the terminal device 1010 exerts various functions by installing a user program on the smart device. Specifically, the terminal device 1010 generates image data captured by the user 1005. Further, the terminal device 1010 accepts the input of the attribute data indicating the attribute of the object 1007. Examples of the "attribute" include the total length, weight, and elapsed time (including age) of the object 1007. Further, the terminal device 1010 has a communication function, and executes transmission / reception of various information with the dimension data calculation device 1020 and the product manufacturing device 1030.

The dimensional data calculation device 1020 can be realized by any computer. Here, the storage unit 1021 of the dimension data calculation device 1020 stores the information transmitted from the terminal device 1010 in association with the identification information that identifies the user 1005 of the terminal device 1010. In addition, the storage unit 1021 stores parameters and the like necessary for executing information processing described later. For example, the storage unit 1021 stores the weighting coefficient W1pi necessary for executing the information processing described later in association with the attribute items of the object 1007 and the like.

Further, as described above, the processing unit 1024 of the dimension data calculation device 1020 functions as an acquisition unit 1024A, an extraction unit 1024B, a conversion unit 1024C, and a calculation unit 1024D. Here, the acquisition unit 1024A acquires the image data taken by the user 1005 and the attribute data of the object 1007. Further, the extraction unit 1024B extracts shape data indicating the shape of the object 1007 from the image data. For example, when "person" is preset as the type of object, a semantic segmentation algorithm is constructed using teacher data for identifying a person. Further, the extraction unit 1024B corrects the image of the object 1007 specified by the grab cut algorithm based on the color image of the specific portion of the object 1007, and generates the shape data of the object 1007 with higher accuracy. Further, the conversion unit 1024C converts the shape data based on the total length data and creates a silhouette. The calculation unit 1024D calculates the dimensional data of each part of the user 1005 by using the shape data converted by the conversion unit 1024C. Here, the calculation unit 1024D obtains a predetermined value Z1i by linearly combining the reduced values of each dimension and the weighting coefficient W1pi optimized for each part of the object 1007. Then, the calculation unit 1024D reduces the dimension using the predetermined value Z1i and the attribute data of the object 1007, and calculates the dimension data of each part of the object 1007 based on the reduced value of each dimension.

The product manufacturing apparatus 1030 is a manufacturing apparatus that manufactures a desired product related to the shape of the object 1007 by using the dimensional data calculated by using the dimensional data calculating device 1020. The product manufacturing device 1030 can be any device capable of automatically manufacturing and processing a product, and can be realized by, for example, a three-dimensional printer.

(1-5-2) Operation of Product Manufacturing System FIG. 5 is a sequence diagram for explaining the operation of the product manufacturing system 1001 according to the present embodiment. Further, FIGS. 6 and 7 are schematic views showing screen transitions of the terminal device 1010.
First, the entire object 1007 is imaged a plurality of times via the terminal device 1010 so as to be photographed from different directions, and a plurality of image data in which the object 1007 is imaged is generated (T1001). Here, a plurality of front and side photographs as shown in FIGS. 6 and 7 are taken.

Next, the user 1005 inputs attribute data indicating the attributes of the object 1007 to the terminal device 1010 (T1002). Here, as the attribute data, the total length data, the weight data, the time-lapse data (including age, etc.) of the object 1007 and the like are input.
Then, these plurality of image data and attribute data are transmitted from the terminal device 1010 to the dimension data calculation device 1020.

When the dimensional data calculation device 1020 receives a plurality of image data and attribute data from the terminal device 1010, the dimensional data calculation device 1020 calculates the dimensional data of each part of the object 1007 using these data (T1003). The terminal device 1010 displays dimensional data on the screen according to the setting.
Then, the product manufacturing apparatus 1030 manufactures a desired product 1006 based on the dimensional data calculated by the dimensional data calculating apparatus 1020 (T1004).

(1-5-3) Features of Product Manufacturing System As described above, the product manufacturing system 1001 according to the present embodiment includes a dimension data calculation device 1020 capable of communicating with the terminal device 1010 owned by the user 1005 and product manufacturing. A device 1030 is provided. The terminal device 1010 (photographing device) captures a plurality of images of the object 1007. The dimension data calculation device 1020 includes an acquisition unit 1024A, an extraction unit 1024B, a conversion unit 1024C, and a calculation unit 1024D. The acquisition unit 1024A acquires the image data of the object 1007 from the terminal device 1010 together with the total length data of the object 1007. The extraction unit 1024B extracts shape data indicating the shape of the object 1007 from the image data. The conversion unit 1024C converts the shape data based on the total length data and creates a silhouette. The calculation unit 1024D calculates the dimensional data of each part of the object 1007 by using the shape data converted by the conversion unit 1024C. The product manufacturing apparatus 1030 manufactures the product 1006 using the dimensional data calculated by the calculation unit 1024D.
With such a configuration, since the dimension data calculation device 1020 calculates each part of the object 1007 with high accuracy, it is possible to provide a desired product related to the shape of the object 1007.

For example, the product manufacturing system 1001 can manufacture a model of an organ by measuring the shape of various organs such as the heart.
Further, for example, various healthcare products can be manufactured by measuring the waist shape of a person.
Further, for example, a figure product of the person can be manufactured from the shape of the person.
Further, for example, a chair or the like suitable for the person can be manufactured from the shape of the person.
Also, for example, a car toy can be manufactured from the shape of a car.
Further, for example, a diorama or the like can be manufactured from an arbitrary landscape painting.

In the above description, the dimension data calculation device 1020 and the product manufacturing device 1030 are described as separate members, but these may be configured as one.

<Second Embodiment>
Hereinafter, the configurations and functions already described will be designated by substantially the same reference numerals and description thereof will be omitted.
(2-1) Configuration of Dimension Data Calculation Device FIG. 8 is a schematic view showing the configuration of the dimensional data calculation device 2120 according to the present embodiment.
The dimensional data calculation device 2120 can be realized by any computer, and includes a storage unit 2121, an input / output unit 2122, a communication unit 2123, and a processing unit 2124. The dimension data calculation device 2120 may be realized as hardware by using LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or the like.

The storage unit 2121 stores various types of information, and is realized by an arbitrary storage device such as a memory and a hard disk. For example, the storage unit 2121 stores the weighting coefficient Wri required for executing the information processing described later in relation to the length and weight of the object. The weighting coefficient is acquired in advance by executing machine learning from teacher data consisting of attribute data and dimension data described later.
The input / output unit 2122 has the same configuration and function as the above-mentioned input / output unit 1022.
The communication unit 2123 has the same configuration and function as the communication unit 1023 described above.

The processing unit 2124 executes various types of information processing, and is realized by a processor such as a CPU or GPU and a memory. Here, the processing unit 2124 functions as the acquisition unit 2124A and the calculation unit 2124D by reading the program stored in the storage unit 2121 into the CPU, GPU, or the like of the computer.

The acquisition unit 2124A acquires the attribute data Dzr (r is the number of elements of the attribute data) including at least one of the total length data, the weight data, and the time-dependent data (including age) of the object.

The calculation unit 2124D calculates the dimensional data of each part of the object by using the attribute data acquired by the acquisition unit 2124A. Specifically, the calculation unit 2124D calculates the dimensional data of each part of the object by quadratic regression of the attribute data using the machine-learned weighting coefficient Wsi. The weighting coefficient is optimized for each part of the object, and the weighting coefficient of the i-th part of the object is expressed as Wsi. In addition, i = 1 to j, and j is the total number of dimensional points for which dimensional data is to be calculated. Further, the symbol s is the number of elements used in the calculation obtained from the attribute data.

Specifically, as shown in FIG. 9, the calculation unit 2124D squares each element x1, x2, x3 of the attribute data Dzr (r = 3) (the value in the first row in FIG. 9, and is a quadratic term. (Also referred to as), the value obtained by multiplying each element (the value in the second line in FIG. 9, also referred to as the interaction term), the value of each element itself (the value in the third line in FIG. 9, which is primary). Dimensional data is calculated using (also called term). In the example shown in FIG. 9, there are nine values obtained from the elements x1, x2, x3 of the three attribute data, and s = 9, so that the weighting coefficient Wsi has i × 9 elements. Become.
(2-2) Features of dimensional data calculation device

As described above, the dimension data calculation device 2120 according to the present embodiment includes an acquisition unit 2124A and a calculation unit 2124D. The acquisition unit 2124A acquires attribute data including at least one of the total length data, the weight data, and the time-lapse data of the object. The calculation unit 2124D calculates the dimensional data of each part of the object by using the attribute data.

Therefore, since the dimensional data calculation device 2120 calculates the dimensional data of each part of the object using the attribute data, it is possible to provide highly accurate dimensional data. Specifically, the calculation unit 2124D calculates the dimensional data of each part of the object with high accuracy by performing a quadratic regression of the attribute data using machine-learned coefficients.
Further, since the dimensional data calculation device 2020 can process a large amount of data at once, it is possible to provide a large number of dimensional data at high speed.

Further, when at least one of the total length data, the weight data, and the time-dependent data is included as the attribute data, the dimensional data of each part of the organism can be calculated with high accuracy.
Further, by incorporating the dimension data calculation device 2120 into a product manufacturing device that manufactures various products, it is possible to manufacture a product that matches the shape of the object.

In the above description, the calculation unit 2124D calculates the dimensional data of each part of the object by secondary regression of the attribute data, but the calculation of the calculation unit 2124D is not limited to this. .. The calculation unit 2124D may obtain the dimension data by linearly combining the attribute data.

(2-3) Application to Product Manufacturing System FIG. 10 is a schematic diagram showing the concept of the product manufacturing system 2001S according to the present embodiment.
The dimensional data calculation device 2120 according to the present embodiment can also be applied to the product manufacturing system 2001S in the same manner as the dimensional data calculation device 1020 according to the first embodiment.
The terminal device 2010S according to the present embodiment may be any as long as it accepts the input of attribute data indicating the attributes of the object 2007. Examples of the "attribute" include the total length, weight, and elapsed time (including age) of the object 1007.

Further, as described above, the processing unit 2124 of the dimension data calculation device 2120 functions as the acquisition unit 2124A and the calculation unit 2124D. The calculation unit 2124D calculates the dimensional data of each part of the object 2007 by using the attribute data acquired by the acquisition unit 2124A. Specifically, the calculation unit 2124D calculates the dimensional data of each part of the object by quadratic regression of the attribute data using the machine-learned weighting coefficient Wsi.

In the product manufacturing system 2001S, since the dimensional data calculation device 2120 calculates each part of the object 2007 with high accuracy, it is possible to provide a desired product related to the shape of the object 2007. In addition, the product manufacturing system 2001S according to the second embodiment can exhibit the same effect as the product manufacturing system 1001 of the first embodiment.

<Third Embodiment>

Hereinafter, the information processing apparatus, the information processing method, the product manufacturing apparatus, and the dimension data calculation system according to the embodiment of the dimension data calculation apparatus of the present invention will be described together with the attached drawings. In the description of the following embodiments, the information processing device and the dimensional data calculation device are implemented as a part of the dimensional data calculation system.

In the accompanying drawings, the same or similar elements are designated by the same or similar reference numerals, and duplicate description of the same or similar elements may be omitted in the description of each embodiment. In addition, the features shown in each embodiment can be applied to other embodiments as long as they do not contradict each other. Furthermore, the drawings are schematic and do not necessarily match the actual dimensions, ratios, etc. Even between drawings, there may be parts where the relationship and ratio of dimensions differ from each other.

In the following description, when a plurality of elements are collectively expressed using a matrix or a vector, they may be expressed in uppercase letters, and when individual elements of the matrix are expressed, they may be expressed in lowercase letters. For example, when indicating a set of shape parameters, it may be expressed as a matrix Λ, and when expressing an element of the matrix Λ, it may be expressed as an element λ.

(3-1) Configuration of Dimension Data Calculation System FIG. 11 is a schematic view showing the configuration of the dimensional data calculation system 3100 according to the present embodiment. The dimensional data calculation system 3100 includes a dimensional data calculation device 3020 and a learning device 3025.

The dimensional data calculation device 3020 and the learning device 3025 can be realized by any computer. The dimensional data calculation device 3020 includes a storage unit 3021, an input / output unit 3022, a communication unit 3023, and a processing unit 3024. Further, the learning device 3025 includes a storage unit 3026 and a processing unit 3027. The dimension data calculation device 3020 and the learning device 3025 may be realized as hardware by using LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or the like.

The storage units 3021, 3026 all store various types of information, and are realized by any storage device such as a memory and a hard disk. For example, the storage unit 3021 stores various data, programs, information, and the like including the object engine 3021A in order to execute information processing related to dimensional data calculation in the processing unit 3024. In addition, the storage unit 3026 stores training data used in the learning stage in order to generate the object engine 3021A.

The input / output unit 3022 is realized by a keyboard, a mouse, a touch panel, and the like, and inputs various information to the computer and outputs various information from the computer.
The communication unit 3023 is realized by a network interface such as an arbitrary network card, and enables communication with a communication device on the network by wire or wirelessly.

The processing units 3024 and 3027 are all realized by a processor such as a CPU (Central Processing Unit) and / or a GPU (Graphical Processing Unit) and a memory in order to execute various information processing. The processing unit 3024 functions as an acquisition unit 3024A, an extraction unit 3024B, a conversion unit 3024C, an estimation unit 3024D, and a calculation unit 3024E by reading the program stored in the storage unit 3021 into the CPU, GPU, or the like of the computer. .. Similarly, the processing unit 3027 functions as the preprocessing unit 3027A and the learning unit 3027B by reading the program stored in the storage unit 3026 into the CPU, GPU, or the like of the computer.

In the processing unit 3024 of the dimension data calculation device 3020, the acquisition unit 3024A acquires the image data in which the object is photographed and the attribute data such as the total length data and the weight data of the object. The acquisition unit 3024A acquires, for example, a plurality of image data obtained by photographing an object from a plurality of different directions by an imaging device.

The extraction unit 3024B extracts shape data indicating the shape of the object from the image data. Specifically, the extraction unit 3024B uses a semantic segmentation algorithm (Mask R-CNN, etc.) prepared for each type of object to extract the object region included in the image data to extract the object. Extract the shape data of. Semantic segmentation algorithms can be constructed using training data in which the shape of the object is not specified.

When the semantic segmentation algorithm is constructed using training data of an object whose shape is unspecified, it may not always be possible to extract the shape of the object with high accuracy. In such a case, the extraction unit 3024B extracts the shape data of the object from the object region by the Grab Cut algorithm. This makes it possible to extract the shape of the object with high accuracy.

Further, the extraction unit 3024B separates the object and the background image other than the object by correcting the image of the object specified by the grab cut algorithm based on the color image of the specific portion of the object. May be good. This makes it possible to generate shape data of the object with higher accuracy.

The conversion unit 3024C silhouettes the shape data based on the total length data. That is, the shape data of the object is converted to generate a silhouette image of the object. As a result, the shape data is standardized. The conversion unit 3024C also functions as a reception unit for inputting the generated silhouette image to the estimation unit 3024D.

The estimation unit 3024D estimates the values of a predetermined number of shape parameters from the silhouette image. The object engine 3021A is used for the estimation. The values of the shape parameters of a predetermined number of objects estimated by the estimation unit 3024D are associated with the dimensional data related to any part of the object.

The calculation unit 3024E calculates the dimensional data of the object associated with the predetermined number of shape parameter values estimated by the estimation unit 3024D. Specifically, the calculation unit 3024E constructs three-dimensional data of a plurality of vertices in the object from the values of the shape parameters of the object estimated by the estimation unit 3024D, and further, the object is based on the three-dimensional data. Calculate dimensional data between any two vertices in an object.

In the processing unit 3027 of the learning device 3025, the preprocessing unit 3027A carries out various preprocessing for learning. In particular, the preprocessing unit 3027A specifies a predetermined number of shape parameters by extracting features of the three-dimensional data of the sample object by dimensionality reduction. In addition, a predetermined number (dimensional) of shape parameter values are obtained for each sample object. The value of the shape parameter of the sample object is stored in the storage unit 3026 as training data.

Further, the preprocessing unit 3027A virtually configures the three-dimensional object of the sample object in the three-dimensional space based on the three-dimensional data of the sample object, and then virtually provides the three-dimensional object in the three-dimensional space. A three-dimensional object is projected from a predetermined direction using an image pickup device. The silhouette image of the sample object is stored in the storage unit 3026 as training data.

The learning unit 3027B learns to associate the silhouette image of the sample object with the value of a predetermined number of shape parameters associated with the sample object. As a result of learning, the object engine 3021A is generated. The generated object engine 3021A can be held in the form of an electronic file. When calculating the dimensional data of the object in the dimensional data calculation device 3020, the object engine 3021A is stored in the storage unit 3021 and referred to by the estimation unit 3024D.

(3-2) Operation of Dimension Data Calculation System The operation of the dimensional data calculation system 3100 of FIG. 11 will be described with reference to FIGS. 12 and 13. FIG. 12 is a flowchart showing the operation (S3010) of the learning device 3025, and generates the object engine 3021A based on the sample object data. FIG. 13 is a flowchart showing the operation of the dimension data calculation device 3020, and calculates the dimension data of the object based on the image data of the object.

(3-2-1) Operation of learning device First, the data of the sample object is prepared and stored in the storage unit 3026 (S3011). In one example, the data prepared is the data of 400 sample objects, including 5,000 3D data for each sample object. The three-dimensional data includes three-dimensional coordinate data of the vertices of the sample object. The three-dimensional data includes vertex information of each mesh constituting the three-dimensional object, mesh data such as the normal direction of each vertex, total length data, weight data, and time-dependent data (including age and the like). Attribute data such as may be included.

Vertex numbers are associated with the three-dimensional data of the sample object. In the above example, three-dimensional data of 5,000 vertices is associated with vertex numbers # 1 to # 5,000 for each sample object. In addition, all or part of the vertex numbers are associated with information on the part of the object. For example, when the object is a "person", vertex number # 20 is associated with "apex", similarly, vertex number # 313 is associated with "left shoulder acromion" and vertex number # 521 is associated with "right shoulder acromion". It is associated with "peak".

Subsequently, the preprocessing unit 3027A performs feature conversion into a shape parameter by reducing the dimension (S3012). Specifically, for each sample object, feature extraction is performed by reducing the dimensions of the three-dimensional data of the sample object. As a result, a predetermined number (number of dimensions) of shape parameters are obtained. In one example, the number of dimensions of the shape parameter is 30. Dimensionality reduction is realized by methods such as principal component analysis and Random Projection.

The preprocessing unit 3027A converts the three-dimensional data into the values of a predetermined number of shape parameters for each sample object by using the projection matrix of the principal component analysis. As a result, it is possible to remove noise from the three-dimensional data of the sample object and compress the three-dimensional data while maintaining the relevant characteristic information.

As in the example above, there are 400 sample object data, each sample object contains 3D (coordinate) data of 5,000 vertices, and each 3D data is characterized by a 30-dimensional shape parameter. It is supposed to be converted. Here, the vertex coordinate matrix of [400 rows, 15,000 columns (5,000 × 3)] representing the data of 400 sample objects is defined as the matrix X. Further, the projection matrix of [15,000 rows, 30 columns] generated by the principal component analysis is defined as the matrix W. By multiplying the vertex coordinate matrix X by the projection matrix W from the right, the matrix Λ which is the shape parameter matrix of [400 rows, 30 columns] can be obtained.

That is, the shape parameter matrix Λ can be calculated from the following mathematical formula.

As a result of matrix operation using the projection matrix W, the 15,000-dimensional data possessed by each of the 400 sample objects is feature-converted into the shape parameters (λ ₁ , ..., λ ₃₀ ) of the 30-dimensional principal components. Will be done. In the principal component analysis, 400 pieces of values for _{λ i (λ 1,1, ...,} λ 400,1) the average value of is calculated to be zero.

Following the acquisition of the shape parameter matrix Λ as a result of S3012, the preprocessing unit 3027A expands the data set of the shape parameters included in the shape parameter matrix Λ by using random numbers (S3013). In the above example, 400 data sets (λ _{i, 1} , ..., λ _{i, 30} (1 ≦ i ≦ 400)) are converted into an extended data set (λ _j ) of 10,000 shape parameters. _{, 1} , ..., λ _{j, 30} (1 ≦ j ≦ 10,000)). Data expansion is performed using random numbers having a normal distribution. The extended data set has a normal distribution with a variance of the values of each shape parameter of 3σ.

The three-dimensional data of the extended data set can be constructed by performing the inverse transformation of the extended data set based on the projection matrix W. In the above example, a matrix of [10,000 rows, 30 columns] extended shape parameter matrix representing 10,000 extended data sets is further plotted in the matrix Λ'(λ _{j, k} , ..., λ _{j, Let k} (1 ≦ j ≦ 10,000 and 1 ≦ k ≦ 30). 10,000 vertex coordinates matrix X representing a three-dimensional data of the sample object 'is expanded configuration parameter matrix lambda' relative, [30 lines, 15,000 columns] transposed matrix ^{W T} of the projection matrix W is Is obtained by multiplying from the right.

That is, the vertex coordinate matrix X'can be calculated from the following mathematical formula, and 5,000 (15,000/3) three-dimensional data are obtained for each sample object expanded to 10,000. be able to.

Following the acquisition of the vertex coordinate matrix X'as a result of S13, the preprocessing unit 3027A generates each silhouette image based on the expanded three-dimensional data of the sample object (S3014). In the above example, for every 10,000 sample objects, a three-dimensional object of the sample object is virtually constructed from 5,000 three-dimensional data in the three-dimensional space. Then, a three-dimensional object is projected using a projection device that is also virtually provided in the three-dimensional space and can be projected from an arbitrary direction. In the above example, it is preferable to acquire two silhouette images in the front direction and the side direction by projection for each 10,000 sample objects. The acquired silhouette image is represented by black and white binarized data.

Finally, the learning unit 3027B associates the value of the shape parameter associated with the sample object with the silhouette image of the sample object by learning (S3015). Specifically, it is preferable to use the set of the shape parameter data set obtained in S3013 and the silhouette image obtained in S3014 as training data to learn the relationship between the two by deep learning.

More specifically, for the sample objects expanded to 10,000 in the above example, the binarized data of each silhouette image is input to the deep learning network architecture. In feature extraction in deep learning, the weighting factors of the network architecture are set so that the data output from the network architecture approaches the values of the 30 shape parameters. For deep learning here, for example, a convolutional neural network (CNN) can be used.

In this way, in S3014, the relationship between the value of the shape parameter associated with the sample object and the silhouette image of the sample object is learned by deep learning, and the network architecture of deep learning is constructed. As a result, the object engine 3021A, which is an estimation model for estimating the value of the shape parameter, is generated in response to the input of the silhouette image of the object.

(3-2-2) Operation of Dimension Data Calculator The Dimension Data Calculator 3020 previously obtains the electronic file of the object engine 3021A generated by the learning device 3025 and the projection information of the principal component analysis obtained by the learning device 3025. It is stored in the storage unit 3021 and used for calculating the dimensional data of the object.

First, the acquisition unit 3024A acquires, through the input / output unit 3122, a plurality of image data obtained by photographing the entire object from different directions via an external terminal device or the like together with the total length data indicating the total length of the object ( S3021). Next, the extraction unit 3024B extracts shape data indicating the shape of each part of the object from each image data (S3022). Subsequently, the conversion unit 3024C executes a rescale process for converting each shape data into a predetermined size based on the total length data (S3023). Through the steps S3021 to S3023, a silhouette image of the object is generated, and the dimension data calculation device 3020 receives the silhouette image of the object.

Next, using the object engine 3021A previously stored in the storage unit 3021, the estimation unit 3024D estimates the value of the shape parameter of the object from the received silhouette image (S3024). Then, the calculation unit 3024E calculates the dimensional data related to the portion of the object based on the value of the shape parameter of the object (S3025).

Specifically, in the calculation of the dimensional data in S3025, first, three-dimensional data of the vertices of the object is constructed from the values of a predetermined number of shape parameters estimated by the object engine 3021A for the object. Here, the inverse transformation of the projection (S3010) related to the dimension reduction carried out by the preprocessing unit 3027A in the learning stage may be performed. More specifically, with respect to the values of the shape parameters of the estimated predetermined number (column vector), by multiplying the transposed matrix W ^T of the projection matrix W according to the principal component analysis from the right, it is possible to obtain three-dimensional data.

In the above example, the three-dimensional data X'' of the object can be calculated from the following mathematical formula with respect to the value Λ'' of the shape parameter of the object.

In S3025, the calculation unit 3024E uses the three-dimensional data to calculate the dimensional data between any two vertices in the object. Here, a three-dimensional object is virtually constructed from the three-dimensional data, and dimensional data between two vertices is calculated along a curved surface on the three-dimensional object. That is, the distance between two vertices can be calculated three-dimensionally so as to follow the three-dimensional shape of the three-dimensional object. In order to calculate the distance between two vertices three-dimensionally, first, two vertices are arranged on a three-dimensional mesh composed of a large number of vertices (5,000 three-dimensional data in the above example). Search for the shortest path connecting between them and identify the mesh through which the shortest path passes. Next, using the vertex coordinate data of the specified mesh, the distance is calculated for each mesh along the shortest path and added up. The total value is the three-dimensional distance between the two vertices. For the calculation of the three-dimensional distance, the vertex information of each mesh constituting the three-dimensional object and the mesh data such as the normal direction of each vertex can be used.

For example, assume that the object is a "person" and the "shoulder width" of the person is calculated. As a preliminary preparation, it is defined in advance that "shoulder width = distance between the apex indicating the acromion of the left shoulder and the apex indicating the acromion of the right shoulder". Further, it is associated in advance that the apex number indicating the acromion of the left shoulder is, for example, # 313, and the apex number of the apex indicating the acromion of the right shoulder is, for example, # 521. These pieces of information are stored in the storage unit 3021 in advance. When calculating the dimensional data, the shortest path from the vertex numbers # 313 to # 521 is specified, and the vertex coordinate data of the mesh specified in relation to the shortest path is used to determine the distance for each mesh along the shortest path. Can be calculated and totaled.

As described above, the dimension data calculation device 3020 of the present embodiment can estimate the values of a predetermined number of shape parameters from the silhouette image with high accuracy by using the object engine 3021A. In addition, since the three-dimensional data of the object can be restored with high accuracy from the values of the shape parameters estimated with high accuracy, the measurement target location is not only the specific part but also between any two vertices with high accuracy. Can be calculated. In particular, the calculated dimensional data between the two vertices is highly accurate because it is calculated along a three-dimensional shape based on a three-dimensional object composed of three-dimensional data.

(3-3) Features of Dimension Data Calculation System a) As described above, the dimensional data calculation system 3100 according to the present embodiment includes a dimensional data calculation device 3020 and a learning device 3025. The information processing device configured as a part of the dimension data calculation device 3020 includes a conversion unit (reception unit) 3024C and an estimation unit 3024D. The conversion unit (reception unit) 3024C receives a silhouette image of the object. The estimation unit 3024D uses the object engine 3021A that associates the silhouette image of the sample object with the values of a predetermined number of shape parameters associated with the sample object, and uses the object engine 3021A to obtain the shape parameters of the object from the received silhouette image. Estimate the value. Then, the value of the estimated shape parameter of the object is associated with the dimensional data related to any part of the object.

Further, the dimension data calculation device 3020 includes an acquisition unit 3024A, an extraction unit 3024B, a conversion unit 3024C, an estimation unit 3024D, and a calculation unit 3024E. The acquisition unit 3024A acquires the image data in which the object is photographed and the total length data of the object. The extraction unit 3024B extracts shape data indicating the shape of the object from the image data. The conversion unit 3024C converts the shape data into a silhouette image based on the total length data. The estimation unit 3024D uses the object engine 3021A that associates the silhouette image of the sample object with the value of a predetermined number of shape parameters associated with the sample object, and obtains the value of a predetermined number of shape parameters from the silhouette image. presume. The calculation unit 3024E calculates the dimensional data of the object based on the estimated values of a predetermined number of shape parameters.

Therefore, the dimension data calculation device 3020 can estimate the values of a predetermined number of shape parameters from the silhouette image with high accuracy by using the object engine 3021A created in advance. Further, by using the value of the shape parameter estimated with high accuracy, it is possible to efficiently and accurately calculate the data related to an arbitrary part of the object. As described above, according to the dimension data calculation device 3020, the dimension data calculated for the object can be provided efficiently and with high accuracy.

By using such a dimensional data calculation device 3020, for example, dimensional data of each part of an organism as an object can be calculated with high accuracy. In addition, it is possible to calculate the dimensional data of each part of an arbitrary object such as a car or various kinds of luggage as an object with high accuracy. Further, by incorporating the dimension data calculation device 3020 into a product manufacturing device that manufactures various products, it is possible to manufacture a product that matches the shape of the object.

By using such a dimensional data calculation device 3020, for example, dimensional data related to each part of an organism as an object can be calculated with high accuracy. In addition, it is possible to calculate the dimensional data of each part of an arbitrary object such as a car or various kinds of luggage as an object with high accuracy. Further, by incorporating the dimension data calculation device 3020 into a product manufacturing device that manufactures various products, it is possible to manufacture a product that matches the shape of the object.

b) In such a dimensional data calculation device 3020, a predetermined number of shape parameters associated with the sample object are specified by reducing the three-dimensional data of the sample object. In particular, dimensionality reduction is performed by principal component analysis. As a result, noise can be effectively removed from the three-dimensional data of the sample object, and the three-dimensional data can be compressed.

c) In the dimension data calculation device 3020, the three-dimensional data of the object is calculated by the inverse conversion of the projection according to the above principal component analysis with respect to the estimated predetermined number of shape parameter values, and the three-dimensional data is the dimension data. Associated with. Thereby, the three-dimensional data can be configured with high accuracy for the input of the silhouette image of the object.

d) In the dimension data calculation device 3020, the silhouette image of the sample object is a projection image in a predetermined direction of the three-dimensional object composed of the three-dimensional data of the sample object. That is, a silhouette image can be obtained by constructing a three-dimensional object using the three-dimensional data of the sample object and then projecting the object. It is preferable to acquire two silhouette images in the front direction and the side direction by projection. As a result, a silhouette image of the sample object can be generated with high accuracy.

e) In the dimension data calculation device 3020, the object engine 3021A is generated by learning the relationship between the silhouette image of the sample object and the value of a predetermined number of shape parameters associated with the sample object. The learning can be performed by deep learning. By learning by deep learning, it is possible to associate the silhouette image of the sample object with the value of the shape parameter of the sample object with high accuracy.

f) The calculation unit 3024E of the dimension data calculation device 3020 configures three-dimensional data of a plurality of vertices in the object from the values of a predetermined number of shape parameters estimated for the object. Then, based on the constructed three-dimensional data, the dimensional data between any two vertices in the object is calculated. That is, the dimensional data is calculated after constructing a three-dimensional object using the three-dimensional data of the object. As a result, the dimensional data between the two vertices can be calculated from the shape of the three-dimensional object of the object, so that the measurement target location is not limited to a specific portion.

In particular, in the calculation unit 3024E of the dimension data calculation device 3020, the dimension data between the two vertices is calculated along the curved surface on the three-dimensional object composed of the three-dimensional data of a plurality of vertices in the object. .. As a result, the dimensional data can be calculated with higher accuracy.

(3-4) Modification example a) In the above description, the acquisition unit 3024A acquires a plurality of image data obtained by photographing the object from different directions. Here, as an imaging device capable of simultaneously photographing an object from a plurality of different directions, a depth data measuring device capable of acquiring depth data together can be applied. An example of a depth data measuring device is a stereo camera. As used herein, the term "stereo camera" means an image pickup device of any form that reproduces binocular parallax by simultaneously photographing an object from a plurality of different directions. On the other hand, a plurality of image data are not always required, and it is possible to calculate the dimensional data of each part even if the image data of the object is one.

b) In the above description, the grab cut algorithm may be adopted to separate the object and the background image other than the object. In addition to this, for example, when the stereo camera described above is used in the acquisition unit 3024A, it is possible to separate the object and the background image other than the object by using the depth data acquired from the stereo camera. Is. This makes it possible to generate shape data of the object with higher accuracy. Further, in addition to the stereo camera, it is possible to obtain depth data by a LiDAR (Light Detection and Ranging) device and separate an object and a background image other than the object. That is, by using an arbitrary machine (depth data measuring device) capable of measuring depth data in the acquisition unit 3024A, it is possible to generate shape data of an object with high accuracy.

c) In the above description, in S3013, the preprocessing unit 3027A performed a data expansion process for expanding the shape parameter data set using random numbers. In the data expansion process, it is sufficient to determine how many the data expansion process should be expanded according to the number of sample objects. If a sufficient number of samples are prepared in advance, the expansion process of S3013 may not be performed.

c) In the above description, the shape parameters (λ ₁ , ..., λ ₃₀ ) were obtained by principal component analysis in the pretreatment unit 3027A of the learning device 3025 for the sample object. Here, the shape parameters when the sample object is a "human" will be further considered. As a result of principal component analysis using three-dimensional data of 400 sample objects and 5,000 vertices as in the above example, the shape parameters when the object is "human" are at least as follows. It was considered to have properties.

[Characteristic 1]
The first-ranked principal component λ ₁ was associated to have a linear relationship with human height. Specifically, as shown in FIG. 14, the larger the main component λ ₁ of the first rank, the smaller the height of a person.

Considering the characteristic 1, when estimating the value of the human shape parameter by the estimation unit 3024D, the height data acquired by the acquisition unit 3024A without using the object engine 3021A is obtained for the first-order main component λ _1. You can use it. Specifically, the value of the first-ranked principal component λ ₁ may be configured to be calculated separately by using a linear regression model using the height of a person as an explanatory variable.

In this case, the main component λ ₁ may be excluded from the learning target even when the object engine 3021A is generated by the learning unit 3027B in the learning stage. As described above, in the learning stage of the learning device 3025, the network architecture is weighted. At this time, there is an error between the _{second and} subsequent principal components excluding the first-rank principal component obtained by principal component analysis of the input silhouette image and the values after the shape parameter λ ₂ which is training data. The network architecture weighting factor may be set to be minimized. As a result, when the object is a "human", the estimation accuracy of the value of the shape parameter in the estimation unit 3024D can be improved in addition to the use of the linear regression model.

Further, when the object engine 3021A is generated in the learning unit 3027B in the learning stage, the weight of the network architecture is minimized so that the error from the value after the shape parameter λ ₁ is minimized including the first-ranked principal component. A coefficient may be set. Then, the value of the first-ranked principal component λ ₁ may be replaced with a separately calculated value by using a linear regression model using the height of a person as an explanatory variable. As a result, when the object is a "human", the estimation accuracy of the value of the shape parameter in the estimation unit 3024D can be improved in addition to the use of the linear regression model.

[Characteristic 2]
FIG. 15 is a schematic graph showing the recall rate of the shape parameter which is the main component. In the graph, the horizontal axis shows the principal components ranked by the contribution rate, and the vertical axis shows the variance explanation rate of the eigenvalues. Then, the bar graph shows the individual variance explanation rate for each rank. In addition, the solid line graph shows the accumulation of the variance explanation rate from the first rank. In FIG. 15, for the sake of simplicity, a graph relating to the 10 principal components up to the 10th rank is shown schematically. The eigenvalue of the covariance matrix obtained in the principal component analysis represents the magnitude of the eigenvector (principal component), and the variance explanation rate of the eigenvalue can be considered as the recall rate for the principal component.

With reference to the graph of FIG. 15, the cumulative variance explanation rate of the 10 principal components from the 1st rank to the 10th rank shows about 0.95 (dashed line arrow). That is, those skilled in the art will understand that the recall rate of the 10 main components from the 1st rank to the 10th rank is about 95%. That is, in the above example, the shape parameter whose characteristics are converted by dimension reduction is 30 dimensions, but the shape parameter is not limited to this, and even if the shape parameter is 10 dimensions, about 95% can be covered. That is, although the number of shape parameters (number of dimensions) is 30 in the above description, it may be about 10 in consideration of the characteristic 2.

d) In the above description, in the pretreatment unit 3027A of the learning device 3025, two silhouette images in the front direction and the side direction were acquired by projection for each 10,000 sample objects. On the other hand, two silhouette images of the object are not always required, and one image is sufficient.

(3-5) Application to Product Manufacturing System An example of applying the above-mentioned dimension data calculation device 3020 to the product manufacturing system 3001 will be described below.

(3-5-1) Configuration of Product Manufacturing System FIG. 16 is a schematic diagram showing the concept of the product manufacturing system 3001 according to the present embodiment. The product manufacturing system 3001 includes a dimension data calculation device 3020 capable of communicating with the terminal device 3010 owned by the user 3005, and a product manufacturing device 3030, and is a system for manufacturing a desired product 3006. In FIG. 16, as an example, the concept when the object 3007 is a person and the product 3006 is a chair is shown. However, in the product manufacturing system 3001 according to the present embodiment, the object 3007 and the product 3006 are not limited thereto.

The terminal device 3010 can be realized by a so-called smart device. Here, the terminal device 3010 exerts various functions by installing a user program on the smart device. Specifically, the terminal device 3010 generates image data captured by the user 3005. Here, the terminal device 3010 may have a stereo camera function that reproduces binocular parallax by simultaneously photographing an object from a plurality of different directions. The image data is not limited to the one taken by the terminal device 3010, and for example, the image data taken by a stereo camera installed in the store may be used.

Further, the terminal device 3010 accepts the input of the attribute data indicating the attribute of the object 3007. The "attribute" includes the total length, weight, elapsed time (including age), etc. of the object 3007. Further, the terminal device 3010 has a communication function, and executes transmission / reception of various information between the dimension data calculation device 3020, the product manufacturing device 3030, and the terminal device 3010.

The dimensional data calculation device 3020 can be realized by any computer. Here, the storage unit 3021 of the dimension data calculation device 3020 stores the information transmitted from the terminal device 3010 in association with the identification information that identifies the user 3005 of the terminal device 3010. In addition, the storage unit 3021 stores parameters and the like necessary for executing information processing for calculating dimensional data.

Further, as described above, the processing unit 3024 of the dimension data calculation device 3020 functions as the acquisition unit 3024A, the extraction unit 3024B, the conversion unit 3024C, the estimation unit 3024D, and the calculation unit 3024E. Here, the acquisition unit 3024A acquires the image data taken by the user 3005 with the stereo camera and the attribute data of the object 3007. Further, the extraction unit 3024B extracts shape data indicating the shape of the object 3007 from the image data. For example, when "person" is preset as the type of object, a semantic segmentation algorithm is constructed using training data for identifying a person. Further, the extraction unit 3024B preferably uses the depth data acquired from the stereo camera to separate the object and the background image other than the object. The conversion unit 3024C converts the shape data into a silhouette image based on the total length data. The conversion unit 3024C also functions as a reception unit for inputting the generated silhouette image to the estimation unit 3024D.

The estimation unit 3024D uses the object engine 3021A that associates the silhouette image of the sample object with the value of a predetermined number of shape parameters associated with the sample object, and obtains the value of a predetermined number of shape parameters from the silhouette image. presume. The calculation unit 3024E calculates the dimensional data of the object based on the estimated values of a predetermined number of shape parameters. Specifically, three-dimensional data of a plurality of vertices in the object is constructed from the values of the shape parameters of the object estimated by the estimation unit 3024D, and further, any two in the object based on the three-dimensional data. Calculate the dimensional data between two vertices.

The product manufacturing apparatus 3030 is a manufacturing apparatus that manufactures a desired product related to the shape of the object 3007 by using at least one dimensional data calculated by using the dimensional data calculating device 3020. As the product manufacturing apparatus 3030, any apparatus capable of automatically manufacturing and processing a product can be adopted, and can be realized by, for example, a three-dimensional printer or the like.

(3-5-2) Operation of Product Manufacturing System FIG. 17 is a sequence diagram for explaining the operation of the product manufacturing system 3001 according to the present embodiment. 18 and 19 are schematic views showing screen transitions of the terminal device 3010.

First, the entire object 3007 is imaged a plurality of times via the terminal device 3010 so as to be captured from different directions, and a plurality of image data in which the object 3007 is imaged is generated (T3001). Here, a plurality of front and side photographs as shown in FIGS. 18 and 19, respectively, are taken. Such front and side photographs should be taken with the stereo camera function of the terminal device 3010 turned on.

Next, the user 3005 inputs attribute data indicating the attributes of the object 3007 to the terminal device 3010 (T3002). Here, as the attribute data, the total length data, weight data, time-lapse data (including age, etc.) of the object 3007 and the like are input. Then, these plurality of image data and attribute data are transmitted from the terminal device 3010 to the dimension data calculation device 3020.

When the dimensional data calculation device 3020 receives a plurality of image data and attribute data from the terminal device 3010, the dimensional data calculation device 3020 calculates the dimensional data of each part of the object 3007 using these data (T3003). The terminal device 3010 displays dimensional data on the screen according to the setting. Then, the product manufacturing apparatus 3030 manufactures a desired product 3006 based on the dimensional data calculated by the dimensional data calculating apparatus 3020 (T3004).

(3-5-3) Features of Product Manufacturing System As described above, the product manufacturing system 3001 according to the present embodiment includes a dimension data calculation device 3020 capable of communicating with the terminal device 3010 owned by the user 3005 and product manufacturing. The device 3030 is provided.

The terminal device 3010 (photographing device) captures a plurality of images of the object 3007. The dimension data calculation device 3020 includes an acquisition unit 3024A, an extraction unit 3024B, a conversion unit 3024C, an estimation unit 3024D, and a calculation unit 3024E. The acquisition unit 3024A acquires the image data in which the object is photographed and the total length data of the object. The extraction unit 3024B extracts shape data indicating the shape of the object from the image data. The conversion unit 3024C converts the shape data into a silhouette image based on the total length data. The estimation unit 3024D uses the object engine 3021A that associates the silhouette image of the sample object with the value of a predetermined number of shape parameters associated with the sample object, and obtains the value of a predetermined number of shape parameters from the silhouette image. presume. The calculation unit 3024E calculates the dimensional data of the object based on the estimated values of a predetermined number of shape parameters. The product manufacturing apparatus 3030 manufactures the product 3006 using the dimensional data calculated by the calculation unit 3024E. With such a configuration, the dimensional data calculation device 3020 calculates each part of the object 3007 with high accuracy, so that a desired product related to the shape of the object 3007 can be provided.

For example, the product manufacturing system 3001 can manufacture a model of an organ by measuring the shape of various organs such as the heart. Further, for example, various healthcare products can be manufactured by measuring the waist shape of a person. Further, for example, a figure product of the person can be manufactured from the shape of the person. Further, for example, a chair or the like suitable for the person can be manufactured from the shape of the person. Also, for example, a car toy can be manufactured from the shape of a car. Further, for example, a diorama or the like can be manufactured from an arbitrary landscape painting.

In the above description, the dimension data calculation device 3020 and the product manufacturing device 3030 are described as separate member devices, but these may be configured as one.

<Fourth Embodiment>
Hereinafter, the configurations and functions already described will be designated by substantially the same reference numerals and description thereof will be omitted.
(4-1) Configuration of Dimension Data Calculation Device FIG. 20 is a schematic view showing the configuration of the dimension data calculation system 4200 according to the present embodiment. The dimensional data calculation system 4200 includes a dimensional data calculation device 4120 and a learning device 4125.

The dimensional data calculation device 4120 includes a storage unit 4121, an input / output unit 4122, a communication unit 4123, and a processing unit 4124. Further, the learning device 4125 includes a storage unit 4126 and a processing unit 4127. The dimension data calculation device 4120 and the learning device 4125 may be realized as hardware by using an LSI, an ASIC, an FPGA, or the like.

The storage units 4121 and 4126 all store various types of information, and are realized by any storage device such as a memory and a hard disk. For example, the storage unit 4121 stores various data, programs, information, and the like including the object engine 4121A in order to execute information processing related to the calculation of dimensional data in the processing unit 4124. In addition, the storage unit 4126 stores training data used in the learning stage in order to generate the object engine 4121A.

The input / output unit 4122 has the same configuration and function as the above-mentioned input / output unit 3022. Further, the communication unit 4123 has the same configuration and function as the communication unit 3023 described above.

The processing unit 4124 functions as the acquisition unit 4124A, the estimation unit 4124D, and the calculation unit 4124E by reading the program stored in the storage unit 4121 into the CPU, GPU, or the like of the computer. Similarly, the processing unit 4127 functions as the preprocessing unit 4127A and the learning unit 4127B by reading the program stored in the storage unit 4126 into the CPU, GPU, or the like of the computer.

In the processing unit 4124 of the dimension data calculation device 4120, the acquisition unit 4124A acquires the attribute data including at least one of the total length data, the weight data, and the time-lapse data (including age) of the object. In the present embodiment, the acquisition unit 4124A also functions as a reception unit for inputting attribute data to the estimation unit 4124D.

The estimation unit 4124D estimates the values of a predetermined number of shape parameters from the attribute data. The object engine 4121A is used for the estimation. The value of the shape parameter of the object estimated by the estimation unit 4124D can be associated with the dimensional data related to any part of the object, as will be described later.

The calculation unit 4124E calculates the dimensional data of the object from the value of the shape parameter of the object estimated by the estimation unit 4124D. Specifically, the calculation unit 4124E constitutes three-dimensional data of a plurality of vertices in the object from the values of the shape parameters of the object estimated by the estimation unit 4124D, and further, the object is based on the three-dimensional data. Calculate dimensional data between any two vertices in an object.

In the processing unit 4127 of the learning device 4125, the preprocessing unit 4127A carries out various preprocessing for learning. In particular, the preprocessing unit 4127A specifies a predetermined number of shape parameters by extracting features of the three-dimensional data of the sample object by dimensionality reduction. The value of the shape parameter of the sample object and the corresponding attribute data are stored in advance in the storage unit 4126 as training data.

It is assumed that corresponding attribute data (total length data, weight data, time-dependent data (including age, etc.)) is prepared as three-dimensional data of the sample object. The corresponding attribute data is stored in the storage unit 4126 as training data.

The learning unit 4127B learns to associate the value of the shape parameter of the sample object with the corresponding attribute data. As a result of learning, the object engine 4121A is generated. The generated object engine 4121A can be held in the form of an electronic file. When the dimensional data calculation device 4120 calculates the dimensional data of the object, the object engine 4121A is stored in the storage unit 4121 and referred to by the estimation unit 4124D.

(4-2) Operation of Dimension Data Calculation System The operation of the dimensional data calculation system 4200 of FIG. 20 will be described with reference to FIGS. 21 and 22. FIG. 21 is a flowchart showing the operation (S4110) of the learning device 4125, and here, the object engine 4121A is generated based on the sample object data. FIG. 22 is a flowchart showing the operation of the dimension data calculation device 4120, and here, the dimension data of the object is calculated based on the image data of the object.

(4-2-1) Operation of Learning Device First, the data of the sample object is prepared and stored in the storage unit 4126 (S4111). In one example, the data to be prepared is the data of 400 sample objects, and 5,000 3D data prepared for each sample object and attribute data prepared for each sample object. Including. The three-dimensional data includes three-dimensional coordinate data of the vertices of the sample object. Further, the three-dimensional data may include the vertex information of each mesh constituting the three-dimensional object and the mesh data such as the normal direction of each vertex.

Further, as in the first embodiment, the three-dimensional data of the sample object is associated with the information of the part together with the vertex number.

Subsequently, the preprocessing unit 4127A performs feature conversion into a predetermined number (dimensions) of shape parameters by reducing the dimensions (S4112). This feature conversion process is also the same as that of the first embodiment. In the above example, as a result of the matrix operation using the projection matrix related to the principal component analysis, the 15,000-dimensional (5,000 × 3) data possessed by each of the 400 sample objects is, for example, a 30-dimensional principal component. The feature is converted to the shape parameter Λ of.

Then, the learning unit 4127B uses a set of the attribute data of the plurality of sample objects prepared in S4111 and the data set of the plurality of shape parameters obtained in S4112 as training data, and machine-learns the relationship between the two. (S4115).

Specifically, the learning unit 4027B obtains the conversion attribute data Y from the attribute data of the object. The element of the transformation matrix Z that associates the element y _r of the transformation attribute data Y with the element λ _m of the shape parameter Λ is expressed as z _rm . The transformation matrix Z is a matrix consisting of [s rows, n columns]. Further, the symbol m is 1 ≦ m ≦ n, and in the above example, n is 30 which is the number of dimensions of the shape parameter Λ. The symbol r is 1 ≦ r ≦ s, and s is the number of elements used in the calculation obtained from the conversion attribute data Y.

For example, it is assumed that the attribute data of the object is composed of the total length data h, the weight data w, and the time data a. That is, the attribute data is a set of elements (h, w, a). In this case, the learning unit 4027B uses the squared value (also referred to as a quadratic term) of each element (h, w, a) of the attribute data of the object and the multiplied value of each element (also referred to as an interaction term). .) And the value of each element itself (also called the primary term).

As a result, the conversion attribute data Y having the following nine elements is obtained.

Next, the learning unit 4027B regresses the pair of the transformation attribute data Y obtained from the attribute data associated with the 400 sample objects and the shape parameter Λ obtained from the three-dimensional data of the sample objects. As a result, the transformation matrix Z consisting of [9 rows, 30 columns] as shown below is obtained.

The data of the transformation matrix Z obtained in this way is stored in the storage unit 4026 as the object engine 4121A.

(4-2-2) Operation of Dimension Data Calculator The Dimension Data Calculator 4120 stores the electronic file of the object engine 4121A generated by the learning device 4125 and the projection information of the principal component analysis obtained by the learning device 4125. It is stored in the unit 4121 and used for calculating the dimensional data of the object.

First, the acquisition unit 4124A acquires the attribute data of the object through the input / output unit 4122 (S4121). As a result, the attribute data of the object is accepted. Next, using the object engine 4121A previously stored in the storage unit 4121, the estimation unit 4124D estimates the value of the shape parameter of the object from the received attribute data (S4124).

For example, it is assumed that the attribute data of the object is composed of the total length data h, the weight data w, and the time data a. That is, the attribute data is a set of elements (h, w, a). In S4124, as described above, the estimation unit 4124D includes the squared value of each element (h, w, a) of the attribute data of the object, the product of each element, and the value of each element itself. Obtain the conversion attribute data Y consisting of.

Finally, the calculation unit 4124E calculates the dimensional data related to the portion of the object based on the value of the shape parameter of the object (S4125). Specifically, the conversion attribute data Y is calculated from the attribute data of the object acquired by the acquisition unit 4124A. Then, the shape parameter Λ is calculated by multiplying the transformation attribute data Y by the transformation matrix Z described above. After that, as in the case of the third embodiment (S3025), the 3D object is virtually constructed from the 3D data, and the dimensional data between the two vertices is calculated along the curved surface on the 3D object. To do. For the calculation of the three-dimensional distance, the vertex information of each mesh constituting the three-dimensional object and the mesh data such as the normal direction of each vertex can be used.

As described above, the dimension data calculation device 4120 of the present embodiment can estimate the values of a predetermined number of shape parameters from the attribute data of the object with high accuracy by using the object engine 4121A. Unlike the third embodiment, there is no need to input an image of the object, and S3022 (shape data extraction process) and S3023 (rescale process) of FIG. 13 are not required, which is efficient.

In addition, since the three-dimensional data of the object can be restored with high accuracy from the values of the shape parameters estimated with high accuracy, the measurement target location is not only the specific part but also between any two vertices with high accuracy. Can be calculated. In particular, the calculated dimensional data between the two vertices is highly accurate because it is calculated along the three-dimensional shape based on the three-dimensional object composed of the three-dimensional data.

(4-3) Features of Dimension Data Calculation System As described above, the dimensional data calculation system 4200 according to the present embodiment includes a dimensional data calculation device 4120 and a learning device 4125. The information processing device configured as a part of the dimension data calculation device 4120 includes an acquisition unit (reception unit) 4124A, an estimation unit 4124D, and a calculation unit 4124E. The acquisition unit (reception unit) 4124A receives the attribute data of the object. The estimation unit 4124D uses the object engine 4121A that associates the attribute data of the sample object with the values of a predetermined number of shape parameters associated with the sample object, and uses the object engine 4121A to obtain the shape parameters of the object from the received attribute data. Estimate the value of. Then, the value of the estimated shape parameter of the object is associated with the dimensional data related to any part of the object.

Therefore, the dimension data calculation device 4120 can efficiently estimate the values of a predetermined number of shape parameters from the attribute data by using the object engine 4121A created in advance. In addition, the estimated shape parameter values are highly accurate. Further, by using the value of the shape parameter estimated with high accuracy, it is possible to efficiently and accurately calculate the data related to an arbitrary part of the object. As described above, according to the dimensional data calculation device 4120, the dimensional data calculated for the object can be provided efficiently and with high accuracy.

(4-4) Application to Product Manufacturing System FIG. 23 is a schematic diagram showing the concept of the product manufacturing system 4001S according to the present embodiment. The dimensional data calculation device 4120 according to the present embodiment can also be applied to the product manufacturing system 4001S in the same manner as the dimensional data calculation device 3020 according to the third embodiment.

The terminal device 4010S according to the present embodiment may be any as long as it accepts the input of attribute data indicating the attributes of the object 4007. Examples of the "attribute" include the total length, weight, elapsed time (including age), and the like of the object 1007.

Further, as described above, the processing unit 4124 of the dimension data calculation device 4120 functions as the acquisition unit 4124A, the estimation unit 4124D, and the calculation unit 4124E. The calculation unit 4124E calculates the dimensional data related to the portion of the object based on the value of the shape parameter of the object obtained by the estimation unit 4124D.

In the product manufacturing system 4001S, since the dimension data calculation device 4120 efficiently and highly accurately calculates the dimension data of the object 1007, it is possible to provide a desired product related to the shape of the object 1007. In addition, the product manufacturing system 4001S according to the fourth embodiment can exhibit the same effect as the product manufacturing system 3001 of the third embodiment.

<Other embodiments>
The present disclosure is not limited to each of the above embodiments as it is. In the present disclosure, the components can be modified and embodied without departing from the gist at the implementation stage. In addition, the present disclosure can form various disclosures by appropriately combining a plurality of components disclosed in each of the above embodiments. For example, some components may be removed from all the components shown in the embodiments. Further, the components may be appropriately combined in different embodiments.

1001 Product manufacturing system 1010 Terminal device 1020 Dimension data calculation device 1021 Storage unit 1022 Input / output unit 1023 Communication unit 1024 Processing unit 1024A Acquisition unit 1024B Extraction unit 1024C Conversion unit 1024D Calculation unit 1030 Product manufacturing device 2001S Product manufacturing system 2120 Dimension data calculation device 2121 Storage unit 2122 Input / output unit 2123 Communication unit 2124 Processing unit 2124A Acquisition unit 2124D Calculation unit 3001 Product manufacturing system 3020 Dimension data calculation device 3021 Storage unit 3022 Input / output unit 3023 Communication unit 3024 Processing unit 3024A Acquisition unit 3024B Extraction unit 3024C Conversion unit 3024D Estimating unit 3024E Calculation unit 3025 Learning device 3026 Storage unit 3027 Processing unit 3027A Preprocessing unit 3027B Learning unit 3100 Dimension data calculation system 4001S Product manufacturing system,
4120 Dimension data calculation device 4121 Storage unit 4122 Input / output unit 4123 Communication unit 4124 Processing unit 4124A Acquisition unit 4124D Estimating unit 4124E Calculation unit 4125 Learning device 4126 Storage unit 4127 Processing unit 4127A Preprocessing unit 4127B Learning unit 4200 Dimension data calculation system

JP-A-2017-018158

Claims (5)

An acquisition unit that acquires the image data of the object and the total length data of the object,
An extraction unit that extracts shape data indicating the shape of the object from the image data,
A conversion unit that converts the shape data based on the total length data,
A calculation unit that calculates dimensional data of each part of the object using the shape data converted by the conversion unit, and a calculation unit.
A dimensional data calculation device including. An acquisition unit that acquires attribute data including at least one of the total length data, weight data, and temporal data of the object, and
A calculation unit that calculates dimensional data of each part of the object using the attribute data,
A dimensional data calculation device including. The reception desk that accepts silhouette images of objects and
Using an object engine that associates a silhouette image of a sample object with a predetermined number of shape parameter values associated with the sample object, the shape parameter values of the object are estimated from the received silhouette image. And the estimation part
An information processing apparatus comprising: The estimated value of the shape parameter of the object is associated with dimensional data associated with any part of the object. The reception section that accepts the attribute data of the object and
The value of the shape parameter of the object is estimated from the received attribute data by using the object engine in which the attribute data of the sample object and the value of a predetermined number of shape parameters associated with the sample object are associated with each other. And the estimation part to do
An information processing apparatus comprising: The estimated value of the shape parameter of the object is associated with dimensional data associated with any part of the object. An acquisition unit that acquires the image data of the object and the total length data of the object,
An extraction unit that extracts shape data indicating the shape of the object from the image data,
A conversion unit that converts the shape data into a silhouette image based on the total length data,
An estimation unit that estimates the values of a predetermined number of shape parameters from the silhouette image using an object engine that associates the silhouette image of the sample object with the values of a predetermined number of shape parameters associated with the sample object. When,
A calculation unit that calculates dimensional data of the object based on the estimated values of a predetermined number of shape parameters, and
A dimensional data calculation device including.