JPWO2019049298A1 - 3D data system and 3D data processing method - Google Patents

Abstract

The 3D data system (1) includes an image acquisition unit (11) that acquires a captured image obtained by capturing the surface of a target subject, and distance information indicating a distance from a reference point to the surface, and the captured image and the distance. An addition unit (21) for adding individuality data indicating the dynamic individuality of the 3D model to the 3D modeling data indicating the 3D model of the subject subject obtained from information, and the individuality data added thereto And a storage unit (22) for storing 3D modeling data.

Description

  The present invention relates to a technique for handling 3D modeling data.

  A technique is known in which a person or an article is photographed to form 3D data and used in a virtual space. Patent Document 1 describes a technique for causing a human 3D model to take different poses or perform motions. More specifically, in Patent Document 1, a subject is captured with a basic pose and other modified poses taken, 3D modeling data of the subject is generated for each pose, and models are compared between different poses. The detection of the fulcrum of rotation of each part is described.

JP 2002-58045 A

  According to the technique of Patent Document 1, individuality other than appearance cannot be given to the 3D modeling data. In contrast, the present invention provides a technique for giving individuality other than appearance to 3D modeling data.

  The present invention provides a captured image obtained by capturing the surface of a subject to be processed, an image acquisition unit that acquires distance information indicating a distance from a reference point to the surface, and the target obtained from the captured image and the distance information. 3D data including an adding unit for adding individuality data indicating dynamic individuality of the 3D modeling data to the subject 3D modeling data, and a storage unit for storing the 3D modeling data to which the individuality data is added Provide a system.

  The 3D data system includes an estimation unit that estimates the individuality of the target subject using the 3D modeling data, and the addition unit uses the individuality data to indicate data indicating the individuality estimated by the estimation unit. It may be added as

  The 3D data system includes a feature acquisition unit that acquires a plurality of feature points from the captured image, and a calculation unit that calculates a feature amount related to a positional relationship between the plurality of feature points, and the estimation unit includes the feature The individuality of the subject subject may be estimated using the quantity.

  The 3D data system may include a classification unit that classifies the target subject into any of a plurality of groups, and the estimation unit may estimate the individuality of the subject based on the classification.

  The classifying unit classifies the plurality of existing subjects into the plurality of groups based on the feature amounts obtained from the captured images of the plurality of subjects, and sets the subject subjects as the plurality of existing subjects. You may classify | categorize into either of the several groups obtained based on the to-be-photographed object's feature-value.

  The feature acquisition unit acquires a static feature amount of the subject from the captured image, and the classification unit classifies the subject into one of the plurality of groups using the static feature amount. May be.

  The image acquisition unit acquires a plurality of captured images each having a different pose with respect to the subject, the feature acquisition unit acquires a dynamic feature amount of the subject from the plurality of captured images, and the classification unit The subject may be classified into any of the plurality of groups using the dynamic feature amount.

  Each of the plurality of groups has representative individuality data indicating the individuality representing the group, and the estimation unit estimates the individuality indicated by the representative individuality data of the group to which the subject belongs as the individuality of the subject. May be.

  Each of the plurality of groups has a representative feature amount representing a feature amount representing the group and representative individuality data representing individuality representing the group, and the feature obtained from the representative feature amount and the subject. A correction unit that corrects the representative individuality data according to a difference from the amount, and the estimation unit estimates the individuality indicated by the representative individuality data corrected by the correction unit as the individuality of the subject. Also good.

  The classifying unit classifies a plurality of existing subjects into the plurality of groups based on feature amounts obtained from captured images of the plurality of subjects, and the estimation unit is configured for each of the plurality of groups. Individual data obtained from a plurality of photographed images with different poses acquired for at least a part of existing subjects belonging to the group may be used as the representative personality data.

  The classification unit classifies the subject according to at least one level selected from a plurality of levels with different numbers of groups after the classification, and the estimation unit includes a first level of the plurality of levels. In the classification, the individuality of the subject is estimated based on the difference between the representative value of the feature amount in the group after the classification and the feature amount of the subject, and the group after the classification is more than the first level. In the second-level classification with a large number, the individuality representing the group after the classification may be estimated as the individuality of the subject.

  The present invention also includes a step of acquiring a photographed image obtained by photographing a surface of a subject, distance information indicating a distance from a reference point to the surface, and 3D modeling data of the subject obtained from the photographed image and the distance information. In contrast, the present invention provides a 3D data processing method comprising the steps of adding dynamic personality data of the 3D modeling data and storing the 3D modeling data to which the personality data is added.

  Furthermore, the present invention provides a captured image obtained by capturing the surface of a subject to be processed, an image acquisition unit that acquires distance information indicating a distance from a reference point to the surface, and the acquired image and the distance information. An addition unit for adding bone data indicating bone structure for giving motion to the 3D model and metadata indicating attributes of the 3D model to 3D modeling data indicating the 3D model of the subject to be processed; and the bone data And a storage unit for storing the 3D modeling data to which metadata is added.

  The metadata may include data that restricts the position or angle of view of a virtual camera used when displaying the 3D model located in the virtual space.

  The metadata may include audio data of the 3D model.

  According to the present invention, individuality other than appearance can be given to 3D modeling data.

The figure which shows the outline of the 3D data system 1 which concerns on one Embodiment. The figure which illustrates the function structure of 3D data system 1. 1 is a diagram illustrating a configuration of a 3D data input system 10. The figure which illustrates the hardware constitutions of 3D data processing system 20. The figure which illustrates the operation | movement which concerns on one Embodiment of 3D data system. The figure which illustrates the details of processing which generates 3D modeling data on a subject. The figure which illustrates adjustment of a standard model. The figure which illustrates the positional relationship of a feature point. The figure which illustrates the detail of the process which adds individuality to 3D modeling data. The figure which illustrates classification of 3D modeling data. The figure which illustrates the motion of 3D modeling data. The figure which illustrates the feature point extracted from a face. The figure which illustrates the detail of the process using 3D modeling data. The figure which shows the outline of the 3D data system 1 which concerns on the modification 1. FIG.

DESCRIPTION OF SYMBOLS 1 ... 3D data system, 10 ... 3D data input system, 11 ... Acquisition part, 12 ... Generation part, 13 ... Acquisition part, 20 ... 3D data processing system, 21 ... Addition part, 22 ... Storage part, 23 ... Classification part, 24 ... Estimation unit, 30 ... Application, 201 ... CPU, 202 ... Memory, 203 ... Storage, 204 ... Network IF

1. Configuration FIG. 1 is a diagram schematically illustrating a 3D data system 1 according to an embodiment. The 3D data system 1 generates 3D modeling data using an image obtained by capturing the surface of a subject (hereinafter referred to as “captured image”), stores the 3D modeling data, and further provides 3D modeling data to an application as required. It is. “Subject” refers to an object for which 3D modeling data is generated, and includes living things such as humans and animals, and inanimate objects such as dolls and furniture. 3D modeling data refers to data for displaying a 3D model. A 3D model refers to solid data in a three-dimensional virtual space. The 3D model includes at least information regarding the surface shape and color of the subject.

  The 3D data system 1 includes a 3D data input system 10, a 3D data processing system 20, and an application 30. The 3D data input system 10 generates 3D modeling data from the captured image. The 3D data input system 10 includes, for example, a so-called 3D scanner. The 3D data processing system 20 processes and stores the 3D modeling data generated by the 3D data input system 10. The application 30 provides a user with a product or service using 3D modeling data. In the figure, one 3D data input system 10 and one application 30 are shown, but the 3D data system 1 may include a plurality of at least one of the 3D data input system 10 and the applications 30. As an example, the 3D data input system 10 is a local system, and the 3D data processing system 20 is a system on a network, a so-called cloud system.

  FIG. 2 is a diagram illustrating a functional configuration of the 3D data system 1. The 3D data system 1 includes an acquisition unit 11, a generation unit 12, an addition unit 21, a storage unit 22, and an output unit 25. The acquisition unit 11 acquires a captured image and distance information (an example of an image acquisition unit). The captured image is an image obtained by capturing the surface of the subject. The distance information is information indicating the distance from the reference point to the surface of the subject. The acquisition unit 11 acquires a captured image and distance information from an imaging device such as a camera and a distance sensor (both not shown in FIG. 2). The generation unit 12 generates 3D modeling data of the subject using the captured image and the distance information. The adding unit 21 adds the dynamic personality data of the 3D modeling data to the 3D modeling data generated by the generating unit 12. Individuality data is data indicating the individuality of the 3D model (in many cases, it corresponds to the individuality of the subject). In this example, the personality data indicates a dynamic personality. The storage unit 22 stores various data, for example, 3D modeling data to which individuality data is added. The output unit 25 outputs 3D modeling data and individuality data in response to a request from the application 30.

  The 3D data system 1 further includes an estimation unit 24. The estimation unit 24 estimates the individuality of the subject subject using the 3D modeling data. The individuality data added by the adding unit 21 to the 3D modeling data includes data indicating the individuality estimated by the estimating unit 24.

  The 3D data system 1 further includes an acquisition unit 13 and a calculation unit 14. The acquisition unit 13 acquires a plurality of feature points from 3D modeling data (or from a captured image) (an example of a feature acquisition unit). These feature points refer to points (to be described later with reference to FIG. 7 and the like) indicating features related to the shape of the 3D model (or subject). The calculation unit 14 calculates a feature amount related to the positional relationship between a plurality of feature points. The estimation unit 24 estimates the individuality of the subject subject using this feature amount.

  The 3D data system 1 further includes a classification unit 23. The classification unit 23 classifies the subject into any of a plurality of groups. The estimation unit 24 estimates the individuality of the subject subject based on this classification.

  In one example, the feature amount related to the positional relationship between a plurality of feature points indicates a static feature of the subject (3D model). Static features are features obtained from a single pose subject. On the other hand, the dynamic feature refers to a feature (to be described later with reference to FIG. 11) for giving a plurality of poses (or motions) to the 3D model. As already described, the individuality estimated by the estimation unit 24 is a dynamic individuality of the subject, that is, a dynamic feature. In this example, the estimation unit 24 estimates a dynamic feature using a static feature.

  In this example, the acquisition unit 11, the generation unit 12. The acquisition unit 13 and the calculation unit 14 are implemented in the 3D data input system 10, and the addition unit 21, the storage unit 22, the classification unit 23, and the estimation unit 24 are implemented in the 3D data processing system 20. Note that the sharing of functions in the 3D data input system 10 and the 3D data processing system 20 is merely an example, and the sharing of functions is not limited to the example of FIG.

  FIG. 3 is a diagram illustrating a configuration of the 3D data input system 10. In this example, the 3D data input system 10 is a so-called 3D scanner. The 3D data input system 10 includes a sensor group SD, a stage T, a frame F, and an image processing apparatus 100. The sensor group SD includes a plurality of sets of cameras C and distance sensors D. In a state where the sensor group SD is fixed, each camera C captures only a limited part of the surface of the subject S. The distance sensor D detects the distance from the position (an example of the reference position) where the distance sensor D is installed to the surface of the subject S. The distance sensor D includes a projection unit that projects an image of a predetermined pattern figure (for example, a lattice) at a wavelength outside visible light such as infrared light, and an imaging unit that reads the projected image. The camera C and the distance sensor D in a set are fixed to a common mount base M, and the optical axes of both are facing substantially the same position. The frame F (sensor group SD) rotates relative to the stage T. The stage T may rotate with the frame F fixed to the installation surface, or the frame F may rotate around the stage T with the stage T fixed to the installation surface. While the frame F and the stage T are relatively rotated, the camera C captures the subject S. In a state where the frame F and the stage T are relatively stationary, the sensor group SD covers only a limited part of the surface of the subject S. However, the sensor group SD captures the entire area of the surface of the subject S by rotating the frame F and the stage T relative to each other by 360 ° and continuously capturing images during that time.

  3 is merely an example of the configuration of the 3D data input system 10, and the 3D data input system 10 is not limited to this configuration. For example, in the 3D data input system 10, a sufficient number of sensor groups SD capable of spatially covering the entire area of the surface of the subject S may be installed at appropriate positions. In this case, the sensor group SD is fixed with respect to the stage T. As another example, the 3D data input system 10 may have a plurality of frames F at predetermined intervals (for example, every 120 °) when the stage T is viewed from above. In this case, if the frame F and the stage T rotate relative to each other by 120 °, the sensor group SD can capture the entire area of the surface of the subject S.

  FIG. 4 is a diagram illustrating a hardware configuration of the 3D data processing system 20. The 3D data processing system 20 is a computer device having a CPU (Central Processing Unit) 201, a memory 202, a storage 203, and a network IF 204. The CPU 201 is a control device that executes processing according to a program and controls other hardware elements of the 3D data processing system 20. The memory 202 is a main storage device that functions as a work area when the CPU 201 executes a program, and includes, for example, a RAM (Random Access Memory). The storage 203 is a nonvolatile auxiliary storage device that stores various programs and data, and includes, for example, at least one of an HDD (Hard Disk Drive) and an SSD (Solid State Drive). The network IF 204 is an interface for performing communication according to a predetermined communication standard (for example, TCP / IP), and includes, for example, a NIC (Network Interface Card).

  In this example, the storage 203 stores a program for causing the computer device to function as the 3D data processing system 20 (hereinafter referred to as “3D data processing program”). When the CPU 201 executes the 3D data processing program, the function of FIG. 2 is implemented in the computer apparatus. The CPU 201 executing the 3D data processing program is an example of the addition unit 21, the storage unit 22, the classification unit 23, and the estimation unit 24.

  The application 30 may be anything as long as it uses 3D modeling data. As an example, the application 30 includes at least one of an ID card, a business card, virtual communication, a video game, dress-up (try-on), measurement, virtual theater, fitness, medical care, and movie production.

2. Operation FIG. 5 is a diagram illustrating an operation (3D data processing method) according to an embodiment of the 3D data system 1. FIG. 5 shows an outline of the operation. In step S1, the 3D data input system 10 generates 3D modeling data of the subject. In step S2, the 3D data processing system 20 performs processing for adding individuality data to the 3D modeling data. By giving individuality data, individuality is given to the movement of the 3D modeling data. In step S3, the application 30 provides products or services to the user using the 3D modeling data and the personality data.

2-1. Operation example 1
2-1.1.3 Generation of 3D modeling data FIG. 6 is a diagram illustrating details of processing for generating 3D modeling data of a subject. FIG. 6 shows 3D modeling data for a new subject (hereinafter referred to as “target subject S”) in a state where the 3D data system 1 has already stored 3D modeling data for each of a plurality of subjects (hereinafter referred to as “existing subjects”). Assume a situation that generates The process in FIG. 6 is started when, for example, the user instructs the 3D data input system 10 to generate 3D modeling data.

  In step S11, the 3D data input system 10 captures the target subject S and obtains a captured image. The target subject S is photographed in a predetermined pose. In addition to the captured image, the 3D data input system 10 acquires an image (for example, an image of an infrared pattern figure; hereinafter referred to as a “pattern image”) for measuring the distance to the camera C.

  In step S12, the 3D data input system 10 generates 3D modeling data from the captured image. The 3D data input system 10 calculates the distance from the camera C to each point in the pattern image, that is, the three-dimensional shape of the surface of the target subject S, using the pattern image. The 3D data input system 10 pastes the captured image on the calculated three-dimensional shape. In this way, 3D modeling data of the subject S is obtained.

  In step S13, the 3D data input system 10 acquires feature points from the 3D modeling data. This feature point is obtained from 3D modeling data. The feature points of the shape are so-called bone end points in the 3D model, for example. The bone is a virtual structure for giving motion to the 3D model.

  In this example, the 3D data input system 10 has a standard model of a 3D model bone structure. The standard model has, for example, a bone structure conforming to the skeleton structure of an actual human body. The standard model is classified into categories such as an adult male, an adult female, an elderly man, an elderly woman, an infant boy, and an infant girl. The 3D data input system 10 selects a standard model corresponding to the user's attribute from the plurality of standard models. For example, the 3D data input system 10 automatically determines a user's attributes (for example, age and sex) using a captured image. Alternatively, the 3D data input system 10 may prompt the user to input his / her attributes and set the user's attributes according to the user's input. The 3D data input system 10 adjusts the selected standard model according to the generated 3D model, and applies the adjusted standard model to the 3D model.

  FIG. 7 is a diagram illustrating adjustment of the standard model. 7A shows the 3D model on the left and the standard model of the bone on the right. In the standard model of bone, the end points are represented by dots (●). These multiple end points are all examples of feature points of the 3D model. For example, the 3D data input system 10 expands and contracts the height (height) of the standard model so as to match the 3D model (FIG. 7A). Further, the 3D data input system 10 moves the position of the end point of each bone to the center position in the direction perpendicular to the bone (FIG. 7B).

  FIG. 7C shows the standard model after adjustment. The end points of each bone in this figure correspond to feature points acquired from 3D modeling data. This figure shows a cross section passing through the virtual center of gravity of the 3D modeling data and parallel to the front of the subject S. Although FIG. 7 shows only a single cross section, the standard model may be adjusted so as to match the 3D model in a plurality of cross sections having different directions.

  Refer to FIG. 6 again. In step S <b> 14, the 3D data input system 10 calculates a feature amount indicating the positional relationship between a plurality of feature points. In this example, the 3D data input system 10 calculates a feature amount indicating a positional relationship with another feature point with respect to a specific feature point. The specific feature points are, for example, feature points at positions corresponding to knee joints. In this case, the positional relationship between a specific feature point and another adjacent feature point refers to the positional relationship of the knee joint with the ankle and hip joint, so-called O leg, X leg, or XO leg. Etc. are shown.

  FIG. 8 is a diagram illustrating the positional relationship between feature points. FIG. 8A shows a normal leg, FIG. 8B shows an O leg, and FIG. 8C shows an X leg. The feature amount related to the specific feature amount is, for example, the distance d1 between the feature point P2 corresponding to the knee and the straight line L with respect to the length L1 of the straight line L connecting the feature point P1 corresponding to the hip joint and the feature point P3 corresponding to the ankle. Defined as the ratio of Note that the specific feature points and the definition of the feature amount related to the specific feature points shown here are merely examples, and may be set in any manner. One 3D model may have a plurality of “specific feature points”. For example, the tip of the toe, knee, elbow, hand, neck, and head may be set as specific feature points.

  Refer to FIG. 6 again. In step S15, the 3D data input system 10 outputs the feature amount and 3D modeling data related to the feature point to the 3D data processing system. In this example, data (hereinafter referred to as “attribute data”) indicating the attributes (ID, age, sex, etc.) of the subject is further added to the 3D modeling data.

2-1-2. Adding Individuality Data FIG. 9 is a diagram illustrating details of processing for adding individuality data to 3D modeling data. This process is started when 3D modeling data is input from the 3D data input system 10, for example.

In step S21, the 3D data processing system 20 classifies the 3D modeling data of the target subject S into any of a plurality of groups. The classification is performed from one of the following viewpoints, for example.
(1) Classification based on static feature values of the subject.
(2) Classification based on the dynamic feature amount of the subject.
(3) Classification based on subject attributes.
(4) Classification by a combination of two or more of the above.

  The classification based on the static feature amount of the subject means that the 3D modeling data is classified using at least one type of static feature amount. One type of static feature value is, for example, a feature value of the positional relationship between the knee joint and the ankle and hip joint. The two types of feature amounts refer to, for example, the feature amount of the positional relationship between the knee joint and the ankle and hip joints, and the feature amount of the positional relationship between the elbow joint and the wrist and shoulder joints. According to this example, a dynamic feature amount can be estimated from a static feature amount of a subject.

  The classification based on the dynamic feature amount of the subject means that the 3D modeling data is classified using at least one type of dynamic feature amount. The feature quantity used for classification is a different kind of feature quantity from the feature quantity related to the individuality to be estimated. According to this example, it is possible to estimate the second type of dynamic feature amount from the first type of dynamic feature amount of the subject. For example, when the individuality to be estimated (an example of the second type of dynamic feature value) is the highest position of the knee in the jump motion, the feature value used for the classification is the highest position of the knee in the walking motion ( An example of the first type of dynamic feature amount).

  Classification based on subject attributes refers to classifying 3D modeling data using at least one subject attribute. The attribute of the subject means, for example, the age, sex, race, nationality, occupation, or past history of the subject. According to this example, a dynamic feature amount can be estimated from the attribute of the subject.

  The classification based on the combination of two or more means a classification using a combination of two or more of a classification based on a static feature quantity, a classification based on a dynamic feature quantity, and a classification based on an attribute of a subject. According to this example, more various estimations can be performed.

  Here, as an example, an example in which classification based on the combination of (4) is used will be described. Specifically, the 3D modeling data is first classified based on the attributes of the subject (first level classification), and further classified based on the static feature amount of the subject (second level classification). The classification based on the attribute of the subject is a classification based on the age and sex of the subject. In this example, the attribute used when selecting the bone structure standard model is used as it is for classification of 3D modeling data. That is, in the first stage classification, the classification of 3D modeling data and the standard model of the bone structure correspond one-to-one.

  FIG. 10 is a diagram illustrating classification of 3D modeling data. FIG. 10 shows the second stage classification. In this example, the second-stage classification is performed using N types of feature amounts. Specifically, the 3D data processing system 20 plots each 3D modeling data in an N-dimensional space with each feature amount as a coordinate axis, and clusters (groups) the plot group according to a mathematical algorithm. Here, for simplicity of explanation, an example of N = 2 is shown. That is, the second-stage classification is performed using two types of feature amounts, that is, feature amount A and feature amount B. In one example, the feature value A is a feature value related to the knee, and the feature value B is a feature value related to the thoracic vertebra. In this figure, each 3D modeling data is plotted on two-dimensional coordinates with the feature amount A on the vertical axis and the feature amount B on the horizontal axis. One plot corresponds to one 3D modeling data. For example, if 3D modeling data and feature quantities of 10,000 people have been obtained in the past, 10,000 plots can be obtained. The 3D data processing system 20 divides or groups these plots into subsets using a known clustering method such as the shortest distance method or the k-means method. In the example of FIG. 10, these plots are classified into five groups G1 to G5.

  For example, when the classification illustrated in FIG. 10 is obtained for the 3D modeling data of the existing subject, the 3D data processing system 20 specifies an existing plot that is closest to the plot of the 3D modeling data of the target subject S. The 3D modeling data of the target subject S is classified into the same group as the existing plot. Alternatively, the 3D data processing system 20 again classifies the 3D modeling data of the target subject S and the existing 3D modeling data using a clustering method, and determines to which group the 3D modeling data of the target subject S belongs.

  As already described, since the first-stage classification is performed for each standard model of the bone structure, the second-stage classification is equivalent to being performed for each standard model. For example, when the target subject S is an adult male, the clustering technique is applied to the adult male population including the existing subject, and the 3D modeling data of the target subject S is classified using the obtained result.

  Refer to FIG. 9 again. In step S22, the 3D data processing system 20 estimates the individuality of the 3D modeling data of the target subject S (that is, the individuality of the target subject S) based on the classification. In this example of movement, individuality refers to information for defining a trajectory (displacement) of feature points of the 3D model, and more specifically, the highest positions of the knee and the hand when the 3D model is walked. The personality data in this example is data indicating the highest positions of the knee and the hand in the walking motion. Alternatively, the personality data may include data indicating the position in the intermediate state in addition to the highest positions of the elbow and the hand.

  FIG. 11 is a diagram illustrating the movement of 3D modeling data. This example shows the relative positional relationship of bones when 3D modeling data is walking. FIG. 11A shows a basic posture. FIG. 11B shows a posture in which the left foot is lifted. FIG. 11C shows a posture in which the left foot is lowered on the ground. FIG. 11D shows a posture in which the right foot is lifted. FIG. 11E shows a posture in which the right foot is lowered. By repeatedly taking these postures in the 3D modeling data, the 3D model can be walked. 11A to 11E correspond to so-called key frames, and the posture between the key frames is calculated by interpolation. The posture of the key frame is defined for each standard model and is stored in advance in the 3D data processing system 20.

In this example, the following data is defined as individuality:
(1) The position of the left knee and the position of the right hand in the posture of FIG.
(2) The position of the right knee and the position of the left hand in the posture of FIG.
More specifically, the deviation (difference) from the position of the feature point in the key frame defined for the standard model is defined as individuality. For example, the deviation vector d (P2) from the left knee position P2s in the posture of the standard model in FIG. 11B is an example of individuality data.

  For a part of the 3D modeling data stored in the 3D data system 1, the movement of the subject is actually shot at the time of shooting for generating the 3D modeling or at another timing. Hereinafter, a subject on which an actual movement is photographed is referred to as a “specific subject”. That is, the personality data of the specific subject is actually estimated, not estimated. For example, the 3D data input system 10 captures a situation where a specific subject is walking, and acquires an image corresponding to the key frame illustrated in FIG. 11 (an example of a plurality of captured images with different poses). The 3D data processing system 20 determines motion data representing each group from the motion data thus obtained. Thus, for each group, individuality data representing the group (representative individuality data) is obtained.

  Individuality data representing a group is, for example, individuality data of a subject randomly selected from specific subjects included in the group. Alternatively, personality data representing a group is personality data of a subject plotted at a position closest to the center coordinates of the group in an N-dimensional coordinate space among specific subjects included in the group. Further alternatively, the individuality data representing a group is a statistical representative value such as an average value of individuality data of a plurality of specific subjects included in the group.

  The 3D data processing system 20 uses the individuality data representing the group to which the 3D modeling data of the target subject S belongs as the individuality data of the 3D modeling data serving as the target subject. This corresponds to estimating that the individuality of the target subject S is the same as the individuality representing the group to which the 3D modeling data of the target subject S belongs. For example, when the target 3D modeling data belongs to the group G4, the 3D data processing system 20 determines to use the individuality data representing the group G4 as the individuality data of the 3D modeling data of the target subject S.

  Refer to FIG. 9 again. In step S23, the 3D data processing system 20 adds individuality data indicating the estimated individuality to the 3D modeling data of the target subject S. Here, “adding” means storing in a state in which at least the correspondence between the two is known. In this way, the application 30 can use the 3D modeling data of the target subject S.

  Note that the individuality data added to one 3D modeling data is not limited to one type. A plurality of types of individuality data may be added to one 3D modeling data. For example, personality data regarding a walking motion, personality data regarding a jumping motion, personality data regarding a punching motion, and personality data regarding a kicking motion may be added to 3D modeling data of a subject. Alternatively, personality data that comprehensively defines these operations may be defined.

2-1-3. Individuality data The individuality data described so far is merely an example. As individuality data, the following data may be included instead of or in addition to the example described above.

(1) Hair hardness When the 3D model moves, the reaction causes the hair to shake. Hair hardness is individuality data that affects the way hair sways. If the hair is hard, the way of shaking is small, and if the hair is soft, the way of shaking is large. For example, if the positions of some hair tips are extracted as feature points, the hardness of the hair can be expressed as the movement of the feature points.

(2) Blink The method of blinking also has personality. For example, there are those who blink a plurality of times at once, those who have a slow blinking speed, and those who have a long time to keep their eyelids closed. For example, if the lower end of the eyelid is extracted as a feature point, the manner of blinking can be expressed as the movement of the feature point.

(3) Facial expressions Smiles and other facial expressions are also unique. FIG. 12 is a diagram illustrating feature points (● in the figure) extracted from the face. For example, if the corners of the mouth, the corners of the eyes, and the ends of facial muscles are extracted as feature points, the facial expression can be expressed as the movement of the feature points.

(4) Voice The personality data may include voice data. The audio data does not include information on the miracle of the feature points extracted from the 3D modeling data, but the dynamic personality of the subject in that the audio is caused by the vibration of the subject's vocal cords and is accompanied by a time change. It can be said that it shows. The voice may correlate with the physical characteristics and attributes of the subject (the boy's voice is high, etc.), and is compatible with the estimation of individuality data.

(5) Skin condition Skin tension and looseness are dynamic personalities that change on a relatively long time scale (decades). It can be said that changes in the skin condition indicate growth and aging. For example, if a feature point is extracted from the unevenness of the face, growth and aging can be expressed as a time change of the feature point.

(6) Movement habits Movements other than the individuality in standard movements such as walking and running, for example, movements such as scratching hair, scratching the head, touching the nose, shaking poorly, etc., represent the individuality of the subject. Sometimes. These operations can also be converted into data.

(7) Viewpoint Position and Angle of View In the application 30, the 3D model is placed in the virtual space. A virtual camera is installed in the virtual space. The application 30 displays the two-dimensional image acquired by the virtual camera on the display device. Since the virtual camera is only virtual, the application 30 can theoretically set the viewpoint position and the angle of view arbitrarily. However, there is a demand for limiting the relative position of the virtual camera based on the 3D model depending on the application. As a specific example, for example, when handling a 3D model of a female idol singer, there is a desire to restrict the virtual camera from coming below the waist of the 3D model. Alternatively, there is a desire to prevent the angle of view of the virtual camera from being reduced beyond a certain range. In order to meet these demands, data for limiting the position of the viewpoint of the virtual camera (relative to the 3D model) can be used as individuality data. With this information, it is possible to define a region that is not shown to the user for each 3D model.

(8) Movable restriction The movable range of the joint represents the individuality of the subject. For example, a gymnast can open both legs (hip joints) 180 °, but a general person can open only about 90 °. If the movable range of each joint is converted into data, it can be used as individuality data. In this case, the movable range of the joint may be limited due to circumstances other than the actual movable range. For example, in a specific 3D model, the movable range of the hip joint is limited to about 60 °. Thus, data for limiting the movable range of the joint can be used as individuality data.

2-1-4. Use of 3D Modeling Data FIG. 13 is a diagram illustrating details of processing using 3D modeling data. The process in FIG. 13 is started when, for example, the user of the application 30 instructs acquisition of 3D modeling data.

  In step S31, the application 30 requests 3D modeling data from the 3D data processing system 20. This request includes information for specifying the 3D modeling data, for example, the ID of the subject. Alternatively, this request may include a search key for searching for desired data from the 3D modeling data stored in the 3D data system 1. In this case, the 3D data processing system 20 transmits a list of 3D modeling data that matches the search key to the application 30. The application 30 selects one 3D modeling data from this list. Information identifying the selected 3D modeling data is output to the 3D data processing system 20.

  This request may include information specifying personality data in addition to information specifying 3D modeling data. In the case where plural types of individuality data are added to one 3D modeling data, the specific application 30 does not necessarily require all the plural types of individuality data. Therefore, the application 30 requests the 3D data processing system 20 only for the personality data that it needs.

  In step S <b> 32, the 3D data processing system 20 outputs the requested 3D modeling data and individuality data to the requesting application 30. When the application 30 requests only specific personality data, the 3D data processing system 20 outputs only the requested personality data in addition to the 3D modeling data.

  In step S33, the application 30 provides a product or service using the 3D modeling data and the personality data acquired from the 3D data processing system 20. In the 3D data system 1, different types of applications can use common 3D modeling data and individuality data. The application 30 includes, for example, at least one of an ID card, a business card, virtual communication, a video game, dress-up (try-on), measurement, virtual theater, fitness, medical care, and movie production. The application 30 may operate on a mobile terminal such as a smartphone, or may operate on a stationary personal computer.

  The ID card is an application used for user identification. In the ID card, the 3D model is displayed instead of the user's photograph.

  A business card is an application for transmitting a user's personal information to other users. The business card data includes user 3D modeling data. For example, the business card data of the user UA is output to another user UB. The user UB can view business card data including the 3D model of the user UA on his computer device.

  Virtual communication is an application for communicating with other users in a virtual space. In the virtual space, each user is displayed using a so-called avatar. As this avatar, the 3D model according to the present embodiment is used. By using virtual communication, for example, a plurality of users at remote locations can hold a conference. In a meeting, each user's avatar has individuality, and reality increases.

  In the video game, the 3D model according to the present embodiment is used as a character appearing in the game. For example, in a fighting game, a player can use his or her 3D model as a player character. This 3D model has movement characteristics (for example, how to run, how to jump, etc.) corresponding to the player.

  Dress-up is an application for dressing a human body model in a virtual space. As this human body model, the 3D model according to the present embodiment is used. The human body model moves in the virtual space with clothes (so-called runway walk). At this time, the human body model has movement characteristics (for example, how to walk) corresponding to the user.

  Measuring is an application for measuring the body size (height, chest circumference, waist circumference, etc.) of the subject S.

  A virtual theater is an application that allows a virtual character (avatar) to perform a performance (singing, drama, dance, etc.) in a virtual space and appreciate the performance. Each character has its own personality. The demonstration is performed on a stage in a virtual space, for example. The user can appreciate this demonstration taken with the virtual camera. The position of the virtual camera (viewpoint) is controlled in accordance with a user instruction, for example. The user can bring a virtual camera to a specific performer, overlook the stage, and freely control the position of the virtual camera.

  Fitness is an application that records a subject's body as a 3D model before and after exercise. By comparing the stored 3D models, it is possible to visually confirm a portion where the muscle has been strengthened by exercise or a portion where extra meat has fallen.

  Medical treatment is an application that records a subject's body as a 3D model before and after treatment. By comparing the stored 3D models, the effects of treatment, medication, and rehabilitation can be visually confirmed.

  Movie production is an application that allows a 3D model to appear in a movie. For example, if the actor's 3D modeling data and individuality data are acquired, the actor can appear as a Nakano character in the movie without actually acting the actor.

2-2. Operation example 2
In the operation example 1, the example in which the individuality data of the target subject S is estimated has been described. In this example, no personality data is estimated. Individuality data is prepared in advance in the 3D data system 1, and a user (for example, the subject S himself / herself) selects desired data from among a plurality of individuality data prepared in advance.

  In the 3D data system 1, tags for searching are assigned to the individuality data. This tag includes, for example, age, sex, race, nationality, occupation, or past history. The user can search individuality data using these attributes as search keys. For example, the 3D data system 1 may apply the specified individuality data to the 3D modeling data of the subject S and display the preview.

  The personality data prepared in advance in the 3D data system 1 may include, for example, personality data of famous athletes and actors. In this case, the personality data is given an identifier (name) that identifies a sports player or an actor. Using personality data on the action of a professional soccer player kicking a ball, for example, in a soccer game, a user can make a 3D model obtained from his captured image appear as a player's character and give this character a famous professional You can kick in the form of a soccer player.

  According to this example, individuality data can be added to 3D modeling data without estimating individuality. The operation example 2 may be used together with the operation example 1. For example, the 3D data system 1 may cause the user to select either addition of individuality data by estimation or addition of individuality data prepared in advance, and add individuality data by a method selected by the user.

3. Modifications The present invention is not limited to the above-described embodiments, and various modifications can be made. Hereinafter, some modifications will be described. Two or more of the embodiments and the following modifications may be used in combination.

3-1. Modification 1
FIG. 14 is a diagram illustrating an outline of the 3D data system 1 according to the first modification. In this example, the 3D data processing system 20 stores a plurality of sets of 3D modeling data, bone data, and metadata. The bone data is data indicating the bone structure of the 3D model. For example, the bone data may be actually measured by photographing a subject in a plurality of poses, or may be obtained by adjusting a standard model of a bone structure prepared in advance according to a 3D model. Good. Metadata refers to data associated with the 3D model, and is data indicating, for example, subject attributes, imaging device attributes, audio, copy restrictions, and the like. It can be said that the individuality data and the attribute data exemplified in the embodiment are examples of metadata. Note that the metadata does not have to include information indicating the 3D model dynamic personality. Even in this case, the 3D data system 1 has an effect of easily providing the application 30 with a set of 3D modeling data, bone data, and metadata.

  The set of 3D modeling data, bone data, and metadata may be predetermined in the 3D data processing system 20. Alternatively, the set of 3D modeling data, bone data, and metadata may be rearranged according to the application 30 or user request. For example, when the metadata includes data that restricts the movable range of the joint (hereinafter referred to as “restriction data”) and audio data, the 3D data processing system 20 stores a plurality of templates of restriction data and audio data. . The 3D data processing system 20 adds the restriction data and audio data selected by the user to the designated 3D modeling data. According to this example, the user can select desired metadata and add it to the 3D modeling data. In the example of FIG. 14, the data set including 3D modeling data may not include bone data. In one example, the 3D data processing system 20 may store only 3D modeling data and metadata (eg, restriction data).

3-2. Modification 2
The method by which the estimation unit 24 estimates the individuality is not limited to that exemplified in the embodiment. Each group after classification may have a feature amount (representative feature amount) representing the group in addition to the individuality data representing the group (representative individuality data). The 3D data processing system 20 includes a correction unit (not shown). The correction unit corrects the representative personality data according to the difference between the representative feature amount and the feature amount obtained from the 3D modeling data of the target subject S. For example, the correcting unit adds the result (vector) obtained by multiplying the representative individuality data by the difference between the representative feature amount and the feature amount obtained from the 3D modeling data of the target subject S and the coefficient (vector). to correct. According to this example, for example, dynamic personality data can be corrected using a difference in static feature values.

3-3. Modification 3
The classification by the classification unit 23 may be divided into a plurality of levels. Each of these multiple levels has a different number of groups after classification. For example, the classification based on the same type of feature amount is classified into a low level (small number of groups), a medium level (medium number of groups), and a high level (large number of groups). Which of the plurality of categories is used is determined by, for example, a user instruction or a request from the application 30. For example, when the low-level (an example of the first level) classification is used, the estimation unit 24 obtains individuality data based on the difference between the representative feature amount of the group and the feature amount of the target subject S as in Modification 2. Estimate (correct). For example, when high-level (an example of the second level) classification is used, the estimation unit 24 uses the representative individuality data of the group as it is as the individuality data of the 3D model of the target subject S.

3-4. Modification 4
In the embodiment, the example in which the individuality data is defined as a deviation vector from the standard data of the dynamic feature amount has been described. However, the individuality data is not limited to a deviation vector, and may be data representing a dynamic feature amount itself (absolute value). In this case, there is no need to define standard data. When using the deviation vector, if the standard data is held on the application 30 side, it is only necessary to transmit the deviation vector as compared with the case of transmitting the standard data and the deviation vector. The amount of data to be reduced can be reduced.

3-5. Modification 5
When classification based on 3D modeling data is performed, the representative individuality data of the group is not limited to actually measured data. Individuality data obtained by estimation may be used as representative individuality data of a group.

3-6. Other Modifications In the 3D data system 1, the sharing of functions in the 3D data input system 10 and the 3D data processing system 20 is not limited to that illustrated in FIG. For example, a function corresponding to the generation unit 12 may be implemented in the 3D data processing system 20, that is, the cloud. Alternatively, a function corresponding to the addition unit 21 or the classification unit 23 may be implemented in the 3D data input system 10.

  The 3D data input system 10, the 3D data processing system 20, and the application 30 may each be implemented in one or more different computer devices, or at least some of the functions may be implemented in a common computer device. A plurality of computer devices may function as the 3D data input system 10, the 3D data processing system 20, or the application 30.

Claims (15)

A captured image obtained by photographing the surface of the subject subject, and an image obtaining unit for obtaining distance information indicating a distance from the reference point to the surface;
An adding unit for adding individuality data indicating the dynamic individuality of the 3D model to 3D modeling data indicating the 3D model of the subject subject obtained from the captured image and the distance information;
A 3D data system comprising: a storage unit that stores the 3D modeling data to which the individuality data is added. Using the 3D modeling data, and having an estimation unit for estimating the individuality of the subject subject,
The 3D data system according to claim 1, wherein the adding unit adds data indicating the individuality estimated by the estimating unit as the individuality data. A feature acquisition unit for acquiring a plurality of feature points from the 3D modeling data;
A calculation unit that calculates a feature amount related to a positional relationship between the plurality of feature points;
The 3D data system according to claim 2, wherein the estimation unit estimates the individuality of the subject subject using the feature amount. A classification unit that classifies the subject subject into any of a plurality of groups;
The 3D data system according to claim 2, wherein the estimation unit estimates the individuality of the subject based on the classification. The classification unit includes:
Classifying the plurality of existing subjects into the plurality of groups based on the feature amounts obtained from the captured images of the plurality of subjects,
The 3D data system according to claim 4, wherein the subject subject is classified into one of a plurality of groups obtained based on feature amounts of the plurality of existing subjects. The feature acquisition unit acquires a static feature amount of the subject from the captured image,
The 3D data system according to claim 4 or 5, wherein the classification unit classifies the subject into any of the plurality of groups using the static feature amount. The image acquisition unit acquires a plurality of captured images with different poses for the subject,
The feature acquisition unit acquires a dynamic feature amount of the subject from the plurality of captured images,
The 3D data system according to any one of claims 4 to 6, wherein the classification unit classifies the subject into any of the plurality of groups using the dynamic feature amount. Each of the plurality of groups has representative personality data indicating individuality representing the group,
The 3D data system according to any one of claims 4 to 7, wherein the estimation unit estimates the individuality indicated by the representative individuality data of the group to which the subject belongs as the individuality of the subject. Each of the plurality of groups has a representative feature amount representing a feature amount representing the group, and representative individuality data representing individuality representing the group,
A correction unit that corrects the representative individuality data according to a difference between the representative feature amount and the feature amount obtained from the subject;
The 3D data system according to any one of claims 4 to 7, wherein the estimation unit estimates the individuality indicated by the representative individuality data corrected by the correction unit as the individuality of the subject. The classifying unit classifies a plurality of existing subjects into the plurality of groups based on feature amounts obtained from captured images of the plurality of subjects,
For each of the plurality of groups, the estimation unit uses, as the representative individuality data, individuality data obtained from a plurality of photographed images with different poses, which are obtained for at least a part of an existing subject belonging to the group. The 3D data system according to any one of claims 4 to 9. The classification unit classifies the subject according to at least one level selected from a plurality of levels with different numbers of groups after the classification,
The estimation unit includes
In the first level classification among the plurality of levels, the individuality of the subject is estimated based on the difference between the representative value of the feature amount in the group after the classification and the feature amount of the subject.
11. The second-level classification in which the number of groups after the classification is larger than the first level, the individuality representing the group after the classification is estimated as the individuality of the subject. 11. 3D data system according to item. A captured image obtained by photographing the surface of the subject, and distance information indicating a distance from the reference point to the surface;
Adding dynamic personality data of the 3D model to 3D modeling data indicating the 3D model of the subject obtained from the captured image and the distance information;
Storing the 3D modeling data to which the individuality data has been added. A captured image obtained by photographing the surface of the subject subject, and an image obtaining unit for obtaining distance information indicating a distance from the reference point to the surface;
Bone data indicating a bone structure for giving motion to the 3D model and metadata associated with the 3D model with respect to 3D modeling data indicating the 3D model of the subject subject obtained from the captured image and the distance information An additional part for adding
A 3D data system comprising: a storage unit that stores the 3D modeling data to which the bone data and metadata are added. The 3D data system according to claim 13, wherein the metadata includes data that limits a position or an angle of view of a virtual camera used when displaying the 3D model located in a virtual space. The 3D data system according to claim 13 or 14, wherein the metadata includes audio data of the 3D model.

Images