JP7418677B1 - Information processing system, information processing method, and information processing program - Google Patents

Abstract

[Problem] To provide an information processing system that can reproduce in a virtual space the movement of clothing that follows the movement of a person in the real world. [Solution] The information processing system is based on multiple images of clothing taken from multiple directions. a generation unit that generates a clothing model that can be displayed in a three-dimensional virtual space, an assignment unit that adds adjustment points to predetermined locations on the clothing model, and multiple bones set on the three-dimensional human body model that makes the clothing model wearable. Among them, an association unit associates bones corresponding to predetermined points of the clothing model with adjustment points of the clothing model, and an association unit that moves the human body model wearing the clothing model based on motion data that defines continuous motion of the human body model. In this case, the movement of the clothing model is simulated, and the human body model is initialized by positioning the bone associated with each adjustment point at the position of each adjustment point in the clothing model with the initial shape, which is the shape at the time the clothing was photographed. The apparatus includes a simulation unit that starts a simulation from a posture. [Selection diagram] Figure 2

Description

The present invention relates to an information processing system, an information processing method, and an information processing program.

In recent years, with the development of information and communication technology, technological development related to virtual spaces has progressed, and many services that utilize virtual spaces have been proposed in the fashion world as well. For example, there are services that allow users to visit virtual stores that reproduce stores that exist in the real world in a virtual space, and browse and purchase products, and clothes and shoes that are worn by avatars that move back and forth in the virtual space. A service is provided that allows users to purchase virtual items (virtual items). In addition, NFT (Non-Fungible Token) technology makes it possible to prevent virtual items from being counterfeited or tampered with, making it possible to trade virtual items while maintaining their inherent value. distribution is progressing (for example, Non-Patent Document 1).

Metaverse Research Institute, “[14 Case Studies] How to use Metaverse in the apparel industry and its benefits”, [online], retrieved on May 22, 2020, Internet <URL: https://metaversesouken.com/metaverse /fashion-2/＞

The applicant has a large collection of highly rare vintage clothing, and is attempting to digitize these clothing items using NFT while preserving their value and distribute them in virtual space. However, when clothing that exists in the real world is digitized and worn by an avatar, it is difficult to reproduce the movement of the clothing that follows the movement of the person in the real world in a virtual space.

An information processing system according to an embodiment of the present invention includes a generation unit that generates a clothing model that can be displayed in a three-dimensional virtual space based on a plurality of images of clothing taken from multiple directions; A provision unit that assigns adjustment points to locations, a bone that corresponds to a predetermined location of the clothing model among a plurality of bones set on a three-dimensional human body model on which the clothing model is worn, and adjustment points of the clothing model. a simulation unit that simulates the movement of the clothing model when the human body model wearing the clothing model is operated based on motion data that defines continuous movements of the human body model; The part starts the simulation from a state in which the human body model is in an initial posture in which bones associated with each adjustment point are positioned at the position of each adjustment point in the clothing model of the initial shape, which is the shape at the time the clothing was photographed. let

In the information processing system according to an embodiment of the present invention, the predetermined points may be at least three or more points in the initial shape of the clothing model that enable calculation of the opening angle of the sleeve with respect to the body of the clothing.

In the information processing system according to an embodiment of the present invention, the simulation unit moves the human body model on a virtual surface based on the motion data, and moves the human body model in accordance with weight data included in the clothing model and corresponding to the material of the clothing. In a simulation, it is determined whether or not a collision will occur between the clothing model worn by the human body model and the virtual surface, and the weight data is changed until the motion based on the motion data is completed without collision. The simulation from the initial posture may be repeatedly executed.

In an information processing system according to an embodiment of the present invention, animation based on motion data of a human body model wearing a clothing model is performed using the movement of the clothing model based on weight data when the motion based on the motion data is completed without occurrence of a collision. The device further includes a storage unit that stores animation data to be displayed in association with the human body model, and the storage unit stores animation data for each of a plurality of combinations of a plurality of motion data, a plurality of human body models, and a plurality of clothing models. It's fine.

In an information processing system according to an embodiment of the present invention, conditions regarding a clothing model including weight data, adjustment points, a human body model, and motion data in a simulation executed by a simulation unit are input, and simulation results based on the conditions are output. The apparatus may further include a learning unit that generates a learning model that outputs adjustment points and weight data for a predetermined clothing model using machine learning.

An information processing system according to an embodiment of the present invention includes a virtual space generation unit that generates a virtual space in which a plurality of clothing models are arranged, and outputs display data for displaying the virtual space on a user's display device to the display device. The output unit further comprises an animation for displaying an animation of a human body model wearing the clothing model selected by the user in response to detecting the user's selection operation of the clothing model in the virtual space. The data may be sent to a display device.

The information processing system according to an embodiment of the present invention may further include a reception unit that receives placement instructions regarding placement of clothing models in the virtual space from a plurality of management users of the virtual space.

An information processing method executed by an information processing apparatus according to an embodiment of the present invention includes the steps of generating a three-dimensional clothing model based on a plurality of images of clothing taken from a plurality of directions; a step of assigning an adjustment point to a location; a step of associating a bone corresponding to a predetermined location of the clothing model with an adjustment point of the clothing model among bones set on a three-dimensional human body model on which the clothing model is to be worn; , simulating the movement of the clothing model when the human body model wearing the clothing model is operated based on motion data that defines continuous motion of the human body model, and the simulating step includes The simulation is started from a state in which the model is in an initial posture in which bones associated with each adjustment point are positioned at the position of each adjustment point in the clothing model of the initial shape that is the shape at the time the clothing was photographed.

An information processing program according to an embodiment of the present invention provides an information processing device with a function of generating a three-dimensional clothing model based on a plurality of images taken of clothing from a plurality of directions, and a function of generating a three-dimensional clothing model based on a plurality of images taken of clothing from a plurality of directions. A function for assigning adjustment points, a function for associating adjustment points of a clothing model with bones set in a three-dimensional human body model on which the clothing model is worn, and a bone that corresponds to a predetermined location on the clothing model. A function for simulating the movement of a clothing model when a human body model wearing the model is operated based on motion data that defines continuous movements of the human body model. is an information processing program that starts a simulation from a state in which the bone associated with each adjustment point is positioned at the position of each adjustment point in a clothing model with an initial shape that is the shape at the time the clothing was photographed. .

According to an embodiment of the present invention, when clothing that exists in the real world is digitized and worn by an avatar, the movement of the clothing that follows the movement of the person in the real world can be made to follow the movement of the avatar in the virtual space. It is possible to provide an information processing system etc. that can reproduce the movement of clothes.

FIG. 1 is a schematic diagram of an information processing system configuration according to an embodiment of the present invention. FIG. 2 is an example of a functional block diagram of an information processing system according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating an example of an information processing method according to an embodiment of the present invention. FIGS. 4(a) to 4(c) are diagrams for explaining the outline of an information processing system according to an embodiment of the present invention. FIGS. 5A and 5B are diagrams for explaining the outline of an information processing system according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating an example of an information processing method according to an embodiment of the present invention. FIG. 7 is an example of a data table stored in the information processing system according to an embodiment of the present invention. FIG. 8 is a schematic diagram showing an example of a virtual space realized in an information processing system according to an embodiment of the present invention.

Hereinafter, one embodiment of the invention (also referred to as the present invention) according to the present disclosure will be described with reference to the drawings. Note that the drawings are just examples, and the present invention is not limited to what is shown in the drawings. For example, the illustrated block diagrams, flowcharts, data tables, display screens of user terminals, etc. of the server (information processing device), user terminal (communication device), etc. are merely examples, and the present invention is not limited thereto.

In addition, in this specification, "virtual space" refers to a virtual information space that does not physically exist, built on an information processing device (computer) or network, in which a user operates an avatar. It includes the so-called ``metaverse,'' which is a space where people can interact with others. Note that an "avatar" is a character object that is an anthropomorphic person, animal, creature, object, etc., and may also be called a "digital human (virtual human)." Note that hereinafter, a person will be used as an example of an avatar, and the avatar may also be referred to as a "human body model."

In addition, in this specification, "clothes" refers to things worn by a person or avatar, including pants, skirts, dresses, all-in-ones, shirts, T-shirts, jackets, coats, and swimsuits, regardless of whether they are Western clothes or Japanese clothes. Including etc. Furthermore, "clothing" includes accessories such as hats, scarves, gloves, ties, and belts; accessories such as necklaces, earrings, watches, glasses, and masks; footwear such as shoes, sandals, and clogs; and masks as sanitary products. may include.

<System configuration>
FIG. 1 is a diagram showing a configuration example of an information processing system according to an embodiment of the present invention. The information processing system 600 provides a clothing model that is a digitized version of clothing that exists in the real world (hereinafter also referred to as "real clothing") that can be worn by a human body model, and virtualizes the movement of clothing in the real world. It is an information communication system that can be reproduced in space. Furthermore, the information processing system 600 may be an information communication system that provides a digital museum in which a plurality of real clothes are digitized and their three-dimensional (3D) shapes are displayed in a viewable state.

Information processing system 600 includes at least server 100, database server 400, and user communication terminals (user terminals) 200 (200A, 200B). The server 100 is connected to the communication terminals 200A and 200B via the network 500, transmits and receives various information to and from the communication terminal 200, and executes various processes related to the information processing system. Note that although only one server 100 is shown in FIG. 1, the number is not limited to this, and a plurality of servers may exist. Further, the server 100 is a distributed server system that operates cooperatively by communicating via a network, and may include an edge server, or may be a so-called cloud server. That is, the server 100 is not limited to a physical server, but may also include a virtual server. Further, the functions or processing of each functional unit of the server 100 may be realized by machine learning or AI (Artificial Intelligence) to the extent possible.

Network 500 may include wireless networks or wired networks, such as wireless LANs (WLANs), wide area networks (WANs), integrated service digital networks (ISDNs), wireless LANs, or code division networks (CDMA). multiple access), LTE (long term evolution), LTE-Advanced, 4th generation communication (4G), 5th generation communication (5G), 6th generation communication (6G) and later mobile communication systems, and combinations thereof. It may be.

The user terminals 200A and 200B may be communication terminals of users who use virtual space services provided by the information processing system 600. In addition, in FIG. 1, only two user terminals are shown as user terminals 200A and 200B for users A and B, respectively. However, there may be as many user terminals as there are users who use the virtual space service, and unless there is a particular distinction, they will simply be referred to as user terminals 200.

The user terminal 200 is installed with an application (hereinafter also referred to as an "app") for using the virtual space service provided by the information processing system 600, and is capable of transmitting and receiving various information to and from the server 100. good. Note that it is not essential to install an application on the user terminal 200, and the user terminal 200 and the server 100 may send and receive various information via a web browser. Although FIG. 1 shows a smartphone as the user terminal 200A and a notebook computer as the user terminal 200B, the user terminal 200 can be any terminal as long as it can realize the functions described in each embodiment described below. It can be any type of terminal. For example, the user terminal 200 may include a computer (e.g., a tablet), a handheld computer device (for example, but not limited to, a PDA (personal digital assistant)), a wearable terminal (such as a glasses-type device or a watch-type device), or an HMD (Head Mount Display). head-mounted display).

The database server 400 stores various information (data) used in the information processing system 600. Note that although only one database server 400 is shown separately from the server 100 in FIG. 1, it may be integrated into the server 100. That is, database server 400 may be volatile memory or nonvolatile memory of server 100. Further, the database server 400 may be composed of a plurality of storage devices. Note that the database server 400 may be connected to the server 100 through a dedicated internal network different from the network 500, or may be connected to the server 100 via the network 500.

<User terminal>
FIG. 2 is a configuration diagram of a user terminal 200 according to an embodiment of the present invention. The hardware configuration and functional configuration of the user terminal 200 according to an embodiment of the present invention will be described using FIG. 2.

(1) Hardware configuration of user terminal The user terminal 200 includes a control section 210, a communication section 220, a display section 230, an input/output section 240, and a storage section 270.

The control unit 210 is typically a processor, and is implemented by a central processing unit (CPU), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), or the like. The control unit 210 reads the program stored in the storage unit 270 and executes the functions and methods shown in each embodiment by executing the code or instructions included in the read program.

The storage unit 270 stores various programs and various data necessary for the operation of the user terminal 200. The storage unit 270 may include, for example, a flash memory or the like, and may also include a memory (RAM (Random Access Memory), ROM (Read Only Memory), etc.) that provides a work area for the control unit 210.

The communication unit 220 may be implemented as hardware such as a NIC (Network Interface Card), communication software, or a combination thereof, and may transmit and receive various data to and from the server 100 via the network 500. The communication may be performed by wire or wirelessly, and any communication protocol may be used as long as mutual communication can be performed. Note that when the communication unit 220 is composed of a physically structured circuit, it may be expressed as a communication circuit.

The display unit 230 is a monitor that displays data according to the display data written in the frame buffer, and may be, for example, a touch panel, a touch display, or the like.

The input/output unit 240 may include an input device for inputting various operations to the user terminal 200 and an output device for outputting processing results processed by the user terminal 200. The input/output unit 240 may have an input device and an output device integrated, or may be separated into an input device and an output device. The input device may be realized by any one of all types of devices capable of receiving an input operation from the user 20 and transmitting information related to the input to the control unit 210, or a combination thereof. Input devices may include, for example, a touch panel, a touch display, a camera, and a microphone. The output device may output the processing results processed by the control unit 210. The output device may include, for example, a display, a touch panel, a speaker, and the like.

(2) Functional configuration of user terminal The user terminal 200 includes a control unit 210. The control unit 210 controls communication between the communication unit 220 and the server 100 via the network 500, and executes transmission and reception of various information. Further, the control unit 210 controls the display of data on the display unit 230, and causes the display unit 230 to display a screen according to information transmitted from the server 100, for example. Further, the control unit 210 controls transmission of various information to and from an external device via the input/output unit 240, and transmits various information to each functional unit in response to a user's input operation received by the input device, for example. It also transmits information from each functional unit to an output device (not shown) such as a touch panel, monitor, or speaker.

<Server>
FIG. 2 is a configuration diagram of the server 100 according to an embodiment of the present invention. The hardware configuration and functional configuration of the server (information processing device) 100 according to an embodiment of the present invention will be described using FIG. 2.

(1) Hardware Configuration of Server The server 100 includes a control unit 110, a communication unit 120, an input/output unit 130, and a storage unit 170 as a hardware configuration. Note that although not shown, the server 100 may be appropriately equipped with a configuration that is included in a general server.

The control unit 110 is typically a processor, and may be a central processing unit (CPU), MPU, or the like. The control unit 110 may execute the functions and methods shown in each embodiment by reading a program stored in the storage unit 170 and executing codes or instructions included in the read program.

The storage unit 170 stores various programs and data necessary for the operation of the server 100. For example, although details will be described later, the storage unit 170 may store data regarding the movement of a clothing model that follows the movement of a human body model wearing the clothing model. The storage unit 170 may include, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. Furthermore, the storage unit 170 may include a memory that provides a work area for the control unit 110.

The communication unit 120 is implemented as hardware such as a NIC, communication software, or a combination thereof, and transmits and receives information between the server 100 and other devices.

(2) Functional configuration of server The server 100 includes a generation unit 111, a provision unit 112, an association unit 113, a simulation unit 114, an output unit 115, a learning unit 116, and a reception unit 117 as functions realized by the control unit 110. Be prepared. Note that in FIG. 2, functional units that are not essential in each of the embodiments described hereinafter may be omitted. Further, the functions or processing of each functional unit may be realized by machine learning or AI to the extent possible.

Hereinafter, one embodiment of the present invention will be described with reference to FIGS. 3 to 7, along with the processing of each functional unit of the server 100.

FIG. 3 is an example of a flowchart showing processing by the server 100 according to one embodiment. First, the generation unit 111 generates a clothing model that can be displayed in a three-dimensional virtual space based on a plurality of images of clothing taken from multiple directions (step S11). Here, "a clothing model that can be displayed in a three-dimensional virtual space" may refer to displaying a clothing model so that the user can visually recognize it in three dimensions.

Note that, as described above, in one embodiment of the present invention, a clothing model is generated by photogrammetry based on a plurality of images of clothing taken from multiple directions. Note that "Photogrammetry" is a method of photographing a subject from various angles, analyzing and integrating the digital images, and creating a three-dimensional 3DCG (3 Dimensional Computer Graphics) model. The method of photographing from multiple directions is not particularly limited, and a large number (for example, 200) of photographing units (cameras) may be regularly installed to surround the subject to photograph a plurality of images. However, a plurality of images may be taken by photographing with a camera while rotating a turntable on which an object is placed. Alternatively, a plurality of images may be taken by moving the camera and not moving the object to be photographed. Note that an existing method may be used to generate a three-dimensional model from a plurality of images.

FIG. 4A is an example of a clothing model generated by the generation unit 111. The generation unit 111 may generate the clothing model 10 after eliminating objects for fixing the clothing, such as a torso or a mannequin, that were used when photographing the clothing through image processing. The clothing model 10 may include a polygon mesh, texture information, and weight data based on the material of the clothing.

The assigning unit 112 assigns adjustment points to predetermined locations on the clothing model 10 (step S12). The "adjustment point" may be a virtual point (marker) used for alignment between a clothing model and a human body model, which will be described later. FIGS. 5A and 5B are diagrams illustrating predetermined locations and adjustment points. In one embodiment, the predetermined locations may be at least three or more points in the initial shape of the clothing model 10 that allow calculation of the opening angle of the sleeves relative to the body of the clothing. Note that the initial shape refers to the shape of the clothing at the time of photographing. Although details will be described later, when a clothing model is worn on a human body model and the movement of the clothing following the movement of the human body model is simulated, a phenomenon occurs in which the clothing model slips off from the human body model. According to one embodiment of the present invention, the possibility that clothing will slip off during simulation can be reduced by using adjustment points provided at predetermined locations on a clothing model.

FIGS. 5A and 5B are schematic diagrams illustrating cases of a dress and a T-shirt as clothing models, respectively. As shown in FIG. 5(a), when the clothing model 10 is a dress, adjustment points are given to the neck part M10, right wrist part M11, left wrist part M12, waist part M13, right leg part M14, and left leg part M15. It's okay to be. Further, as shown in FIG. 5(b), when the clothing model 10' is a T-shirt, adjustment points may be given to the neck part M20, right cuff part M21, left cuff part M22, and waist part M23. Note that the adjustment point may be specified based on the operation of the management user from a management user terminal (not shown) operated by the management user who provides the virtual space service. Alternatively, the assigning unit 112 automatically assigns adjustment points to any clothing model using a learning model generated in advance by learning the correspondence between clothing models and adjustment points assigned to the clothing models. May be granted. Further, the assigned adjustment points may be able to be modified by the user, or the correction may be performed by a program.

Note that, as described above, the adjustment points may be set depending on the type of clothing. For example, T-shirts are worn around the neck, right wrist and left wrist; skirts are worn around the waist, right leg and left leg; dresses are worn around the neck, right wrist, left wrist, waist, right leg and left leg; pants are worn around the waist and right knee. , the left knee, right foot, and left foot may be provided with adjustment points. Further, the adjustment points can be applied to clothing without sleeves, and for example, in the case of a tank top, the adjustment points may be set at both shoulders and the waist, and in the case of a tube top, the adjustment points may be set at the parts corresponding to both sides and the waist. In addition, when specifying adjustment points at the management user terminal, the adjustment points to be set may be notified at the management user terminal depending on the type of clothing model.

The association unit 113 associates the adjustment points of the clothing model 10 with the bones corresponding to predetermined locations on the clothing model 10, among the bones set in the three-dimensional human body model on which the clothing model 10 is to be worn (step S13). Here, FIG. 4(b) shows the relationship between the human body model 21 and the bones 22. In general, a bone indicates an element that is a unit of movement for moving a three-dimensional model in three-dimensional animation (3D animation), and animation is generated by the three-dimensional model following the movement of the bone while deforming. . When creating an animation of a human body, joints 23 are defined in the same way as for the human body, and bones 22 are connected to each other by joints 23, so that the bones can be bent or rotated at the joints 23.

In the example shown in FIG. 5A, the association unit 113 uses the adjustment points given to the neck, right wrist, left wrist, waist, right leg, and left leg portions in the clothing model 10 to , to the bones corresponding to the right wrist, left wrist, waist, right leg, and left leg. Furthermore, in the example shown in FIG. 5B, the association unit 113 uses the adjustment points given to the neck, right cuff, left cuff, and waist in the clothing model 10' to the neck, right elbow, and other points in the human body model 21. , the left elbow, and the bones corresponding to the waist. Here, "associating" may refer to setting the adjustment points and the bone positions so that they match. This causes the adjustment point to move following the movement of the bone.

The simulation unit 114 simulates the movement of the clothing model 10 when the human body model 21 wearing the clothing model 10 is moved based on the motion data (step S13). The motion data may be data that defines continuous motion of the human body model 21. The simulation of the movement of the clothing model 10 may be a so-called cloth simulation, which is a physical calculation related to physical laws that act on cloth.

Processing of the server 100 according to an embodiment of the present invention will be continued with reference to the flowchart of FIG. 6 . The simulation unit 114 starts from a state in which the human body model 21 is in an initial posture in which bones associated with each adjustment point are initially positioned at the position of each adjustment point in the clothing model of the initial shape that is the shape at the time the clothing was photographed. A simulation is started (step S21). This will be explained using FIGS. 4 and 5.

The clothing model 10 in FIG. 5 has an initial shape, and the simulation unit 114 makes sure that the adjustment points of the clothing model 10 in the initial shape of FIG. 5 match the bones of the human body model 21 associated with the adjustment points. Additionally, the posture of the human body model 21 may be changed. Here, "transformation" may include changing the lengths of parts of the human body model 21, such as the arms, shoulder width, and sitting height. Further, the simulation unit 114 calculates the shoulder angle θ in the clothing model 10 in the initial shape shown in FIG. 5 using the following equation. Here, the shoulder angle θ may be the opening angle of the sleeve with respect to the body of the garment in the initial shape of the garment model 10.

Note that in the above equation, α is a vector between the neck and wrist, and β is a vector between the neck and waist.

The simulation unit 114 transforms the position of the arm of the human body model 21 to a position where the opening angle from the torso is θ (initial posture), and then performs a cross simulation with the human body model 21 wearing the clothing model 10. You may start. Note that the simulation unit 114 moves the human body model 21 on a virtual surface based on the motion data, and performs a simulation according to the weight data according to the material of the clothing included in the clothing model 10 (step S22). . For example, as shown in FIG. 4C, the simulation unit 114 may simulate changes in the clothing model over time when the human body model 21 walks on the ground in the virtual space. At this time, the simulation unit 114 performs a cloth simulation based on the weight set to the polygon mesh of the clothing according to the type of cloth (silk, denim, linen material, etc.).

The simulation unit 114 determines whether or not a collision occurs between the clothing model 10 worn by the human body model 21 and a virtual surface (ground) while the human body model 21 is performing a motion based on the motion data. (Step S23). The occurrence of a collision means that the clothing model 10 has slipped off the human body model 21. If the determination in step S23 is YES, the simulation unit 114 may change the weight data set for the clothing model 10 (step S24). The clothing model 10 may slip down depending on, for example, the relationship between the shoulder width of the human body model 10 and the depth of the neckline of the clothing model 10. If slip-off occurs, the simulation unit 114 may change the weight of the clothing model 10 in a direction that controls the slip-off. For example, the simulation unit may partially increase the weight of the clothing model 10. That is, in addition to the weights for reproducing gravity, which are generally set in the cloth simulation, the simulation unit 114 partially changes (makes heavier) the weights in the clothing model 10 to prevent the occurrence of shedding. May be prevented. Note that the portions of the clothing model 10 whose weights are changed may be, for example, shoulder portions, collar portions, etc., but are not limited thereto. Also, the weight may be reduced. The simulation unit 114 may perform the cross simulation in step S22 again using the weight data changed in step S24.

During the simulation, if no collision occurs between the ground and the clothing model 10 (NO in step S23) and the movement of the human body model 21 based on the motion data is completed (that is, the simulation is successful), the simulation unit 114 determines whether the collision occurs. The animation data for displaying the animation based on the motion data of the human body model 21 wearing the clothing model 10 with the movement of the clothing model 10 according to the weight data when the motion based on the motion data is completed without occurrence of The information is stored in the database server 400 in association with each other (step S25). The animation data may include temporal changes in the polygon mesh of the clothing model 10 (temporal changes in the shape of the clothing model 10) when the human body model 21 is made to walk while wearing the clothing model 10.

The simulation unit 114 may perform the above-described simulation for a plurality of combinations of a plurality of motion data, a plurality of human body models, and a plurality of clothing models. Then, animation data when the simulation is successful may be stored in the database server 400. That is, the database server 400 may store the animation data for each of a plurality of combinations of a plurality of motion data, a plurality of human body models, and a plurality of clothing models. An example of a data table is shown. Note that the figure is an example, and the data stored in the database server 400 may be smaller or larger than this, and the data format is not limited to what is shown in the figure.

Animation data table TB10 includes identification information for multiple clothing models (cloth_1, cloth_2, cloth_3) for each identification information for identifying each human body model (in the illustrated example, "avatar_X", "avatar_Y", "avatar_Z"). , animation data associated with motion data identification information (motion_1, motion_2) are stored such that they can be identified by the animation data identification information (ID: Identifier). For example, the animation data identified by the animation ID "animation_X11" is the time change in the shape of the clothing model when the human body model "avatar_X" wears the clothing model "cloth_1" and moves according to the motion data "motion_1". shows. In addition, the animation data identified by the animation ID "animation_X22" is the temporal change in the shape of the clothing model when the human body model "avatar_X" wears the clothing model "cloth_2" and moves according to the motion data "motion_2". shows.

The user terminal 200 can receive from the user an operation to select a human body model, clothing, or motion for which an animation is to be displayed, and can transmit the selection operation to the server 100 as an animation request. The output unit 115 of the server 100 may select animation data according to the request from the database server 400 and output it to the user terminal 200 via the communication unit 120. Based on the animation data received from the server 100, the display unit 230 of the user terminal 200 displays an animation in which the clothes worn by the avatar 21 move in accordance with the walking of the avatar 21, as shown in FIG. 4(c). be able to.

As described above, according to an embodiment of the present invention, it is possible to generate clothing data, which is three-dimensional data of real clothing, using photogrammetry, and to reproduce in animation how the clothing moves in accordance with the movement of a human body model. can. At this time, by starting the simulation by matching the shape of the human body model to the clothing data, it is possible to reduce the possibility that the clothing will fall off the human body model and improve the probability that the cross simulation will be successful.

According to an embodiment of the present invention, the learning unit 116 inputs conditions related to the clothing model including weight data, adjustment points, human body model, and motion data in the simulation executed by the simulation unit 114, A learning model that outputs adjustment points and weight data for a predetermined clothing model may be generated by machine learning that outputs simulation results.

This reduces the probability that the simulation will fail and reduces the computational load.

<Display of clothing models>
As mentioned above, the applicant has a large collection of highly rare vintage clothing, and provides a virtual museum (digital museum) that displays these clothing items. FIG. 8 is an example of a digital museum displayed on the user terminal 200. As shown in FIG. 8, the digital museum 40 has a plurality of clothing models 41 arranged in a virtual space, and the user may be able to visit the digital museum via an avatar. That is, the server 100 may further include a virtual space generating section (not shown), and may generate display information for displaying a virtual space in which clothing models are arranged. Note that the above-described animation for the selected clothing model may be started on the user terminal 200 by using the selection of the clothing model 41 by the user's avatar as a trigger.

Note that the reception unit 117 of the server 100 may receive placement instructions regarding the placement of the clothing models 41 in the virtual space (digital museum) 40 from a plurality of management users of the virtual space. That is, the server 100 may simultaneously connect multiple user terminals 200 to the same virtual space 40 in response to requests from the user terminals 200 of multiple management users. This may mean that avatars of multiple administrative users exist simultaneously within the same virtual space 40.

In an embodiment of the present invention, data for displaying a plurality of clothing models 41 is stored, for example, in a database server 400, and an administrative user accesses the database server 400 to place clothing models 41 in the virtual space 40. You may do so. Note that "placing in a virtual space" may mean setting the position coordinates of the clothing model 41 in the coordinate system set for the virtual space. This allows the user to view the clothing model 41 as if it were placed within the virtual space 40.

The clothing model 41 placed in the virtual space 40 may be operable (accessible) by an administrative user. For example, according to one embodiment of the invention, the administrative user's avatar may be able to carry a clothing model 41 within the virtual space 40. That is, the receiving unit 117 of the server 100 receives an instruction for arranging the clothing model 41 in response to an operation instruction inputted through the input/output unit 240 at the user terminal 200 of the management user 100 . The output unit 115 of the server 100 outputs display information for displaying a moving image of the administrative user's avatar carrying the clothing model 41 on the user terminal 200 of another administrative user existing in the virtual space 40 in accordance with the placement instruction. may be output to the user terminal 200 of another management user. Note that avatars in the virtual space 40 may be able to converse with each other through voice chat or text chat using existing technology, and management users may be able to exchange opinions regarding the arrangement of the clothing models 41.

Note that different permissions may be granted to administrative users, and the administrative users who can access the database server 400 and move the clothing model 41 within the virtual space 40 may be restricted. This makes it possible to avoid duplication of the same clothing model 41.

In this way, according to one embodiment of the present invention, multiple users can work together, so it is possible to efficiently design a virtual space.

Note that objects that can be placed in the virtual space 40 are not limited to clothing models. For example, in the virtual space 40, creatures such as animals, fish, and insects, plants such as plants, flowers, and trees, and crafts and works of art such as paintings, ceramics, and sculptures may be arranged according to instructions from the user. .

Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art will be able to easily make various changes and modifications based on the present disclosure. Therefore, it should be noted that these variations and modifications are included within the scope of the present invention. For example, the functions included in each component, step, etc. can be rearranged to avoid logical contradictions, and multiple components, steps, etc. can be combined into one or divided. It is. Further, the structures shown in the above embodiments may be combined as appropriate.

For example, the above description describes how weight data is adjusted when a simulation fails. However, the parameters to be adjusted are not limited to these, and may also be changes such as the size of the polygon mesh or the bones associated with the adjustment points.

Further, in the above description, a mode has been described in which animations for combinations of a human body model, a clothing model, and motion data are stored in advance, and the animation data is output in accordance with the user's selection. However, a cross simulation may be performed in which a human body model (user avatar) of the user is generated and the user avatar is made to wear a clothing model.

The program of each embodiment of the present disclosure may be provided in a state stored in a storage medium readable by an information processing device. The storage medium is a "non-temporary tangible medium" that can store a program. Programs include, for example, software programs and control programs. When each functional part of the server is realized by software, the server has a generation part 111, a provision part 112, an association part 113, a simulation part 114, an output part 115, It functions as a learning section 116 and a reception section 117.

The storage medium may, where appropriate, include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.), hard drives, etc. Disk drive (HDD), hybrid hard drive (HHD), optical disk, optical disk drive (ODD), magneto-optical disk, magneto-optical drive, floppy diskette, floppy disk drive (FDD), magnetic tape, solid-state drive (SSD), a RAM drive, a secure digital card or drive, any other suitable storage medium, or a suitable combination of two or more thereof. Storage media may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

Further, the program of the present disclosure may be provided to a server via any transmission medium (communication network, broadcast wave, etc.) that can transmit the program.

Each embodiment of the present disclosure may also be implemented in the form of a data signal embedded in a carrier wave, where the program is embodied by electronic transmission. Note that the program of the present disclosure may be implemented using, for example, a script language such as JavaScript (registered trademark) or Python, C language, Go language, Swift, Koltin, Java (registered trademark), or the like.

100 Server (information processing device)
110 Control unit 111 Generation unit 112 Addition unit 113 Association unit 114 Simulation unit 115 Output unit 116 Learning unit 117 Reception unit 120 Communication unit 130 Input/output unit 170 Storage unit 200 User terminal (communication device)
210 Control unit 220 Communication unit 230 Display unit 240 Input/output unit 270 Storage unit 400 Database server 500 Network 600 Information processing system 10, 10' Clothing model 21 Human body model 22 Bone 23 Joint M10 to M15, M20 to M23 Adjustment point 40 Virtual space 41 Clothing model

Claims (8)

a generation unit that generates a three-dimensional clothing model based on multiple images of clothing taken from multiple directions;
an assigning unit that assigns adjustment points to predetermined locations on the clothing model;
an association unit that associates a bone corresponding to a predetermined location of the clothing model with an adjustment point of the clothing model among bones set on a three-dimensional human body model on which the clothing model is worn;
a simulation unit that simulates the movement of the clothing model when the human body model wearing the clothing model is operated based on motion data that defines continuous movements of the human body model;
The simulation unit is configured to perform a state in which the human body model is in an initial posture in which a bone associated with each adjustment point is positioned at a position of each adjustment point in the clothing model having an initial shape that is the shape at the time the clothing was photographed. starting the simulation, causing the human body model to move on a virtual surface based on the motion data, and in the simulation according to weight data corresponding to the material of the clothing included in the clothing model, Determining whether or not a collision occurs between the clothing model worn by the human body model and the virtual surface, and changing the weight data until the motion based on the motion data is completed without a collision occurring. while repeatedly executing the simulation from the initial posture ,
Information processing system. 2. The information processing system according to claim 1, wherein the predetermined points are at least three points in the initial shape of the clothing model that enable calculation of the opening angle of the sleeve with respect to the body of the clothing. the animation data for displaying an animation based on the motion data of the human body model wearing the clothing model with the movement of the clothing model according to the weight data when the motion based on the motion data is completed without the occurrence of a collision; It further includes a memory unit that stores information in association with the human body model.
The information processing system according to claim 1 , wherein the storage unit stores the animation data for each of a plurality of combinations of a plurality of motion data, a plurality of human body models, and a plurality of clothing models. In the simulation executed by the simulation unit, conditions regarding the clothing model including weight data, adjustment points, human body model, and motion data are input, and the simulation results based on the conditions are output. Adjustment is made to a predetermined clothing model by machine learning. The information processing system according to claim 3 , further comprising a learning section that generates a learning model that outputs point and weight data. a virtual space generation unit that generates a virtual space in which a plurality of the clothing models are arranged;
further comprising an output unit that outputs display data for displaying the virtual space on a user's display device to the display device,
The output unit, in response to detecting a selection operation of the clothing model in the virtual space by the user, generates animation data for displaying an animation of a human body model wearing the clothing model selected by the user. The information processing system according to claim 4 , wherein the information processing system transmits the information to the display device. The information processing system according to claim 5 , further comprising a reception unit that receives placement instructions regarding placement of the clothing model in the virtual space from a plurality of management users of the virtual space. An information processing method executed by an information processing device, the method comprising:
generating a three-dimensional clothing model based on a plurality of images taken of the clothing from multiple directions;
adding adjustment points to predetermined locations of the clothing model;
a step of associating a bone corresponding to a predetermined location of the clothing model among bones set on a three-dimensional human body model on which the clothing model is to be worn with an adjustment point of the clothing model;
simulating the movement of the clothing model when the human body model wearing the clothing model is operated based on motion data that defines continuous movements of the human body model;
In the step of simulating, the human body model is in an initial posture in which a bone associated with each adjustment point is positioned at a position of each adjustment point in the clothing model having an initial shape that is the shape at the time the clothing was photographed. start the simulation from the state , move the human body model on a virtual surface based on the motion data, and perform the simulation according to weight data according to the material of the clothing included in the clothing model. , determine whether a collision occurs between the clothing model worn by the human body model and the virtual surface, and update the weight data until the motion based on the motion data is completed without collision. An information processing method that repeatedly executes a simulation from the initial posture while changing the initial posture . In the information processing device,
A function that generates a three-dimensional clothing model based on multiple images of clothing taken from multiple directions,
a function of adding adjustment points to predetermined locations of the clothing model;
A function of associating adjustment points of the clothing model with bones corresponding to predetermined locations of the clothing model among bones set on a three-dimensional human body model on which the clothing model is worn;
realizing a function of simulating the movement of the clothing model when the human body model wearing the clothing model is operated based on motion data that defines continuous movements of the human body model;
The simulation function is such that the human body model is in an initial posture in which a bone associated with each adjustment point is positioned at a position of each adjustment point in the clothing model having an initial shape that is the shape at the time the clothing was photographed. start the simulation from the state , move the human body model on a virtual surface based on the motion data, and perform the simulation according to weight data according to the material of the clothing included in the clothing model. , determine whether a collision occurs between the clothing model worn by the human body model and the virtual surface, and update the weight data until the motion based on the motion data is completed without collision. An information processing program that repeatedly executes a simulation from the initial posture while changing the initial posture .

Images