WO2023218861A1

WO2023218861A1 - Simulation device

Info

Publication number: WO2023218861A1
Application number: PCT/JP2023/015234
Authority: WO
Inventors: 佐恵木村
Original assignee: 株式会社Ｎｔｔドコモ
Priority date: 2022-05-12
Filing date: 2023-04-14
Publication date: 2023-11-16

Abstract

A simulation device in which each of a plurality of clothing data includes shape data which expresses the three-dimensional shape of clothing and tag data which expresses what number in a sequence said clothing is to be put on by the user, said simulation device having a processing device equipped with: a receiving unit for receiving two or more pieces of clothing selected by a user from among a plurality of pieces of clothing; an identification unit for identifying the overlap order of the two or more pieces of clothing on the basis of the two or more clothing data which correspond to the two or more pieces of clothing received by the receiving unit; and an image generation unit for generating a first synthesized image in which a three-dimensional image of two or more pieces of clothing overlaps a three-dimensional image of the user according to the identified overlap order, on the basis of body shape data which expresses the three-dimensional body shape of the user, and the two or more clothing data.

Description

simulation device

The present invention relates to a simulation device.

In recent years, simulation devices have been used that simulate the appearance of users trying on virtual costumes. In particular, a simulation device is sometimes used that generates a video in which the user moves while trying on virtual costumes in accordance with the user's own movements.

For example, Patent Document 1 generates motion body shape data by moving the user's 3D body shape data three-dimensionally, and dresses the 3D body shape data at each time included in the motion body shape data with a costume indicated by costume data. Thus, a virtual try-on system that generates try-on images is disclosed.

Patent No. 5605885

However, the virtual try-on system according to Patent Document 1 does not even consider the order in which multiple costumes are layered. Specifically, Patent Document 1 describes, as an example, a case where a T-shirt and jeans are worn. However, the technology according to Patent Document 1 does not consider which of the T-shirt and jeans is closer to the user's body.

Therefore, an object of the present invention is to provide a simulation device that can specify and simulate the order in which two or more costumes are layered when the user wears them.

A simulation device according to a preferred aspect of the present invention includes a processing device that simulates a case where a user wears multiple costumes by referring to a plurality of costume data in one-to-one correspondence with a plurality of costumes; Each of the plurality of costume data includes shape data indicating a three-dimensional shape of the costume, and tag data associated with the shape data and indicating the number of times the costume is worn by the user, The processing device includes a reception unit that receives input of two or more costumes selected by the user from among the plurality of costumes, and a processing device that receives input of two or more costumes respectively corresponding to the two or more costumes received by the reception unit. a specifying unit that specifies a stacking order indicating that the two or more costumes are stacked on top of each other based on the body shape data representing the three-dimensional body shape of the user and the two or more costume data; The simulation device includes an image generation unit that generates a first composite image in which three-dimensional images of the two or more costumes are superimposed on the dimensional image according to the specified superimposition order.

According to the present invention, when a user wears two or more costumes in layers, it is possible to provide a simulation device that can specify and simulate the order in which they are layered.

FIG. 1 is a diagram showing the overall configuration of a simulation system 1 according to a first embodiment. FIG. 2 is a block diagram showing a configuration example of a scanning device 20. FIG. A configuration example of the first data set DS1. Configuration example of costume data CD. A table showing an example of the correspondence between costume IDs and stacking order indexes. An example of installation of the imaging device 24. FIG. 3 is a block diagram showing a configuration example of a server 30. FIG. A configuration example of the first database DB1. 1 is a block diagram showing a configuration example of a terminal device 10. FIG. A configuration example of the second database DB2. An example of the configuration of corresponding data RD. An example of the operation screen OM. FIG. 3 is a functional block diagram of the image generation unit 116. FIG. 3 is an explanatory diagram of the operation of the image generation unit 116. FIG. 3 is an explanatory diagram of the operation of the image generation unit 116. 5 is a flowchart showing the operation of the terminal device 10.

1: First Embodiment The configuration of a simulation system 1 including a terminal device 10 as a simulation device according to a first embodiment of the present invention will be described below with reference to FIGS. 1 to 15.

1-1: Configuration of First Embodiment 1-1-1: Overall Configuration FIG. 1 shows the overall configuration of a simulation system 1 according to the first embodiment. As shown in FIG. 1, the simulation system 1 includes a terminal device 10, a scanning device 20, and a server 30. In the simulation system 1, the terminal device 10, the scanning device 20, and the server 30 are communicably connected to each other via the communication network NET. Note that in FIG. 1, it is assumed that an end user U uses a terminal device 10. End user U is an example of a "user."

The terminal device 10 is a device that allows an end user U to simulate his or her own appearance when trying on two or more virtual costumes. In particular, the terminal device 10 specifies the order in which multiple costumes are stacked on top of each other, that is, the stacking order, and then simulates a full-body image of the end user U trying on two or more virtual costumes. The terminal device 10 is an example of a "simulation device."

The scanning device 20 three-dimensionally scans two or more actual costumes and generates image data indicating a three-dimensional image of the costumes and costume data regarding the characteristics of the costumes, which are used when the terminal device 10 performs a simulation. do. The costume data includes a stacking order index used by the terminal device 10 to specify the stacking order of two or more costumes. The scanning device 20 outputs the generated image data and costume data as one data set to the server 30.

The server 30 acquires image data and costume data from the scanning device 20. Additionally, the server 30 outputs image data to the terminal device 10. Furthermore, the server 30 acquires a costume ID indicating the ID of the costume that the end user U tries on from the terminal device 10, and outputs costume data corresponding to the acquired costume ID to the terminal device 10.

1-1-2: Configuration of Scanning Device FIG. 2 is a block diagram showing an example of the configuration of the scanning device 20. The scanning device 20 includes a processing device 21 , a storage device 22 , a communication device 23 , an imaging device 24 , a display 25 , and an input device 26 . The elements of scanning device 20 are interconnected using one or more buses for communicating information.

The processing device 21 is a processor that controls the entire scanning device 20. Further, the processing device 21 is configured using, for example, a single chip or a plurality of chips. The processing device 21 is configured using, for example, a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, registers, and the like. Note that some or all of the functions of the processing device 21 may be implemented using hardware such as a DSP, ASIC, PLD, and FPGA. The processing device 21 executes various processes in parallel or sequentially.

The storage device 22 is a recording medium that can be read and written by the processing device 21. Furthermore, the storage device 22 stores a plurality of programs including the control program PR2 executed by the processing device 21. The storage device 22 also stores a first data set DS1. The first data set DS1 is a data set corresponding to a three-dimensional model of each costume.

FIG. 3 shows an example of the configuration of the first data set DS1. As shown in FIG. 3, the first data set DS1 is a set of costume ID, image data PD, and costume data CD. The costume ID indicates the ID of the costume scanned by the scanning device 20. As an example, the first alphabet of the costume ID corresponds to the type of costume, for example, a T-shirt, a long-sleeved shirt, and pants. Further, the numerical value following the alphabet indicates the item number of the costume. The image data PD indicates a three-dimensional image obtained as a result of three-dimensional scanning of the costume. The costume data CD is data regarding the characteristics of the costume. The costume data CD includes data obtained by analyzing the three-dimensional image and data input by the user of the scanning device 20. In the example shown in FIG. 3, the data set with costume ID=A001 includes data "H001.dae" as image data PD and data "O001.dst" as costume data CD.

FIG. 4 shows an example of the configuration of the costume data CD. The costume data CD includes shape data FD and tag data GD. Shape data FD is data indicating the three-dimensional shape of the costume. The shape data FD includes structure data SD, texture data TD, and joint data JD. The structure data SD is data regarding the structure of the costume. Specifically, the structure data SD includes data indicating what shape and thickness of cloth is placed at what position in one costume, and how the cloths are joined to each other. The texture data TD includes data indicating the material, stiffness, color, pattern, gloss, and texture of each cloth. When a general wearer of the costume wears the costume, the joint data JD includes data indicating positions corresponding to the joints of the general wearer in the costume. Furthermore, when a general wearer of the costume wears the costume, the joint data JD includes data indicating the relative position of the skeleton of the general wearer with respect to the costume. Note that the joint data JD indicates a predetermined range as a position in a costume that corresponds to the position of the skeleton and joints of the wearer of the costume.

The tag data GD is data used to classify costumes. The tag data GD includes, for example, a stacking order index used by the terminal device 10 to specify a stacking order indicating that a plurality of costumes are stacked on top of each other.

FIG. 5 is a table showing an example of the correspondence between costume IDs and stacking order indexes. Note that in FIG. 5, for convenience of explanation, examples of three-dimensional images of each costume indicated by image data PD corresponding to each costume ID are also shown.

In the example shown in FIG. 5, the stacking order index corresponding to the costume whose first alphabet in the costume ID is "A", that is, the T-shirt, is "1". The stacking order index corresponding to the costume whose first alphabet in the costume ID is “B”, that is, the long-sleeved shirt, is “3”. The stacking order index corresponding to a costume whose first alphabet in the costume ID is "C", that is, a sweater or a hoodie, is "4". The stacking order index corresponding to the costume whose first alphabet in the costume ID is “D”, that is, the jacket, is “5”. The stacking order index corresponding to the costume whose first alphabet in the costume ID is “E”, that is, pants, is “2”. The stacking order index corresponding to the costume whose first alphabet in the costume ID is “F”, that is, the coat, is “6”.

When the terminal device 10 described below uses three-dimensional images of two or more costumes to simulate the end user U trying on the two or more costumes, the terminal device 10 uses the three-dimensional images of two or more costumes to simulate the appearance of the end user U trying on the two or more costumes. The stacking order of two or more costumes is specified using the order index. Specifically, the terminal device 10 overlays a three-dimensional image of a costume with a relatively large stacking order index over a three-dimensional image of a costume with a relatively small stacking order index. Each costume shown in the example of FIG. 5 is arranged in the order of T-shirt → pants → long-sleeved shirt → sweater or hoodie → jacket → coat for each type of costume, relative to the three-dimensional image showing the body shape of the end user U. Located inside.

Returning to FIG. 4, part or all of the costume data CD may be automatically generated by the generation unit 212 using the analysis results of the analysis unit 213, which will be described later. Alternatively, some or all of these costume data CDs may be generated based on input information input by the user of the scanning device 20 via the input device 26.

Returning to FIG. 2, the communication device 23 is hardware as a transmitting and receiving device for communicating with other devices. The communication device 23 is also called, for example, a network device, a network controller, a network card, a communication module, or the like. The communication device 23 may include a connector for wired connection and an interface circuit corresponding to the connector. Furthermore, the communication device 23 may include a wireless communication interface. Examples of connectors and interface circuits for wired connections include products compliant with wired LAN, IEEE1394, and USB. Furthermore, examples of the wireless communication interface include products compliant with wireless LAN, Bluetooth (registered trademark), and the like.

The imaging device 24 images the outside world where the object exists. In particular, in this embodiment, the imaging device 24 images the costume. Further, the imaging device 24 outputs imaging information indicating an image obtained by imaging the costume. Furthermore, the imaging device 24 includes, for example, a lens, an imaging element, an amplifier, and an AD converter. The light collected through the lens is converted by the image sensor into an image signal, which is an analog signal. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is output to the processing device 21.

FIG. 6 shows an example of the installation of the imaging device 24 in this embodiment. In the example shown in FIG. 6, the scanning device 20 includes eight imaging devices 24-1 to 24-8. Note that it is only an example that the scanning device 20 includes eight imaging devices 24. Scanning device 20 can include any number of imaging devices 24 . In the example shown in FIG. 6, the imaging devices 24-1 to 24-8 are fixed to the frame F, and image the costume C placed in the hollow inside the frame F from 360° directions in three axes: top, bottom, left, right, front and back. do. A generation unit 212, which will be described later, generates a three-dimensional image of the costume C based on imaging information indicating a plurality of images captured by the imaging devices 24-1 to 24-8.

Returning to FIG. 2, the display 25 is a device that displays images and text information. The display 25 displays various images under the control of the processing device 21. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display 25.

The input device 26 accepts operations from the user of the scanning device 20. For example, the input device 26 includes a keyboard, a touch pad, a touch panel, or a pointing device such as a mouse. Here, when the input device 26 includes a touch panel, it may also serve as the display 25.

The processing device 21 reads the control program PR2 from the storage device 22 and executes it. As a result, the processing device 21 functions as an acquisition section 211, a generation section 212, an analysis section 213, and a communication control section 214.

The acquisition unit 211 acquires imaging information indicating a captured image of the costume C from the imaging device 24. The acquisition unit 211 also acquires input information input by the user of the scanning device 20 via the input device 26 . The input information includes, for example, a stacking order index.

The generation unit 212 generates image data PD representing a three-dimensional image of the costume C based on the imaging information acquired by the acquisition unit 211 from the imaging device 24. Further, the generation unit 212 generates costume data CD using the input information acquired by the acquisition unit 211 from the input device 26. Furthermore, the generation unit 212 generates the costume data CD using also analysis information indicating the analysis result obtained by analyzing the three-dimensional image of the costume C by the analysis unit 213, which will be described later. The generation unit 212 also generates a first data set DS1, which is a data set including a costume ID and a set of image data PD and costume data CD corresponding to the costume ID. The generation unit 212 stores the generated first data set DS1 in the storage device 22.

The analysis unit 213 analyzes the three-dimensional image of the costume C generated by the generation unit 212. The analysis unit 213 extracts, for example, features related to the shape of the costume C as a result of analyzing the three-dimensional image. Analysis information indicating the extracted features is output to the generation unit 212. As described above, the generation unit 212 generates costume data CD using the analysis information acquired from the analysis unit 213.

The communication control unit 214 causes the communication device 23 to transmit the first data set DS1 stored in the storage device 22 to the server 30.

As a result, the user of the scanning device 20 can easily create image data PD and costume data CD without having to manually input all of the data elements that make up image data PD and costume data CD one by one. Further, the simulation system 1 can simulate virtual try-on using the image data PD and costume data CD that are simply produced.

1-1-3: Server Configuration FIG. 7 is a block diagram showing an example of the configuration of the server 30. The server 30 includes a processing device 31, a storage device 32, a communication device 33, a display 34, and an input device 35. Each element included in the server 30 is interconnected using one or more buses for communicating information.

The processing device 31 is a processor that controls the entire server 30. Further, the processing device 31 is configured using, for example, a single chip or a plurality of chips. The processing device 31 is configured using, for example, a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, registers, and the like. Note that some or all of the functions of the processing device 31 may be implemented using hardware such as a DSP, ASIC, PLD, and FPGA. The processing device 31 executes various processes in parallel or sequentially.

The storage device 32 is a recording medium that can be read and written by the processing device 31. Furthermore, the storage device 32 stores a plurality of programs including the control program PR3 executed by the processing device 31. The storage device 32 also stores a first database DB1.

FIG. 8 shows an example of the configuration of the first database DB1. The first database DB1 is a database in which a first data set DS1 acquired from the scanning device 20 via the communication device 33 by the acquisition unit 311, which will be described later, is accumulated.

Returning to FIG. 7, the communication device 33 is hardware as a transmitting/receiving device for communicating with other devices. The communication device 33 is also called, for example, a network device, a network controller, a network card, a communication module, or the like. The communication device 33 may include a connector for wired connection and an interface circuit corresponding to the connector. Furthermore, the communication device 33 may include a wireless communication interface. Examples of connectors and interface circuits for wired connections include products compliant with wired LAN, IEEE1394, and USB. Furthermore, examples of the wireless communication interface include products compliant with wireless LAN, Bluetooth (registered trademark), and the like.

The display 34 is a device that displays images and text information. The display 34 displays various images under the control of the processing device 31. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display 34.

The input device 35 accepts operations from the user of the server 30. For example, the input device 35 includes a keyboard, a touch pad, a touch panel, or a pointing device such as a mouse. Here, when the input device 35 includes a touch panel, it may also serve as the display 34.

The processing device 31 reads the control program PR3 from the storage device 32 and executes it. As a result, the processing device 31 functions as an acquisition section 311, an extraction section 312, and a communication control section 313.

The acquisition unit 311 acquires the first data set DS1 from the scanning device 20 via the communication device 33. The acquisition unit 311 stores the acquired first data set DS1 in the first database DB1. The acquisition unit 311 also acquires selection information indicating the selection result of the costume selected by the end user U from the terminal device 10 via the communication device 33, as described later.

The extraction unit 312 extracts costume data CD from the first database DB1 based on the selection information acquired by the acquisition unit 311. More specifically, the extraction unit 312 uses the costume ID included in the selection information to extract costume data CD linked to the costume ID from the first database DB1.

The communication control unit 313 causes the communication device 33 to transmit the set of costume ID and image data PD stored in the first database DB1 to the terminal device 10. As an example, the communication control unit 313 causes the communication device 33 to transmit all costume ID and image data PD pairs stored in the first database DB1 to the terminal device 10. Furthermore, the communication control unit 313 outputs the costume data CD extracted by the extraction unit 312 to the terminal device 10 via the communication device 33 as corresponding data RD that is paired with the costume ID to which the costume data CD corresponds. .

1-1-4: Configuration of Terminal Device FIG. 9 is a block diagram showing an example of the configuration of the terminal device 10. The terminal device 10 includes a processing device 11 , a storage device 12 , a communication device 13 , an imaging device 14 , a display 15 , and an input device 16 . Each element included in the terminal device 10 is interconnected using a single bus or multiple buses for communicating information.

The processing device 11 is a processor that controls the entire terminal device 10. Further, the processing device 11 is configured using, for example, a single chip or a plurality of chips. The processing device 11 is configured using, for example, a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, registers, and the like. Note that some or all of the functions of the processing device 11 may be implemented using hardware such as a DSP, an ASIC, a PLD, and an FPGA. The processing device 11 executes various processes in parallel or sequentially.

The storage device 12 is a recording medium that can be read and written by the processing device 11. Furthermore, the storage device 12 stores a plurality of programs including the control program PR1 executed by the processing device 11. The storage device 12 also stores a second database DB2, corresponding data RD, body shape data BD, and learning model LM.

FIG. 10 shows an example of the configuration of the second database DB2. The second database DB2 is a database that stores a set of costume ID and image data PD acquired from the server 30 via the communication device 13 by the acquisition unit 111, which will be described later. As an example, the second database DB2 stores all pairs of costume IDs and image data PD stored in the server 30.

FIG. 11 shows an example of the configuration of the corresponding data RD. The correspondence data RD is correspondence data RD that an acquisition unit 111 (described later) acquires from the server 30 via the communication device 13. As described above, the corresponding data RD is a set of the costume ID included in the selection information output from the terminal device 10 to the server 30 and the costume data CD extracted by the server 30 using the costume ID.

Returning to FIG. 9, the body shape data BD is data representing the body shape of the end user U in three dimensions. Specifically, the body shape data BD is data indicating a three-dimensional image representing the body shape of the end user U. The body shape data BD includes skeleton data KD representing the skeleton of the end user U. More specifically, the skeleton data KD represents a change in the posture of the end user's U skeleton in accordance with the end user's U motion. The skeleton data KD includes, for example, temporal and discrete data indicating the posture of the skeleton of the end user U at a plurality of points in time during the period in which the end user U uses the terminal device 10. Furthermore, the skeleton data KD includes data regarding the joints of the end user U.

The trained model LM is a trained model used when the data generation unit 112 (described later) generates body shape data BD of the end user U based on a captured image of the end user U captured by the imaging device 14 (described later). be. That is, the trained model LM receives the captured image of the end user U as input, and outputs the body shape data BD. The trained model LM is a trained model that has learned the relationship between a plurality of captured images of different end users and a plurality of body shape data that correspond one-to-one with the plurality of end users. The learned model LM is generated by learning teacher data in the learning phase. The teacher data used to generate the learned model LM includes a plurality of sets of data representing a captured image of the end user and body shape data. Further, the trained model LM is generated outside the terminal device 10. In particular, it is preferable that the learned model LM be generated in a server (not shown). In this case, the terminal device 10 acquires the learned model LM from a server (not shown) via the communication network NET.

The communication device 13 is hardware as a transmitting/receiving device for communicating with other devices. The communication device 13 is also called, for example, a network device, a network controller, a network card, a communication module, or the like. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. Furthermore, the communication device 13 may include a wireless communication interface. Examples of connectors and interface circuits for wired connections include products compliant with wired LAN, IEEE1394, and USB. Furthermore, examples of the wireless communication interface include products compliant with wireless LAN, Bluetooth (registered trademark), and the like.

The imaging device 14 images the outside world where the object exists. In particular, in this embodiment, the imaging device 14 captures a full-body image of the end user U. For example, when the terminal device 10 is installed in a store that sells costumes, the imaging device 14 captures a full-body image of the end user U who has visited the store in the state of clothing at the time of the visit. Further, the imaging device 14 outputs imaging information indicating a captured image obtained by imaging the end user U. Furthermore, the imaging device 14 includes, for example, a lens, an imaging element, an amplifier, and an AD converter. The light collected through the lens is converted by the image sensor into an image signal, which is an analog signal. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is output to the processing device 11.

The display 15 is a device that displays images and text information. The display 15 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display 15.

FIG. 12 is an example of the operation screen OM displayed on the display 15. In the operation screen OM illustrated in FIG. 12, a list of three-dimensional images of costumes that are candidates for costumes to be tried on by the end user U is displayed in the leftmost column. In the example of FIG. 12, three-dimensional images of candidates for a T-shirt, long-sleeved shirt, sweater or hoodie, jacket, pants, and coat are displayed in order from the top. These three-dimensional images are shown in the second database DB2 by image data PD linked to the costume ID of each costume. The end user U selects an outfit to try on from among these outfit candidates using the input device 16, which will be described later. In the operation screen OM illustrated in FIG. 12, a three-dimensional image of the costume selected by the end user U is displayed to the right of the costume candidate field. Furthermore, on the operation screen OM illustrated in FIG. 12, a first composite image SP1, which is a three-dimensional image of the whole body of the end user U, is displayed to the right of the column in which the selection results of the end user's U costume are displayed. Displayed as a virtual try-on result. The three-dimensional image of the whole body of the end user U moves in accordance with the actual movement of the end user U.

Returning to FIG. 9, the input device 16 accepts operations from the end user U. For example, the input device 16 includes a keyboard, a touch pad, a touch panel, or a pointing device such as a mouse. Here, when the input device 16 includes a touch panel, it may also serve as the display 15.

The processing device 11 reads the control program PR1 from the storage device 12 and executes it. As a result, the processing device 11 functions as an acquisition section 111 , a data generation section 112 , a reception section 113 , a communication control section 114 , a stack order specification section 115 , an image generation section 116 , and a display control section 117 .

The acquisition unit 111 acquires the set of costume ID and image data PD and the corresponding data RD from the server 30 via the communication device 13. The acquisition unit 111 stores the set of costume ID and image data PD acquired from the server 30 in the second database DB2. The acquisition unit 111 also stores the corresponding data RD acquired from the server 30 in the storage device 12 . Furthermore, the acquisition unit 111 acquires imaging information indicating a captured image of the end user U's whole body from the imaging device 14 .

The data generation unit 112 generates body shape data BD that represents the body shape of the end user U in three dimensions by inputting the imaging information acquired by the acquisition unit 111 into the learning model LM. The body shape of the end user U indicated by the body shape data BD is data indicating the body shape of the end user U without clothes on. Further, as described above, the body shape data BD includes the end user U's skeletal data KD. The skeleton data KD is data representing the skeleton of the end user U.

The reception unit 113 receives two or more costumes selected by the end user U from among the plurality of costumes. Specifically, the reception unit 113 receives selection information indicating the selection results of two or more costumes, which is input by the end user U using the input device 16 while viewing the operation screen OM.

The communication control unit 114 causes the communication device 13 to transmit the selection information received by the reception unit 113 to the server 30. As described above, in response to the output of the selection information to the server 30, the acquisition unit 111 acquires the corresponding data RD via the communication device 13. The correspondence data RD includes costume data CD corresponding to the costume selected by the end user U. Further, as shown in FIG. 4, the costume data CD includes tag data GD. Furthermore, the tag data GD includes a stacking order index.

The stacking order identifying unit 115 identifies the stacking order of two or more costumes selected by the end user U, based on the tag data GD included in the costume data CD acquired by the acquiring unit 111. As described above, the stacking order specifying unit 115 specifies that the larger the stacking order index of the costume, the more outwardly the clothes are stacked when viewed from the three-dimensional image showing the body shape of the end user U. The stacking order specifying unit 115 is an example of a “specific unit”.

The image generation unit 116 generates the first composite image SP1 based on the body shape data BD and the two or more costume data CDs included in the corresponding data RD. Specifically, the image generation unit 116 superimposes the three-dimensional image of the two or more costumes on the three-dimensional image of the body shape of the end user U according to the stacking order of the two or more costumes specified by the stacking order identifying unit 115. A first composite image SP1 is generated. More specifically, the image generation unit 116 reads the costume ID and costume data CD from the corresponding data RD. Next, the image generation unit 116 refers to the second database DB2 using the read costume ID, thereby reading out the image data PD linked to the costume ID. In addition, the image generation unit 116 superimposes the three-dimensional image of the costume indicated by the image data PD on the three-dimensional image of the body shape of the end user U in the stacking order specified by the stacking order specifying unit 115. A composite image SP1 is generated. When the image generation unit 116 superimposes the 3D image of the costume on the 3D image of the body shape of the end user U, a more natural first composite image SP1 is generated by using the shape data FD included in the costume data CD. be done. Here, a three-dimensional image of the body shape of the end user U is represented by body shape data BD.

As a result, when the end user U wears two or more costumes in layers, the terminal device 10 can specify and simulate the order in which they are layered. In particular, when the end user U using the terminal device 10 wears two or more costumes in layers, there is no need to input information regarding the stacking order of the two or more costumes from the terminal device 10. The end user U can view a full-body image of himself or herself virtually trying on the plurality of costumes, with the stacking order automatically specified.

FIG. 13 shows a functional block diagram of the image generation unit 116. The image generation section 116 includes an area identification section 116-1 and a modification section 116-2.

The area specifying unit 116-1 specifies the first area. In the first area, a three-dimensional image of the first costume worn by the end user U is superimposed on a three-dimensional image showing the body shape of the end user U, without intervening another costume between the body of the end user U. In this case, this is a region in which a part of the three-dimensional image of the first costume is included inside the three-dimensional image of the body shape of the end user U.

The modification unit 116-2 performs modification to push out a part of the three-dimensional image of the first costume included in the first region from the three-dimensional image of the end user U.

FIGS. 14A and 14B are explanatory diagrams of the operation of the image generation unit 116 including the area identification unit 116-1 and the correction unit 116-2. As shown in FIG. 14A, the image generation unit 116 overlays the three-dimensional image CM of the first costume on the three-dimensional image BM of the body shape of the end user U. As a result, a second composite image SP2 corresponding to the set of the three-dimensional image BM of the body shape of the end user U and the three-dimensional image CM of the first costume is generated. For convenience of explanation, only outlines of the three-dimensional image BM of the end user U and the three-dimensional image CM of the first costume are shown in FIG. 14A.

As shown in FIG. 14A, in a first region AR1 including a portion corresponding to the shoulders of a three-dimensional image BM of the body shape of the end user U, a first costume is placed inside the three-dimensional image BM of the body shape of the end user U. Part of the 3D image commercial is included. The area specifying unit 116-1 specifies the first area AR1.

FIG. 14B is an enlarged view of the vicinity of the first region AR1 shown in FIG. 14A. As shown by the arrow in FIG. 14B, the modification unit 116-2 executes a modification to push out a part of the 3D image CM of the first costume included in the first region AR1 from the 3D image BM of the end user U. do.

With this modification, when the end user U tries on underwear virtually, for example, it is possible to prevent the three-dimensional image showing the underwear from digging into the three-dimensional image BM showing the body shape of the end user U. As a result, the end user U can see a full-body image of himself wearing underwear in a more natural state.

Although not shown, the area specifying unit 116-1 specifies the second area similarly to the example shown in FIGS. 14A and 14B. The second area is generated when a three-dimensional image of the second costume worn by the end user U over the first costume is superimposed on the second composite image SP2 modified by the modification unit 116-2. This is an area that includes a part of the three-dimensional image of the second costume inside SP2. The modification unit 116-2 performs modification to push out a part of the three-dimensional image of the second costume included in the second region from the second composite image SP2.

With this modification, for example, when an end user U who has virtually tried on underwear virtually tries on a jacket, the three-dimensional image BM showing the body shape of the end user U and the three-dimensional image showing the underwear It is possible to prevent the three-dimensional image showing the jacket from digging into the composite image. As a result, the end user U can see a full-body image of himself wearing the jacket in a more natural state.

In the correspondence data RD, the costume data CD is linked to the costume ID to which each of the first costume and the second costume corresponds. As described above, costume data CD includes joint data JD. In addition, as mentioned above, joint data JD indicates, when a typical wearer wears the costume, the range of joint positions of the typical wearer in the costume. Contains the data shown.

On the other hand, the skeletal data KD included in the body shape data BD includes data indicating the positions of the joints of the end user U. The modification unit 116-2 further expands and contracts the first costume and the second costume, so that the positions of the joints of the end user U included in the skeletal data KD are adjusted to the joint positions of a typical wearer in one costume. may be included in the corresponding range.

Returning to FIG. 9, the image generation unit 116 further compares the joint data JD with the skeletal data KD representing changes in the posture of the end user U in accordance with the movements of the end user U, and generates the first composite. A moving image is generated as image SP1. As mentioned above, joint data JD indicates where the general wearer's skeleton is located relative to the costume, assuming that the costume is worn by a general wearer. Contains data indicating whether The image generation unit 116 adjusts the movement of the end user U while ensuring that the range in which the skeleton of a typical wearer is relatively located in the costume includes the position of the skeleton of the end user U indicated by the skeleton data KD. The three-dimensional image of the costume is transformed accordingly. In transforming the three-dimensional image of the costume, the structure data SD and texture data TD included in the costume data CD are used.

As a result, the terminal device 10 can simulate, as a moving image, a full-body image of the end user U while trying on a plurality of costumes. The end user U can view a video in which a virtual full-body image of the end user U who has tried on a plurality of costumes moves in accordance with the user's own movements. In particular, the terminal device 10 moves the joint data JD included in the shape data FD indicating the three-dimensional shape of the costume in accordance with the movement of the skeleton data KD that deforms in accordance with the movement of the end user U. It becomes possible to move the virtual full-body image of U more naturally.

The display control unit 117 causes the display 15 to display the operation screen OM shown in FIG. In particular, the display control unit 117 causes the display 15 to display the first composite image SP1 generated by the image generation unit 116.

1-2: Operation of the first embodiment FIG. 15 is a flowchart showing the operation of the terminal device 10 according to the first embodiment.

In step S1, the processing device 11 functions as the acquisition unit 111. The processing device 11 acquires imaging information indicating a captured image of the end user U's whole body from the imaging device 14 .

In step S2, the processing device 11 functions as the data generation unit 112. The processing device 11 uses the imaging information acquired in step S1 to generate body shape data BD that represents the body shape of the end user U in three dimensions.

In step S3, the processing device 11 functions as the reception unit 113. The processing device 11 receives selection information input by the end user U using the input device 16 and indicating the selection results of two or more costumes.

In step S4, the processing device 11 functions as the communication control unit 114. The processing device 11 causes the communication device 13 to transmit selection information indicating the selection result received in step S3 to the server 30.

In step S5, the processing device 11 functions as the acquisition unit 111. The processing device 11 obtains correspondence data RD from the server 30 via the communication device 13 .

In step S6, the processing device 11 functions as the stacking order specifying unit 115. The processing device 11 specifies the stacking order of two or more costumes based on the costume data CD included in the corresponding data RD acquired in step S5.

In step S7, the processing device 11 functions as the image generation unit 116. The processing device 11 generates a first composite image SP1 based on body shape data BD representing the body shape of the end user U in three dimensions and two or more pieces of costume data CD. Specifically, the processing device 11 adds two or more images to the three-dimensional image of the end user U by referring to the shape data FD included in the costume data CD according to the stacking order of the two or more costumes specified in step S6. A first composite image SP1 is generated by overlapping the three-dimensional images of the costumes described above.

In step S8, the processing device 11 functions as the display control section 117. The processing device 11 causes the display 15 to display the first composite image SP1 generated in step S7.

1-3: Effects of the First Embodiment According to the above description, the terminal device 10 as a simulation device can provide information to end users by referring to a plurality of costume data CDs in one-to-one correspondence with a plurality of costumes. A case is simulated in which U wears layers of clothing. Each of the plurality of costume data CDs includes shape data FD indicating the three-dimensional shape of the costume, tag data GD associated with the shape data FD, and indicating the number of times the costume will be worn by the end user U. including. The terminal device 10 includes a receiving section 113, a stacking order specifying section 115, and an image generating section 116. The reception unit 113 receives two or more costumes selected by the end user U from among the plurality of costumes. The stacking order identifying unit 115 identifies the stacking order of the two or more costumes based on the two or more costume data CDs corresponding to the two or more costumes received by the reception unit 113. The image generation unit 116 generates a stacking order of the identified two or more costumes in the three-dimensional image of the end user U based on the body shape data BD representing the body shape of the end user U in three dimensions and the two or more costume data CD. Accordingly, a first composite image SP1 is generated in which two or more three-dimensional images of costumes are superimposed.

Since the terminal device 10 has the above configuration, when the end user U wears two or more costumes in layers, the order in which they are layered can be specified and simulated. In particular, when the end user U using the terminal device 10 wears two or more costumes in layers, the layering order can be automatically identified without inputting information regarding the layering order of the two or more costumes from the terminal device 10. In this state, the user can see a full-body image of himself or herself virtually trying on the plurality of costumes.

Also, according to the above description, the two or more costumes include the first costume worn by the end user U without any other costume interposed between the first costume and the body of the end user U. The image generation section 116 includes an area identification section 116-1 and a modification section 116-2. The area specifying unit 116-1 specifies the first area AR1. The first area AR1 is such that when the three-dimensional image of the first costume is superimposed on the three-dimensional image of the end user U, a part of the three-dimensional image of the first costume is inside the three-dimensional image of the end user U. This is an area to get into. The modification unit 116-2 performs modification to push out a part of the three-dimensional image of the first costume included in the first region AR1 from the three-dimensional image of the end user U.

Since the terminal device 10 has the above configuration, when the end user U tries on underwear virtually, for example, the 3D image showing the underwear is compared to the 3D image showing the body shape of the end user U. Can eliminate digging. As a result, the end user U can see a full-body image of himself wearing underwear in a more natural state.

Also, according to the above description, the two or more costumes include the second costume that the end user U wears over the first costume. The area specifying unit 116-1 specifies the second area. When the 3D image of the second costume is superimposed on the second composite image SP2 corresponding to the set of the 3D image of the end user U and the 3D image of the first costume, the second area becomes the second composite image SP2. This is an area in which a part of the three-dimensional image of the second costume enters. The modification unit 116-2 performs modification to push out a part of the three-dimensional image of the second costume included in the second region from the second composite image SP2.

Since the terminal device 10 has the above configuration, for example, when an end user U who has virtually tried on an underwear virtually tries on a jacket, the terminal device 10 displays a three-dimensional image showing the body shape of the end user U. It is possible to prevent the three-dimensional image showing the outerwear from digging into the composite image with the three-dimensional image showing the underwear. As a result, the end user U can see a full-body image of himself wearing the jacket in a more natural state.

According to the above description, the body shape data BD includes the skeleton data KD representing the skeleton of the end user U. The shape data FD corresponds to the relative position of the skeleton of the end user U with respect to the above costume and the joints of the end user U in the above costume, assuming that the end user U wears the above costume. Contains joint data JD indicating the position. The image generation unit 116 collates the skeletal data KD, which is deformed according to the motion of the end user U, and the joint data JD, and generates a moving image as the first composite image SP1.

Since the terminal device 10 has the above configuration, the end user U can view a video in which a virtual full-body image of the end user U trying on two or more costumes moves in accordance with the user's own movements. In particular, the terminal device 10 operates the joint data JD included in the shape data FD indicating the three-dimensional shape of the costume in accordance with the movement of the skeleton data KD that deforms in accordance with the end user's U movement. As a result, the terminal device 10 is able to move the virtual full-body image of the end user U more naturally.

According to the above description, the simulation system 1 includes the scanning device 20 and the terminal device 10 described above. The scanning device 20 three-dimensionally scans the costume described above to generate costume data CD.

Since the simulation system 1 has the above-described configuration, the administrator of the simulation system 1 can easily input costume image data PD without manually inputting each data element constituting the image data PD and costume data CD one by one. You can also create a costume data CD. Furthermore, the simulation system 1 can simulate virtual try-on using the image data PD and the costume data CD that are simply produced.

2: Modification The present disclosure is not limited to the embodiments illustrated above. Specific modes of modification are illustrated below. Two or more aspects arbitrarily selected from the examples below may be combined.

2-1: Modification 1
In the above embodiment, the tag data GD includes a stacking order index used by the terminal device 10 to specify the stacking order of two or more costumes. However, the tag data GD may include type data indicating the type of costume instead of the stacking order index. In this case, the storage device 12 provided in the terminal device 10 stores a correspondence table that describes the correspondence between types of clothing such as T-shirts, long-sleeved shirts, and sweaters, and stacking order indices. The stacking order specifying unit 115 may identify the stacking order of the costumes by comparing the types of costumes specified in the costume data CD obtained from the server 30 with the correspondence table.

2-2: Modification 2
In the above embodiment, the user of the scanning device 20 inputs a stacking order index indicating the stacking order of the costumes from the input device 26. However, the scanning device 20 may generate the costume stacking order index using the trained model. More specifically, the generation unit 212 included in the scanning device 20 inputs image data PD representing a three-dimensional image of the costume C generated by itself into the learned model, thereby determining the stacking order index of the costume C. may be output from the trained model. The learned model is generated by machine learning using teacher data including a plurality of sets of image data PD indicating three-dimensional images of costumes and stacking order indices of costumes.

Alternatively, as described in Modification 1, if the tag data GD includes type data indicating the type of costume instead of the stacking order index, the scanning device 20 uses the learned model to identify the type of costume. May be generated. More specifically, the generation unit 212 included in the scanning device 20 learns the type of costume C by inputting image data PD representing a three-dimensional image of the costume C generated by itself into the learned model. It may also be output from the model. The learned model is generated by machine learning using teacher data including a plurality of sets of image data PD indicating three-dimensional images of costumes and types of costumes.

2-3: Modification 3
In the above embodiment, the terminal device 10 outputs selection information indicating the selection results of two or more costumes received by the reception unit 113 to the server 30, and acquires correspondence data RD corresponding to the selection information from the server 30. was. However, the terminal device 10 may not output the selection information to the server 30 and may unconditionally acquire the correspondence data RD for all costumes from the server 30.

2-4: Modification example 4
In the overall configuration shown in FIG. 1, the terminal device 10, the scanning device 20, and the server 30 are separate bodies. However, two or more of the terminal device 10, scanning device 20, and server 30 may be housed in the same housing. That is, two or more devices shown in the overall configuration of FIG. 1 may be realized as a single device.

3: Others (1) In the embodiments described above, the storage device 12, the storage device 22, and the storage device 32 are exemplified as ROM, RAM, etc. discs, Blu-ray discs), smart cards, flash memory devices (e.g. cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppies ) disk, magnetic strip, database, server, or other suitable storage medium. The program may also be transmitted from a network via a telecommunications line. Further, the program may be transmitted from the communication network NET via a telecommunications line.

(2) In the embodiments described above, the information, signals, etc. described may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc., which may be referred to throughout the above description, may refer to voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may also be represented by a combination of

(3) In the embodiments described above, the input/output information may be stored in a specific location (for example, memory) or may be managed using a management table. Information etc. to be input/output may be overwritten, updated, or additionally written. The output information etc. may be deleted. The input information etc. may be transmitted to other devices.

(4) In the embodiments described above, the determination may be made using a value expressed using 1 bit (0 or 1) or a truth value (Boolean: true or false). Alternatively, the comparison may be performed by comparing numerical values (for example, comparing with a predetermined value).

(5) The order of the processing procedures, sequences, flowcharts, etc. illustrated in the embodiments described above may be changed as long as there is no contradiction. For example, the methods described in this disclosure use an example order to present elements of the various steps and are not limited to the particular order presented.

(6) Each of the functions illustrated in FIGS. 1 to 15 is realized by an arbitrary combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using one physically or logically coupled device, or may be realized using two or more physically or logically separated devices directly or indirectly (e.g. , wired, wireless, etc.) and may be realized using a plurality of these devices. The functional block may be realized by combining software with the one device or the plurality of devices.

(7) The programs exemplified in the above-described embodiments are instructions, instruction sets, codes, codes, regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names. Should be broadly construed to mean a segment, program code, program, subprogram, software module, application, software application, software package, routine, subroutine, object, executable, thread of execution, procedure, function, etc.

Additionally, software, instructions, information, etc. may be sent and received via a transmission medium. For example, if the software uses wired technology (coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), etc.) and/or wireless technology (infrared, microwave, etc.) to create a website, When transmitted from a server or other remote source, these wired and/or wireless technologies are included within the definition of transmission medium.

(8) In each of the above embodiments, the terms "system" and "network" are used interchangeably.

(9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a predetermined value, or other corresponding information. It may also be expressed as

(10) In the embodiments described above, the terminal device 10, the scanning device 20, and the server 30 may be mobile stations (MS). A mobile station is defined by a person skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless It may also be referred to as a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology. Further, in the present disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

(11) In the embodiments described above, the terms "connected", "coupled", or any variations thereof refer to direct or indirect connections between two or more elements. Refers to any connection or combination, including the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connection" may be replaced with "access." As used in this disclosure, two elements may include one or more wires, cables, and/or printed electrical connections, as well as in the radio frequency domain, as some non-limiting and non-inclusive examples. , electromagnetic energy having wavelengths in the microwave and optical (both visible and invisible) ranges.

(12) In the embodiments described above, the statement "based on" does not mean "based solely on" unless specified otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

(13) The terms "determining" and "determining" used in this disclosure may encompass a wide variety of operations. "Judgment" and "decision" include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, and inquiry. (e.g., searching in a table, database, or other data structure), and regarding an ascertaining as a "judgment" or "decision." In addition, "judgment" and "decision" refer to receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and access. (accessing) (e.g., accessing data in memory) may include considering something as a "judgment" or "decision." In addition, "judgment" and "decision" refer to resolving, selecting, choosing, establishing, comparing, etc. as "judgment" and "decision". may be included. In other words, "judgment" and "decision" may include regarding some action as having been "judged" or "determined." Further, "judgment (decision)" may be read as "assuming", "expecting", "considering", etc.

(14) In the embodiments described above, when “include”, “including” and variations thereof are used, these terms are used in the same manner as the term “comprising”. , is intended to be comprehensive. Furthermore, the term "or" as used in this disclosure is not intended to be exclusive or.

(15) In the present disclosure, when articles are added by translation, such as a, an, and the in English, the present disclosure does not include the fact that the nouns following these articles are plural. good.

(16) In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." Note that the term may also mean that "A and B are each different from C". Terms such as "separate", "coupled", etc. may also be interpreted similarly to "different".

(17) Each aspect/embodiment described in the present disclosure may be used alone, in combination, or may be switched and used in accordance with execution. In addition, notification of prescribed information (for example, notification of "X") is not limited to explicit notification, but may also be done implicitly (for example, by not notifying the prescribed information). Good too.

Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modifications and changes without departing from the spirit and scope of the present disclosure as determined by the claims. Accordingly, the description of the present disclosure is for illustrative purposes only and is not meant to be limiting on the present disclosure.

DESCRIPTION OF SYMBOLS 1... Simulation system, 10... Terminal device, 11... Processing device, 12... Storage device, 13... Communication device, 14... Imaging device, 15... Display, 16... Input device, 20... Scanning device, 21... Processing device, 22 ...Storage device, 23...Communication device, 24...Imaging device, 25...Display, 26...Input device, 30...Server, 31...Processing device, 32...Storage device, 33...Communication device, 34...Display, 35...Input device , 111... Acquisition section, 112... Data generation section, 113... Reception section, 114... Output section, 115... Stacking order specification section, 116... Image generation section, 116-1... Area specification section, 116-2... Correction section, DESCRIPTION OF SYMBOLS 117... Display control part, 211... Acquisition part, 212... Generation part, 213... Analysis part, 214... Output part, 311... Acquisition part, 312... Extraction part, 313... Output part, AR1... 1st area, DB1... th... 1 database, DB2...second database, DS1...first data set, PR1, PR2, PR3...control program, SP1...first composite image, SP2...second composite image

Claims

a processing device that simulates a case in which a user wears multiple costumes by referring to a plurality of costume data in one-to-one correspondence with a plurality of costumes;
Each of the plurality of costume data is
Shape data indicating the three-dimensional shape of the costume,
tag data associated with the shape data and indicating the number of times the costume is worn by the user;
The processing device includes:
a reception unit that receives input of two or more costumes selected by the user from among the plurality of costumes;
a specifying unit that specifies a stacking order indicating that the two or more outfits are stacked on top of each other, based on two or more outfit data corresponding to the two or more outfits received by the reception unit;
Based on the body shape data representing the three-dimensional body shape of the user and the two or more costume data, three-dimensional images of the two or more costumes are superimposed on the three-dimensional image of the user according to the specified stacking order. an image generation unit that generates a first composite image;
A simulation device comprising:
The two or more costumes include a first costume worn by the user without intervening another costume between the corresponding costume and the user's body,
The image generation unit includes:
When a three-dimensional image of the first costume is superimposed on the three-dimensional image of the user, a first area including a part of the three-dimensional image of the first costume is identified within the three-dimensional image of the user. an area identification unit to
a modification unit that pushes a part of the three-dimensional image of the first costume included in the first area to the outside of the three-dimensional image of the user;
The simulation device according to claim 1, comprising:
The two or more costumes include a second costume that the user wears over the first costume,
The area specifying unit adds the second costume to a second composite image corresponding to a set of a three-dimensional image of the user modified by the modification unit and a three-dimensional image of the first costume modified by the modification unit. when the three-dimensional images of the second costume are superimposed, specifying a second region in which a part of the three-dimensional image of the second costume is included in the second composite image;
The simulation device according to claim 2, wherein the modification unit pushes a part of the three-dimensional image of the second costume included in the second region to the outside of the second composite image.
The body shape data includes skeletal data representing changes in the posture of the user's skeleton in accordance with the user's movements,
The shape data includes joint data regarding the user's joints when the user wears the costume indicated by the shape data,
The joint data indicates a relative position of the skeleton of the user with respect to the costume, and a position corresponding to the joint of the user in the costume,
The simulation device according to claim 1, wherein the image generation unit generates a moving image as the first composite image by comparing the skeletal data with the joint data.