JP2002058045A - System and method for entering real object into virtual three-dimensional space - Google Patents

Abstract

PROBLEM TO BE SOLVED: To simply fetch tree-dimensional model data of a real object such as a man's body, an article, or the like into a virtual three-dimensional space to make the three-dimensional model take different pause or perform a motion. SOLUTION: An object 7 such as a man's body, an article, or the like is photographed by a plurality of sets of multiple-eye stereoscopic camera (for example, 9 video camera disposed in 3×3 matrix) installed in a store, etc., and output dynamic picture image data is transmitted to a modeling server 1 through a network 8. The modeling server 1 forms a three-dimensional model of the object 7 from the dynamic picture image. This three-dimensional model is formed so as to move based on a motion of the object 7 when photographing. A user system 4 receives the three-dimensional model, and fetches it into a virtual three-dimensional space of applications such as a virtual fitting of cloths, an experience game, or the like to move it.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a system and a method for displaying three-dimensional model data of a real object in a virtual three-dimensional space.

[0002]

2. Description of the Related Art A system using a virtual three-dimensional space by a computer has been proposed for various uses, such as trying on clothes and experiencing games. For example, JP-A-10-
No. 124574 discloses a method of creating three-dimensional model data of a user's body from a photograph or dimensional data of the user's body, incorporating the three-dimensional model of the user's body into a virtual three-dimensional space of a computer, and creating a three-dimensional model of clothes. There is disclosed a system that allows a try-on simulation of clothes or lipstick by attaching or attaching lipstick color data. A fitting system similar or related to this is disclosed in JP-A-10-340282 and JP-A-11-2102.
No. 03347 and JP-A-11-265243.

[0003] Japanese Patent Application Laid-Open No. 11-3437 discloses a system in which a virtual three-dimensional space is applied to an interactive game. This is because texture mapping of a two-dimensional image of a game player captured by a camera onto a three-dimensional model of a character existing in a virtual three-dimensional space of the game makes the player himself a character of the game. It is to direct.

[0004]

In the above-described conventional game system, the three-dimensional model of the character existing in the virtual three-dimensional space has a fixed shape that is completely unrelated to the physical characteristics of the game player. It has something. In that respect, the level of reality that the player himself becomes a character in the game is still unsatisfactory. On the other hand, in the above-mentioned conventional fitting system, since the three-dimensional model data of the user's body is taken into the virtual three-dimensional space, the reality that the user himself tries on the fitting is very high.

[0005] However, the above-mentioned prior art does not provide any specific method or means for creating three-dimensional model data of a user's body. If generating the three-dimensional model data of the user's body requires the user to have very expensive facilities or requires a lot of trouble and cost, the virtual space as described above is used. It is difficult to put a system that uses the technology into practical use.

Accordingly, an object of the present invention is to generate three-dimensional model data of a real object such as a human body or an article without imposing an excessive burden on a user, and to generate the three-dimensional model in a virtual three-dimensional space. To be able to capture it.

[0007] Another object of the present invention is to further enhance the reality of a virtual three-dimensional space in which three-dimensional model data of a real object is fetched. And to make the user perform motion.

[0008]

According to a first aspect of the present invention, there is provided a system for allowing a real object to appear in a virtual three-dimensional space of a computer application used by a user. A real image data receiving unit that can communicate with a stereo image capturing device and receives real image data obtained by stereo shooting an actual object from the stereo image capturing device;
Modeling means for creating a three-dimensional model of the object in a predetermined data format that can be taken into a three-dimensional space, and outputting the created three-dimensional model data in a manner that can be provided to a user or a computer application used by the user And a three-dimensional model output means.

Using this system, a user takes an image of an object, such as his or her body or an article, which the user wants to capture in a virtual three-dimensional space of a computer application, using a stereo imaging device, and obtains the actual image data of the object. When transmitted to the system, the three-dimensional model data of the object can be received from the system, so that the received three-dimensional model data can be imported into the own computer application.

In a preferred embodiment, the system resides on a communication network, such as the Internet, as a modeling server. Then, for example, a user photographs a desired object with a stereo photographing device installed in a store such as a department store, a game center or a convenience store, or a stereo photographing device owned by the user himself, and the actual photograph data is transmitted through a communication network. When the data is transmitted to the modeling server, the modeling server creates a three-dimensional model of the object, and returns the three-dimensional model to the computer system of the store or the user's own computer system through the communication network. Thus,
A user can easily acquire a three-dimensional model of a desired object and incorporate it into a desired application such as virtual fitting or an interactive game.

In a preferred embodiment, the actual photograph data of the object photographed by the stereo photographing apparatus includes actual photograph data of a plurality of poses photographed when the object takes different poses. For example, if the stereo photographing device uses a video camera, for example, when the user takes a picture of himself or herself while taking various poses or performing a certain motion, many different poses can be obtained. Actual photograph data is obtained. The modeling means receives the real shot data of such different poses, and creates three-dimensional model data having a configuration capable of taking different poses or performing motions based on the data. This allows the user to capture the created three-dimensional model data into the virtual three-dimensional space of the computer application and cause the three-dimensional model to take various different poses or perform motions.

In a preferred embodiment, the stereo photographing apparatus uses a video camera, and when a real object is performing a certain motion, it photographs the motion and outputs moving image data of the motion. The modeling means receives the moving image data and creates three-dimensional model data based on the moving image data so as to perform the same motion as that performed by the real object. Therefore, the user can cause the three-dimensional model to perform the same motion as the real object in the virtual three-dimensional space. Further, in a preferred embodiment, the modeling means generates the three-dimensional model data so as to follow the motion performed by the real object substantially in real time during the shooting by the stereo shooting device and to perform the same motion. I do. for that reason,
For example, if the user captures his / her own three-dimensional model data output in real time from the modeling means into a virtual three-dimensional space such as a game, for example, while taking a picture of himself / herself, when the user makes a certain motion, In the virtual three-dimensional space, the user's three-dimensional model performs exactly the same motion.
Thereby, high reality as if the user himself entered the virtual three-dimensional space can be obtained.

A system according to a second aspect of the present invention is a combination of the above-described stereo photographing device and a modeling device.

A system according to a third aspect of the present invention is a combination of the above-described stereo photographing device and modeling device, and a computer device capable of executing a computer application for taking a created three-dimensional model into a virtual three-dimensional space. is there.

[0015]

FIG. 1 shows an overall configuration of a virtual fitting system according to an embodiment of the present invention.

A modeling server 1 and a computer system (hereinafter referred to as a “costume supplier system”) 2 managed by costume suppliers such as costume manufacturers and costume shops.
, A virtual fitting server 3, a user's computer system (for example, a personal computer or a game computer, hereinafter, referred to as a "user system") 4, and a computer installed in a store such as a department store, a game center, or a convenience store. A system (hereinafter referred to as a “store system”) 5 is communicably connected through a communication network 8 such as the Internet. FIG. 1 shows only one modeling server 1, costume supplier system 2, virtual fitting server 3, user system 4, and store system 5, but there may be a plurality of each. In particular,
Usually, a plurality of costume supplier systems 2, user systems 4, and store systems 5 exist in accordance with the number of costume suppliers, users, and stores, respectively.

Stores such as department stores, game centers and convenience stores are also provided with a stereo photographing system 6 connected to the store system 5. As will be described later in detail with reference to FIG.
A space 6A such as a room having a size in which the user 7 can enter and take various poses, and a plurality of multi-view stereo cameras 6B arranged around the space 6A so as to be able to photograph the space 6A. , 6B,... Each of the multi-view stereo cameras 6B is composed of, for example, nine video cameras arranged in a 3 × 3 matrix, and real-image data output from the nine cameras is photographed using stereoscopic vision as described later. It is used to create a distance image of the object. As shown, a user 7 enters the space 6A of the stereo photographing system 6, and when the user 7 is photographed by a plurality of multi-view stereo cameras 6B, 6B,..., The multi-view stereo cameras 6B, 6B,. Is transmitted to the store system 5.

The store system 5 sends the actual photographing data of the user's body received from the stereo photographing system 6 to the modeling server 1 via the communication network 8. The modeling server 1 creates three-dimensional model data of the user's body by performing processing as will be described later in detail with reference to FIGS. 9 to 13 using the real image data of the user's body received from the store system 5. I do. Modeling server 1
Saves the created three-dimensional model data of the user's body in the user database 1A, and thereafter transmits the three-dimensional model data of the user's body to the store system 5 through the communication network 8. The store system 5 sends the three-dimensional model data of the user's body to the user system 4 via the communication network 8 (or via a portable recording medium such as a recording disk). Alternatively, the modeling server 1 may directly transmit the three-dimensional model data of the user's body stored in the user database 1A to the user system 4 via the communication network 8 when requested by the user system 4.

Incidentally, it is also possible for the user to have the stereo photography system 6 himself. In that case, a plurality of (for example, two or three) multi-view stereo cameras 6B and 6
.. Are installed in their own rooms, and real-image data from the multi-view stereo cameras 6B, 6B,.
May be sent to the modeling server 1 through. The price of the multi-view stereo camera 6B itself is less than 100,000 yen at the time of filing of the present application, and will be further reduced in the future. Therefore, the number of users who can have the stereo photography system 6 by themselves will increase in the future. There will be.

The costume supplier system 2 creates three-dimensional model data of various costumes (clothes, shoes, hats, accessories, bags, etc.) provided by the costume supplier and accumulates them in the costume database 2A. The three-dimensional model data of the costumes is sent to the virtual fitting server 3 via the communication network 8 or a disk recording medium. Alternatively, the costume supplier system 2 photographs the costume (or a person wearing the costume) with a stereo photography system similar to or similar to the stereo photography system 6 of the store, and sends the actual photography data to the modeling server 1 for modeling. The server 1 may create three-dimensional model data of the costume, and may receive the three-dimensional model data of the costume from the modeling server 1 and send it to the virtual fitting server 3 (or directly communicate from the modeling server 1). It may be sent to the virtual fitting server 3 via the network 8).

The virtual fitting server 3 is, for example, a website of a department store or a clothing store, and stores three-dimensional model data of various costumes received from the costume supplier system 2 or the like in the costume database 3A for each supplier. And a virtual fitting program 3B that can be executed by the user system 4. And
When requested by the user system 4, the virtual fitting server 3 sends the three-dimensional model data of the various costumes and the virtual fitting program to the user system 4 via the communication network 8.

The user system 4 includes the modeling server 1
The user installs the three-dimensional model data of the user's body, the three-dimensional model data of various costumes received from the virtual fitting system 3 and the virtual fitting program received from the virtual fitting system 3 in an auxiliary storage device 4A such as a hard disk drive, and Execute the virtual try-on program according to the instructions of. The three-dimensional model data of the user's body and the three-dimensional model data of the costume are made in a predetermined data format so that the virtual fitting program can take them into the virtual three-dimensional space. The virtual fitting program takes in the 3D model data of the user's body and the 3D model data of various costumes into a virtual 3D space, and puts the desired costume on the user's 3D model in the virtual 3D space. It poses and moves in any way you want it to appear, renders an image of its appearance from any viewpoint, and displays it on the display screen. The virtual fitting program also maps any color or texture to any part of the three-dimensional model data of the user's body or costume by using a known technique, so that the user can tan, apply various makeups, or apply hair. You can simulate the appearance of dyeing or changing the color of clothing,
By adding enlargement, reduction, deformation, and replacement with other models to the dimensional model data using known techniques, it is possible to simulate the appearance of fat, thin, tall, and changed hairstyles. it can. Further, the virtual fitting program can receive an order for an arbitrary costume from the user and send the order to the virtual fitting server 3.

According to this virtual fitting system, the user can go to a store such as a department store, a game center or a convenience store and install the stereo photographing system installed there without having the equipment for three-dimensional modeling. By photographing one's own body in step 6, the user can create three-dimensional model data of his own body and import it into his own computer. Using his own three-dimensional model data, the virtual three-dimensional space of the computer You can try on various costumes with high reality. In addition, as will be described later, it becomes possible to take in and use the user's own three-dimensional model data not only in virtual fitting but also in a virtual three-dimensional space of another application such as an interactive game. Also, in costume suppliers, etc., actual photographed data of the user or 3
By acquiring and utilizing the dimensional model data in a manner that does not invade the privacy of the user with the consent of the user, it is possible to design and manufacture costumes optimally suited to the user's body at lower cost than usual, It is possible to develop and design new ergonomically advanced costumes based on detailed data of the human body that cannot be obtained by measurement.

FIGS. 2 and 3 show the processing procedure of this virtual fitting system in more detail. FIG. 2 shows a processing procedure for creating three-dimensional model data of the user's body, which is performed mainly by the modeling server 1. FIG. 3 shows a processing procedure for causing the user system to execute the virtual fitting performed mainly by the virtual fitting server 3.

First, a processing procedure for creating three-dimensional model data of a user's body will be described with reference to FIGS.

(1) As shown in FIG. 1, the user 7
Go to a store such as a department store, a game center, or a convenience store, pay a fee, and enter the stereo photographing system 6 installed there with light clothes as much as possible.

(2) As shown in FIG. 2, when the store system 5 receives a fee from the user, the store system 5
(Step S11), the modeling server 1 accepts access from the store system 5 (S1).

(3) On the side of the store system 5, the whole body of the user is photographed by the stereo photographing system 6, and actual photographing data of the whole body is transmitted to the modeling server 1 (S 12). The modeling server 1 receives actual photograph data of the whole body (S2).

(4) The modeling server 1 creates body shape three-dimensional model data representing the shape of the whole body of the user based on the received real-body image data of the whole body (S3).

(5) On the side of the store system 5, the stereo photographing system 6 photographs a local area, typically a user's face, to be modeled in more detail than the whole body. 1 (S13). The modeling server 1 receives the local photographed data (S4). In addition, this local imaging may be performed by a method of imaging only the local at a higher magnification or higher resolution than the whole body, separately from the imaging of the whole body, or a method necessary for local imaging from the beginning. The whole body and the local area may be photographed at the same time by photographing the whole body at a magnification or high resolution. (In the latter case, the actual photographed data of the whole body can be reduced to a necessary and sufficient low resolution after the photographing. it can).

(6) The modeling server 1 creates local three-dimensional model data representing the shape of the user's local area, particularly the face, based on the received local real data (S5).

(7) The modeling server 1 inserts the corresponding local three-dimensional model data into a local part such as a face of the three-dimensional body model data of the whole body to thereby obtain the detailed shape of the whole body and the local part such as the face of the user. A standard whole body model representing the shape is created (S6). The modeling server 1 transmits the standard full-body model to the store system 5 (S7), and the store system 5 receives the standard full-body model (S1).
4).

(8) The store system 5 transmits the received standard full-body model to the user system 4 via the communication network 8 or outputs it to a portable recording medium such as a CD-ROM (S15). The user system 4 is connected to the store system 5 via the communication network 8 or C
The standard whole body model is received and stored from a portable recording medium such as a D-ROM (S21). Here, the received standard full-body model may be rendered by the store system 5 or the user system 4 and displayed on the display screen so that the user can confirm whether or not there is a problem with the standard full-body model.

(9) The modeling server 1 collects a fee from the store (or user) when the store system 5 normally receives the standard full-body model and it is confirmed that there is no problem with the standard full-body model. Is performed, and the data of the charging result is sent to the store system 5 (S8).
The store system 5 receives the charging result data (S1).
6).

Next, with reference to FIG. 1 and FIG. 3, a processing procedure for causing the user system to execute virtual fitting will be described.

(1) The costume supplier system 2 creates three-dimensional model data of various costumes (S31),
The virtual fitting server 3 transmits the data to the virtual fitting server 3 (S32), and receives the three-dimensional model data of the various costumes and stores the data in the costume database (S41).

(2) The user system 4 applies for access to the virtual fitting system 3 at any time (S5).
1). The virtual fitting system 3 includes a user system 4
When the access is received (S42), the virtual try-on program and the three-dimensional model data of various costumes are transmitted to the user system 4 (S43). The user system 4 installs the received virtual fitting program and the three-dimensional model data of various costumes on its own device so that the virtual fitting program can be executed (S52). It is not always necessary to download the virtual fitting program and the three-dimensional model data of the costume from the virtual fitting system 3 to the user system 4 at the same time. The virtual fitting program and the three-dimensional model data of the costume may be downloaded at another time, or the virtual fitting program and one or both of the three-dimensional model data of the costume may be downloaded to a CD-ROM instead of through a communication network. It may be recorded on a solid recording medium such as a ROM, distributed to users, and installed in the user system 4.

(3) The user system 4 executes the virtual try-on program at any time (S53).

(4) The user can input an order for an arbitrary costume into the virtual fitting program, and the virtual fitting program transmits order data for the costume to the virtual fitting server 3 ( S54).

(5) Upon receiving the order data from the user system 4, the virtual fitting server 3 sends the order data to the costume supplier system 2 of the costume supplier that provides the costume.
The order data for the costume is sent (S44), and the settlement processing is performed to send settlement-related data such as an invoice to the user system 4 or the costume supplier system (S45). Costume Supplier System 2
Receives order data and payment-related data from the virtual fitting server 3 and performs necessary business processing (S33, S3)
4). The user system 4 receives the payment-related data from the virtual fitting server 3 and causes the user to confirm the data (S
55).

FIG. 4 shows an example of a virtual fitting window displayed on the display screen of the user system by the virtual fitting program.

The virtual fitting window 500 includes a show stage window 501, a camera control window 502, a model control window 503, and a costume room window 504.

The virtual try-on program causes the user's standard full body model 506 to stand on the stage in a virtual three-dimensional space imitating the space of the fashion show stage, and causes the standard full body model 506 to take a predetermined pose. Alternatively, a predetermined motion is performed, a two-dimensional color image photographed at a predetermined zoom magnification by a camera arranged at a predetermined position is rendered, and the two-dimensional color image is rendered as shown in the show stage window 5 as shown in the figure.
01 is displayed.

In the costume room window 504, two-dimensional color images 508, 508,... Of the three-dimensional models of various costumes viewed from the front in a basic form, and "wear" and "take off"
And a button of “to shopping cart” is displayed. When the user selects any one of the costume images 508, 508,... Displayed on the costume room window 504 and operates the “wear” button, the virtual fitting program executes the user standard displayed on the show stage window 501. The selected costume 3
The dimensional model 507 is put on. When the user operates the “take off” button, the virtual try-on program removes the three-dimensional model 507 of the selected costume from its standard full-body model 506.

When the user operates the “front”, “back”, “left” or “right” button in the camera control window 502, the virtual fitting program causes the user's standard whole body model 506 to be displayed in the virtual three-dimensional space. Since the position of the camera that is shooting is moved forward, backward, left, or right, the image displayed in the show stage window 501 changes according to the movement of the camera. When the user operates the “zoom in” or “zoom out” button in the camera control window 502, the virtual fitting program increases the zoom magnification of the camera capturing the user's standard full body model 506 in a virtual three-dimensional space. Or reduce it, so show stage window 5
The image displayed at 01 changes according to the change of the zoom magnification.

When the user operates the “pose 1” or “pose 2” button in the model control window 503, the virtual fitting program places the “pose 1” or “pose 1” on the user's standard whole body model 506 in the virtual three-dimensional space. The user is caused to take a pose (for example, a cautious posture, a resting posture, etc.) assigned to “pose 2”. When the user operates the “Motion 1” or “Motion 2” button in the model control window 503, the virtual fitting program places “Motion 1” or “Motion 2” on the user's standard whole body model 506 in the virtual three-dimensional space. (For example, walking to the front end of the stage, making a U-turn, walking and returning, making several turns, etc.). Thus, the user's standard whole body model 5
When causing the user 06 to take a pose or perform a motion specified by the user, the virtual fitting program also moves the three-dimensional model 507 of the costume worn by the user's standard whole body model 506 in accordance with the pose or the motion. .

Thus, the user can wear a desired costume on his / her standard whole body model 506 to perform a fashion show, and confirm the quality of the costume. When the user selects an arbitrary costume image 508, 508,... From the costume room window 504 and operates the “go to shopping cart” button, the virtual fitting program displays the selected costume as a list of order candidates. Add to shopping cart. Later, when the user opens a predetermined order window (not shown) and performs an ordering operation, the virtual fitting program creates order data for the costumes in the shopping cart and transmits the order data to the virtual fitting site.

Now, as shown in FIG. 4, in order to cause the user's standard full body model 506 to take a plurality of poses and perform motions, the user's standard full body model 506 must be able to do so. Must be configured. For example, the following two types of configurations of the standard whole body model 506 can be considered.

(1) The standard whole body model 506 is a separate three-dimensional model in which each part of the body is jointed with a joint, like a real human. By rotating the three-dimensional model of each part with the joint as a fulcrum (ie, bending at the joint), the standard whole body model 506 can take various postures.

(2) A standard whole body model 506 is prepared for each of a number of different poses. If an arbitrary pose is selected from these many standard whole body models 506 and placed in the virtual three-dimensional space, the figure of the arbitrary pose can be displayed. In addition, by replacing these many standard whole body models 506 in a virtual three-dimensional space at high speed in an order according to a pose change of an arbitrary motion, it is possible to display a figure performing an arbitrary motion. .

Of the above, the method of (2) for preparing a standard whole body model having many different poses will be described later with reference to FIGS.
As can be understood from the method of creating a three-dimensional model described with reference to, this can be easily performed by creating a three-dimensional model for each frame of a moving image output from the multi-view stereo camera.

On the other hand, the method (1) of creating a standard whole body model having joints can be performed by a processing procedure as shown in FIGS. 5 and 6, for example. Here, FIG.
It shows a processing flow performed by the modeling server to create a standard whole body model having joints, and corresponds to steps S2 to S6 in FIG. 2 already described. FIG. 6 shows the configuration of a three-dimensional body shape model created in the course of this processing flow.

As shown in step S61 of FIG. 5, the modeling server first receives actual photograph data from the stereo photographing system when the user takes a certain basic pose and a plurality of other modified poses. As described later with reference to FIGS. 9 to 13, a user performs shooting while performing a certain motion in a stereo shooting system, and forms a moving image output from the multi-lens stereo camera at that time. This is equivalent to receiving a series of many frame images (that is, actual photograph data of a number of poses slightly different from each other).

Next, as shown in step S62, the modeling server creates three-dimensional body shape model data of the whole body of the user for each pose from the photographed data of many different poses. The body shape three-dimensional model data of each pose created at this time is, as shown by reference numeral 600 in FIG. 6, three-dimensional model data that captures the whole body of the user as a solid body (hereinafter, referred to as a whole body integrated model). is there.

Next, as shown in step S63, the modeling server sends the whole-body integrated model 60 between different poses.
By comparing 0 and detecting a bending point when it deforms, that is, a fulcrum of rotation of each part, the positions of the joints such as the shoulder, elbow, hip joint, and knee are determined by, for example, the whole body integrated model 600 in the basic pose. I do. Each part of the whole body integrated model 600 separated by the joints is the head, neck,
It determines which part, such as the left and right upper arm, the left and right lower arm, the left and right hand, the chest, the belly, the waist, the left and right thigh, the left and right thigh, the left and right foot, etc.

Next, as shown in step S64, the modeling server sends the whole body integrated model 600 of the basic pose.
Is divided into the above-described three-dimensional models of a number of parts, and as shown by reference numeral 601 in FIG. 6, the three-dimensional models 602 to 618 of those parts are jointed by joints (shown by black dots). (Hereinafter referred to as “part joint model”).

Next, as shown in step S65, the modeling server sends the created part joint model 601
Is applied to a predetermined part (for example, the face part of the head 602), and this is used as a standard whole body model of the user.

By using the standard full-body model that bends at joints created in this way, the virtual fitting program of the user system can cause the standard full-body model to take a plurality of poses and perform motions. it can. FIG. 7 shows a processing flow of the virtual fitting program for that. FIG. 8 illustrates an operation performed on the standard whole body model and the three-dimensional model of the costume performed in the process of this processing flow.

As shown in FIG. 7, in the virtual fitting program, in step S71, the user's standard whole body model 6
Get 01. In step S72, three-dimensional model data of the costume selected by the user is obtained. The three-dimensional model data of this costume is represented by reference numeral 620 in FIG.
A plurality of parts 621 to 6 like the standard whole body model of the user
27, and these portions 621 to 627 are joined by joints indicated by black dots.

Next, as shown in step S73, the virtual fitting program executes a virtual three-dimensional
The three-dimensional model data 620 of the costume is aligned with the standard whole body model 601 of the user (that is, the costume is put on).

Next, as shown in step S74, the virtual fitting program executes a virtual three-dimensional
As shown by reference numeral 630 in FIG. 8, the standard full-body model 601 and the three-dimensional model data 620 of the costume are bent at joints such that the standard full-body model 601 wearing the costume takes a pose or motion specified by the user. And transform it. Then, as shown in step S75, a two-dimensional image obtained by viewing the deformed standard whole body model 601 and the three-dimensional model data 620 of the costume from the camera position designated by the user at the zoom magnification designated by the user. Is rendered, and the show stage window 50 shown in FIG.
1 is displayed.

Hereinafter, the configuration of the stereo photographing system 6 shown in FIG. 1 will be described in detail. FIG. 9 shows a schematic overall configuration of the stereo photography system 6.

A predetermined three-dimensional space 20 into which a modeling object (in this example, a human, but any object) 10 is set is set. At a plurality of different locations around the space 20, multi-view stereo cameras 11, 12, 1
3 are fixed respectively. In this embodiment, there are three multi-view stereo cameras 11, 12, and 13. However, this is a preferable example, and any number of two or more cameras may be used. The lines of sight 14, 15, 16 of the multi-lens stereo cameras 11, 12, 13 extend into the space 20 in different directions.

The output signals of the multi-lens stereo cameras 11, 12 and 13 are input to the arithmetic unit 18. The arithmetic unit 18
Based on input signals from the multi-view stereo cameras 11, 12, and 13, three-dimensional model data of the object 10 is created. Here, the arithmetic unit 18 is shown as a single block for convenience, but the combination of the store system 5 and the modeling server 1 in the virtual fitting system shown in FIG.
It means the functional part that performs dimensional modeling.

Each of the multi-view stereo cameras 11, 12 and 13 has three or more, whose positions are relatively different and their lines of sight are substantially parallel, preferably nine arranged in a 3 × 3 matrix.
, 17R independent video cameras 17S, 17R,. One video camera 17S located at the center of the 3 × 3 matrix is called a “reference camera”. Each of the eight video cameras 17R,..., 17R positioned so as to surround the reference camera 17S is called a “reference camera”. The reference camera 17S and one reference camera 17R are
A pair of stereo cameras, which is a minimum unit applicable to stereo vision, is configured. Therefore, the reference cameras 17S and 8S
The reference cameras 17 constitute eight pairs of stereo cameras arranged radially around the reference camera 17S.
The eight pairs of stereo cameras enable highly accurate and stable distance data for the object 10 to be calculated. Here, the reference camera 17S is a color or monochrome camera. When a color image is to be displayed on the television monitor 19, a color camera is used as the reference camera 17S. On the other hand, a black and white camera is sufficient for the reference cameras 17R,..., 17R, but a color camera may be used.

Each of the multi-view stereo cameras 11, 12, 13 has nine video cameras 17S, 17R,.
Output nine moving images. First, the arithmetic unit 18
Is the 9 output from the first multi-lens stereo camera 11
The latest frame images (still images) of the moving images of the book are captured, and the nine still images (that is, one reference image from the reference camera 17S and eight reference cameras 17R,.
The latest distance image of the object 10 (that is, the object 10 represented by the distance from the reference camera 17S) by the known multi-view stereoscopic vision based on the eight reference images from the 17R.
Image). In parallel with the above, the arithmetic unit 18 uses the same method as described above to execute the second multi-view stereo camera 1.
For both the second and third multi-view stereo cameras 13, the latest distance images of the object 10 are created. Subsequently, the arithmetic unit 18 includes three multi-view stereo cameras 11,
The latest three-dimensional model of the object 10 is created by the method described later in detail using the latest distance image created for each of 12 and 13.

The arithmetic unit 18 repeats the above operation every time the latest frame of a moving image is taken in from the multi-view stereo cameras 11, 12, and 13.
0 is created. When the object 10 moves, it follows the object 10 in real time or in a state close thereto, and the arithmetic unit 1
8 changes the latest three-dimensional model created.

Hereinafter, the internal configuration and operation of the arithmetic unit 18 will be described in more detail.

The arithmetic unit 18 uses the following plural coordinate systems. That is, as shown in FIG. 9, in order to process an image from the first multi-view stereo camera 11, a first camera orthogonal coordinate system having coordinate axes adapted to the position and orientation of the first multi-view stereo camera 11 i1, j1, and d1 are used. Similarly, in order to process images from the second multi-view stereo camera 12 and the third multi-view stereo camera 13, respectively, the positions of the second multi-view stereo camera 12 and the third multi-view stereo camera 13 are set. A second camera orthogonal coordinate system i2, j2, d2 and a third camera orthogonal coordinate system i3, j3, d3 respectively adapted to the orientation are used. Further, a predetermined one global rectangular coordinate system x, y, z is used to define a position in the space 20 and to process the three-dimensional model of the object 10.

As shown in FIG. 9, the arithmetic unit 18 divides the entire space 20 into Nx, Ny, and Nz voxels (voxes) along the coordinate axes of the global coordinate system x, y, and z.
l: a small cube) 30,. Therefore, the space 20 is constituted by Nx × Ny × Nz voxels 30,. A three-dimensional model of the object 10 is created using these voxels 30,. Hereinafter, the overall coordinate system x, y,
The coordinates by z are represented by (vx, vy, vz).

FIG. 10 shows the internal configuration of the arithmetic unit 18.

The arithmetic unit 18 includes the multi-view stereo processing unit 6
1, 62, 63, a pixel coordinate generation unit 64, a multi-view stereo data storage unit 65, voxel coordinate generation units 71, 72, 7
3. It has a voxel data generation unit 74, 75, 76, an integrated voxel data generation unit 77, and a modeling unit 78.
As described above, in the virtual fitting system shown in FIG. 1, the arithmetic unit 18 includes the store system 5 and the modeling server 1.
As to which of the many components 61 to 78 of the arithmetic device 18 is handled by the store system 5 and which is handled by the modeling server 1, various modes can be adopted. Hereinafter, the processing functions of these components 61 to 78 will be described.

(1) Multi-view stereo processing units 61, 62, 6
3. Multi-view stereo processing units 61, 62, 63 are provided for the multi-view stereo cameras 11, 12, 13 in a one-to-one correspondence. Since the functions of the multi-view stereo processing units 61, 62, and 63 are the same as each other, the first multi-view stereo processing unit 61 will be representatively described.

The multi-view stereo processing section 61 transmits the nine video cameras 17S, 17
The latest frames (still images) of the nine moving images output by R,..., 17R are captured. These nine still images are grayscale luminance images in the case of a monochrome camera, and are luminance images of, for example, three color components of R, G, and B in the case of a color camera. The R, G, and B luminance images become gray-scale luminance images similar to those of a monochrome camera when they are integrated. The multi-view stereo processing unit 61 sets a single luminance image from the reference camera 17S (as it is in the case of a black-and-white camera, or in the case of a color camera, by integrating R, G, and B into a gray scale) as a reference image. , And 17R, eight luminance images from the other eight reference cameras (black and white cameras) 17R,..., 17R. Then, the multi-view stereo processing unit 61 creates a pair of each of the eight reference images and the reference image (eight pairs are formed), and for each pair, calculates a parallax for each pixel between both luminance images by a predetermined method. Ask.

Here, as a method of obtaining the parallax, for example, a method disclosed in Japanese Patent Application Laid-Open No. H11-175725 can be used. The method disclosed in Japanese Patent Application Laid-Open No. H11-175725 is simply as follows. First, one pixel is selected on the reference image, and a window area of a predetermined size (for example, 3 × 3 pixels) centered on the selected pixel is extracted from the reference image. Next, a pixel (referred to as a corresponding candidate point) at a position shifted from the selected pixel by a predetermined parallax on the reference image is selected, and a window region of the same size centered on the corresponding candidate point is extracted from the reference image. Then, the similarity of the luminance pattern (for example, between pixels corresponding to positions in both window regions) between the window region of the corresponding candidate point extracted from the reference image and the window region of the selected pixel extracted from the reference image. Calculate the reciprocal of the square addition value of the luminance value difference). While sequentially changing the parallax from the minimum value to the maximum value and moving the corresponding candidate point,
For each corresponding candidate point, the calculation of the similarity between the window area of the corresponding candidate point and the window area of the selected pixel from the reference image is repeated. From the result, the corresponding candidate point with the highest similarity is selected, and the parallax corresponding to the corresponding candidate point is determined as the parallax in the selected pixel. Such determination of parallax is performed for all pixels of the reference image. From the parallax of each pixel of the reference image, the distance between the portion corresponding to each pixel of the target object and the reference camera is determined on a one-to-one basis. Therefore, by calculating the parallax for all the pixels of the reference image, a distance image that represents the distance from the reference camera to the object for each pixel of the reference image is obtained as a result.

The multi-view stereo processing unit 61 calculates a distance image for each of the eight pairs by the above method, integrates the eight distance images by a statistical method (for example, calculates an average), and The result is output as the final distance image D1.
The multi-view stereo processing unit 61 also outputs the luminance image Im1 from the reference camera 17S. Further, the multi-view stereo processing unit 61 outputs a reliability image Re1 representing the reliability of the distance image D1.
And output. Here, the reliability image Re1 is an image indicating the reliability of the distance of each pixel indicated by the distance image D1 for each pixel. For example, from the result of calculating the similarity for each parallax while changing the parallax as described above for each pixel of the reference image, the similarity difference between the parallax with the highest similarity and the parallaxes immediately before and after it is calculated. And can be used as the reliability of each pixel. In this example, the larger the difference between the similarities, the higher the reliability.

As described above, from the first multi-view stereo processing unit 61, three types of outputs of the luminance image Im1, the distance image D1, and the reliability image Re1 viewed from the position of the first multi-view stereo camera 11 are output. can get. Therefore, the luminance images Im1, Im2, Im3, the distance images D1, D2, D3, and the reliability images Re1, R from the three camera positions are obtained from the three multi-view stereo processing units 61, 62, 63.
e2 and Re3 are obtained (the images output from these multi-view stereo processing units are collectively referred to as “stereo output images”).

(2) Multi-view stereo data storage section 65 The multi-view stereo data storage section 65 stores stereo output images from the three multi-view stereo processing sections 61, 62, 63, that is, luminance images Im1, Im2, Im3, and distances. The images D1, D2, D3 and the reliability images Re1, Re2, Re3 are input, and their stereo output images are multi-view stereo processing units 61, 62, 6 as shown in the figure.
3 is stored in the storage areas 66, 67, and 68 corresponding to 3. Then, the multi-view stereo data storage unit 65 stores the coordinates (the coordinates in the camera coordinate system of each of the multi-view stereo cameras 11, 12, and 13 shown in FIG. 9) indicating the processing target pixel from the pixel coordinate generation unit 64. (represented by (i11, j11))
The value of the pixel indicated by the pixel coordinates (i11, j11) is represented by a luminance image Im1, I
m2, Im3, distance images D1, D2, D3 and reliability images Re1, Re2, Re3
And output it.

That is, the multi-view stereo data storage unit 65
When the pixel coordinates (i11, j11) are input, the luminance image Im1, the distance image D1, and the reliability image Re1 in the first storage area 66
From the first camera coordinate system i1, j1, d1 (i11, j
11) The luminance of the pixel corresponding to (11) Im1 (i11, j11) and the distance D1 (i11, j1
1) and the reliability Re1 (i11, j11) are read, and the coordinates in the second camera coordinate system i2, j2, d2 are obtained from the luminance image Im2, the distance image D2, and the reliability image Re2 in the second storage area 67. (i11, j1
The luminance Im2 (i11, j11) and the distance D2 (i11, j1) of the pixel corresponding to (1)
1) and the reliability Re2 (i11, j11), and from the luminance image Im3, the distance image D3, and the reliability image Re3 in the third storage area 68, in the third camera coordinate system i3, j3, d3. Coordinates
The luminance Im3 (i11, j11) of the pixel corresponding to (i11, j11) and the distance D3
(i11, j11) and the reliability Re3 (i11, j11) are read and their values are output.

(3) Pixel Coordinate Generation Unit 64 The pixel coordinate generation unit 64 generates coordinates (i11, j11) indicating the pixel to be subjected to the three-dimensional model creation processing, and stores the multi-view stereo data storage unit 65 and the voxel. Coordinate generators 71, 7
2, 73. The pixel coordinate generation unit 64 performs, for example, raster scanning on the entire range or a partial range of the above-described stereo output image, so that the coordinates (i11, j11) of all the pixels in the range are obtained.
Are sequentially output.

(4) Voxel coordinate generation units 71, 72, 7
3. Three voxel coordinate generation units 71, 72, 73 are provided corresponding to the three multi-view stereo processing units 61, 62, 63. Since the functions of the three voxel coordinate generation units 71, 72, and 73 are the same, the first voxel coordinate generation unit 71 will be described as a representative.

The voxel coordinate generation section 71 receives the pixel coordinates (i11, j11) from the pixel coordinate generation section 64, and stores the pixel coordinates (i11, j11) in the multi-view stereo data storage section 6.
The distance D1 (i1
Enter 1, j11). Input pixel coordinates (i11, j11) and distance D
1 (i11, j11) indicates the coordinates of one location on the outer surface of the object 10 in the first camera coordinate system i1, j1, d1. Therefore, the voxel coordinate generation unit 71 converts the coordinate values of the first camera coordinate system i1, j1, d1 incorporated in advance into the global coordinate system x,
By executing a process of converting into the coordinate values of y and z, the pixel coordinates (i11, j11) and the distance D1 (i11, j11) of the input first camera coordinate system i1, j1, d1 are converted into the global coordinate system x. , y, z (x11, y
11, z11). Next, the voxel coordinate generation unit 71
Determines whether or not the converted coordinates (x11, y11, z11) are included in which voxel 30 in the space 20. If the voxel 30 is included in a certain voxel 30, the voxel 30 (that is, It outputs the coordinates (vx11, vy11, vz11) of one of the voxels in which the outer surface of the object 10 is estimated to exist. On the other hand, if the converted coordinates (x11, y11, z11) are not included in any of the voxels 30 in the space 20, a predetermined value indicating that they are not included (that is, the coordinates are outside the space 20) Outputs the coordinates (xout, yout, zout) of

As described above, the first voxel coordinate generation unit 71 calculates the voxel coordinates (vx11, vy11) at which the outer surface of the object 10 is located based on the image from the first multi-view stereo camera 11. , vz11). Similarly, the second and third voxel coordinate generation units 72 and 73
Voxel coordinates (vx12, vy12, vz12) and (vx13, vy13, vz13) where the outer surface of the object 10 is estimated based on the images from the third and third multi-view stereo cameras 12, 13.
Are output.

The three voxel coordinate generators 71, 72, 7
3 repeats the above processing for all pixel coordinates (i11, j11) output from the pixel coordinate generation unit 64. As a result, all the voxel coordinates at which the outer surface of the object 10 is estimated to be located are obtained.

(5) Voxel data generation units 74, 75,
76 Three voxel data generation units 74, 75, 76 are provided corresponding to the three multi-view stereo processing units 61, 62, 63. Since the functions of the three voxel data generators 74, 75, and 76 are the same, the first voxel data generator 74 will be described as a representative.

The voxel data generator 74 sends the voxel coordinates (vx11, vy) from the corresponding voxel coordinate generator 71.
11, vz11), and if the value is not (xout, yout, zout), the data input from the multi-view stereo data storage unit 65 for the voxel coordinates (vx11, vy11, vz11) is stored. That data is, that is, its voxel coordinates (vx1
1, vy11, vz11), the distance D1 (i11, j11) of the pixel corresponding to the luminance I
It is a set of three types of values: m1 (i11, j11) and reliability Re1 (i11, j11). These three values are converted to their voxel coordinates (vx1
1, vy11, vz11) and voxel distance Vd1
(vx11, vy11, vz11), voxel luminance Vim1 (vx11, vy11, vz11)
And stored as voxel reliability Vre1 (vx11, vy11, vz11) (the set of values associated with each voxel like these is referred to as “voxel data”).

After the pixel coordinate generating section 64 has generated the coordinates (i11, j11) of all the pixels to be processed, the voxel data generating section 74 stores the voxel data accumulated for all the voxels 30,. Output. The number of voxel data stored for each voxel is not constant. For example, some voxels store a plurality of voxel data, and some voxels do not store any voxel data. A voxel in which no voxel data is stored is a voxel for which it is not estimated that the outer surface of the object 10 exists there based on a captured image from the first multi-view stereo camera 11.

As described above, the first voxel data generation unit 74 calculates the voxel data Vd1 (vx11, vy11, vz11) and Vim1 (vx11) based on the image captured from the first multi-view stereo camera 11 for all voxels. vx11, vy11, vz11), Vre
Outputs 1 (vx11, vy11, vz11). Similarly, the second and third
The voxel data generation units 75 and 76 also generate voxel data Vd2 (vx12, vy12, vz12) and Vim2 (vx12) based on the captured images from the second and third multi-view stereo cameras 12 and 13, respectively, for all voxels. , vy12, vz12), Vre
2 (vx12, vy12, vz12) and Vd3 (vx13, vy13, vz13), Vim3 (vx1
3, vy13, vz13) and Vre3 (vx13, vy13, vz13) are output.

(6) Integrated Voxel Data Generation Unit 77 The integrated voxel data generation unit 77 includes the voxel data Vd1 (vx11, vy11, vz11) and Vim1 (vx11) input from the three voxel data generation units 74, 75, and 76 described above. vx11, vy11, vz1
1), Vre1 (vx11, vy11, vz11) and Vd2 (vx12, vy12, vz12), V
im2 (vx12, vy12, vz12), Vre2 (vx12, vy12, vz12) and Vd3 (v
x13, vy13, vz13), Vim3 (vx13, vy13, vz13), Vre3 (vx13, vy
13, vz13) is accumulated and integrated for each voxel 30 to obtain an integrated luminance Vim (vx14, vy14, vz14) of each voxel.

The following is an example of the integration method.

A. In the case of a voxel in which multiple voxel data are stored, the average of the stored multiple brightnesses is integrated into the integrated brightness Vim (vx14, v
y14, vz14). In this case, a variance value of a plurality of accumulated luminances is obtained.If the variance value is equal to or more than a predetermined value, it is considered that the voxel has no data, and for example, the integrated luminance Vim (vx14, vy14, vz14 ) = 0.

Alternatively, the highest reliability is selected from the plurality of stored reliability levels, and the brightness corresponding to the highest reliability level is defined as the integrated brightness Vim (vx14, vy14, vz14). In this case, if the highest reliability is lower than a predetermined value, it is considered that the voxel has no data, and for example, the integrated luminance Vim (vx14, vy14, vz14) = 0 may be set.

Alternatively, a weighting factor is determined from the accumulated reliability, and a value obtained by multiplying the corresponding brightness by the weighting factor and averaging is set as an integrated brightness Vim (vx14, vy14, vz14).

Alternatively, it is considered that the shorter the distance between the camera and the object is, the higher the reliability of luminance is. Therefore, the shortest one of the plurality of stored distances is selected, and one corresponding to the shortest distance is selected. Integrated brightness Vim (vx14, vy14,
vz14).

Alternatively, a method obtained by modifying or combining the above methods (1) to (4).

B. In the case of a voxel in which only one voxel data is stored, the stored one brightness is used as the integrated brightness Vim (vx1
4, vy14, vz14).

Alternatively, when the reliability is equal to or more than a predetermined value, the luminance is set to the integrated luminance Vim (vx14, vy14, vz14). When the reliability is less than the predetermined value, it is determined that the voxel has no data. It is assumed that the integrated luminance Vim (vx14, vy14, vz14) = 0.

C. In the case of a voxel in which voxel data is not stored, it is assumed that the voxel has no data, and for example, the integrated luminance Vim (vx14, vy14, vz14) = 0.

The integrated voxel data generation unit 77 generates an integrated luminance Vim (vx14, vy14, vz1) of all the voxels 30,.
4) is obtained and output to the modeling unit 78.

(7) Modeling Unit 78 The modeling unit 78 receives the integrated luminance Vim (vx14, vy14, vz14) of all the voxels 30,..., 30 in the space 20 from the integrated voxel data generation unit 77. Integrated brightness Vim (vx14,
A voxel whose vy14, vz14) has a value other than “0” means a voxel in which it is estimated that the outer surface of the object 10 exists. Therefore, the modeling unit 78 sets the integrated luminance Vim
Voxel coordinates where (vx14, vy14, vz14) has a value other than "0"
Based on (vx14, vy14, vz14), 3 of the outer surface of the object 10
A three-dimensional model representing a three-dimensional shape is created. As the three-dimensional model, for example, an integrated luminance Vim (vx14, vy14, vz14)
The coordinates of voxels whose has a value other than "0" (vx14, vy14, vz14)
Is a polygon data expressing a three-dimensional shape by a large number of polygons obtained by connecting close to each other in a closed loop. The 3D model generated here is
When it is a model of the whole body of the user, it is the whole body integrated model 600 as already described with reference to FIGS. The modeling unit 78 may convert the whole-body integrated model 600 into the part joint model 601 by the processing procedure already described with reference to FIGS. 5 and 6, or may output the whole-body integrated model 600 as it is. You may.

The processing of each section described above (1) to (7) is repeated for each frame of the moving image output from the multi-view stereo cameras 11, 12, and 13. As a result, object 1
A large number of three-dimensional models are generated one after another at high speed following the motion of zero in real time or in a state close thereto.

FIG. 11 shows a second arithmetic unit 200 which can be replaced with the arithmetic unit 18 shown in FIGS. 9 and 10.
Is shown.

In the arithmetic unit 200 shown in FIG. 11, the multi-view stereo processing units 61, 62, 63, the pixel coordinate generation unit 6
4. The multi-view stereo data storage unit 65, the voxel coordinate generation units 71, 72, 73, and the modeling unit 78 have exactly the same functions as the processing units with the same reference numbers of the arithmetic unit 18 shown in FIG. Arithmetic unit 2 shown in FIG.
10 differs from the arithmetic unit 18 shown in FIG. 10 in that the target plane tilt calculators 91, 92, and 93 are added and the output of the target plane tilt calculators 91, 92, and 93 is processed. Voxel data generation units 94 and 9
5, 96 and the function of the integrated voxel data generation unit 97. Hereinafter, the difference will be described.

(1) Object plane inclination calculating sections 91, 92, and 93 Three object plane inclination calculating sections 91, 92, and 93 are provided corresponding to the three multi-view stereo processing sections 61, 62, and 63, respectively. Since the functions of these target plane inclination calculating units 91, 92, and 93 are the same as each other, the first target plane inclination calculating unit 91 will be described as a representative.

When the coordinates (i11, j11) are input from the pixel coordinate generator 64, the target plane inclination calculator 91 receives the coordinates (i11, j1).
A window having a predetermined size (for example, 3 × 3 pixels) centering on 1) is set, and the distances for all the pixels in the window are stored in the distance image in the corresponding storage area 66 of the multi-view stereo data storage unit 65. Input from D1. Next, under the assumption that the outer surface of the object 10 (hereinafter, referred to as the object surface) in the window area is a plane, the object plane inclination calculator 91 calculates the distance between all pixels in the window. Based on this, the inclination between the target plane in the window and a plane (zero inclination plane) perpendicular to the line of sight 14 from the multi-view stereo camera 11 is calculated.

As a calculation method, for example, a normal vector of the target surface is obtained by the least square method using each distance in the window, and the normal vector and the camera 11
From the vector of the line of sight 14 from
There is a method in which the i-direction component Si11 and the j-direction component Sj11 of the difference vector are extracted and used as inclinations Si11 and Sj11 of the target surface.

As described above, the first target plane inclination calculator 91 calculates the inclinations Si11 and Sj11 of the target plane viewed from the first multi-view stereo camera 11 by using all the pixels of the reference image taken by the camera 11. Is calculated and output. Similarly, 2
The third and third target plane tilt calculators 92 and 93 also calculate the tilts Si12 and Sj12 and Si13 and Sj13 of the target plane viewed from the second and third multi-view stereo cameras 12 and 13, respectively, with the respective cameras 12, The calculation and output are performed for all pixels of the reference image captured in step S13.

(2) Voxel data generators 94 and 95,
96 Three voxel data generating units 94, 95, 96 corresponding to the three multi-view stereo processing units 61, 62, 63, respectively.
Is provided. These voxel data generators 94 and 9
Since the functions of the fifth and 96 are the same, the first voxel data generator 94 will be described as a representative.

The voxel data generator 94 receives the voxel coordinates (vx11, vy11, vz11) from the corresponding voxel coordinate generator, and if the value is not (xout, yout, zout), the voxel coordinates (vx11, Voxel data for vy11, vz11) is stored. As the voxel data to be stored, the first storage area 6 in the multi-view stereo data storage unit 65 for the pixel corresponding to the voxel coordinates (vx11, vy11, vz11) is stored.
6 and the gradients Si11, Sj1 of the target surface output from the first target surface tilt calculator 91.
Vim
1 (vx11, vy11, vz11), Vsi1 (vx11, vy11, vz11), Vsj1 (vx1
1, vy11, vz11).

After the pixel coordinate generator 64 has generated the coordinates (i11, j11) of all the pixels to be processed, the voxel data generator 94 stores the voxel data Vim1 stored for all the voxels 30,. (vx11, vy11, vz11), Vsi1
(vx11, vy11, vz11) and Vsj1 (vx11, vy11, vz11) are output.

Similarly, the second and third voxel data generating units 95 and 96 also execute the processing for all the voxels 30,.
The voxel data Vim2 (vx12, vy12, vz12) and Vsi2 (vx12, vy12, vv) based on the captured images from the second and third multi-view stereo cameras 12 and 13 accumulated for 30 respectively.
z12), Vsj2 (vx12, vy12, vz12) and Vim3 (vx13, vy13, vz1
3), Vsi3 (vx13, vy13, vz13) and Vsj3 (vx13, vy13, vz13) are output, respectively.

(3) Integrated Voxel Data Generation Unit 97 The integrated voxel data generation unit 97 outputs the voxel data Vim1 (v) from the three voxel data generation units 94, 95, and 96.
x11, vy11, vz11), Vsi1 (vx11, vy11, vz11), Vsj1 (vx11, vy
11, vz11) and Vim2 (vx12, vy12, vz12), Vsi2 (vx12, vy12, v
z12), Vsj2 (vx12, vy12, vz12) and Vim3 (vx13, vy13, vz1
3), Vsi3 (vx13, vy13, vz13) and Vsj3 (vx13, vy13, vz13) are accumulated and integrated for each voxel 30, thereby obtaining an integrated luminance Vim (vx14, vy14, vz14) of each voxel.

As an integration method, there is the following method. Here, the processing is performed on the assumption that the reliability of the multi-view stereo data is higher as the inclination of the target surface is smaller.

A. In the case of a voxel in which a plurality of voxel data are accumulated, the i-direction component Vsi1 (vx11, vy11, vz1
1) and the sum of squares of the j-direction component Vsj1 (vx11, vy11, vz11),
The brightness corresponding to the gradient with the smallest sum of squares is integrated brightness.
Vim (vx14, vy14, vz14). In this case, when the value of the smallest sum of squares is larger than a predetermined value, it is determined that the voxel has no data, and for example, the integrated luminance Vim (vx14, v
(y14, vz14) = 0.

Alternatively, the average value of the accumulated i-components and the average value of the j-components of the plurality of gradients are obtained, and the i-component and j
Only the slope that falls within a predetermined range centered on the average value of the components is extracted, the brightness corresponding to the extracted slope is extracted, and the average value of the extracted brightness is integrated brightness Vim (vx14, vy14, vz1
4).

B. In the case of a voxel in which only one voxel data is stored, the stored one brightness is used as the integrated brightness Vim
(vx14, vy14, vz14). In this case, the stored 1
If the sum of squares of the i component and the j component of the number of inclinations is equal to or greater than a predetermined value, there is no data in the voxel, and for example, the integrated luminance Vim (vx14, vy14, vz14) = 0 may be set.

C. In the case of a voxel in which voxel data is not stored, it is assumed that the voxel has no data, and for example, the integrated luminance Vim (vx14, vy14, vz14) = 0.

In this way, the integrated voxel data generation unit 97 sets the integrated luminance Vim (vx14, vy14, vz1
4) is calculated and sent to the modeling unit 78. The processing of the modeling unit 78 is as described above with reference to FIG.

FIG. 12 shows a third arithmetic unit 300 which can be replaced with the arithmetic unit 18 shown in FIGS. 9 and 10.
Is shown.

Arithmetic unit 300 shown in FIG. 12 differs from arithmetic units 18 and 200 shown in FIGS. 10 and 11 in the processing procedure for creating voxel data as follows. That is, the arithmetic units 18 and 200 shown in FIGS. 10 and 11 scan the output image of the multi-view stereo processing unit, and for each pixel in the image, the corresponding voxel 30
Is found from the space 20 and voxel data is allocated. On the contrary, the arithmetic unit 300 shown in FIG. 12 scans the space 20 first, and
For each 0, corresponding stereo data is found from the output image of the multi-view stereo processing unit and assigned to each voxel.

The arithmetic unit 300 shown in FIG. 12 includes a multi-view stereo processing unit 61, 62, 63, a voxel coordinate forming unit 10
1. Pixel coordinate generation units 111, 112, 113, distance generation unit 114, multi-view stereo data storage unit 115, distance coincidence detection units 121, 122, 123, voxel data generation units 124, 125, 126, integrated voxel data generation unit 1
27 and a modeling unit 78. Among them, the multi-view stereo processing units 61, 62, 63 and the modeling unit 78
It has exactly the same function as the processing unit with the same reference number of the arithmetic unit 18 shown in FIG. 10 already described. Other functions of the processing unit are different from those of the arithmetic unit 18 shown in FIG. Hereinafter, the difference will be described. In the following description, the coordinates representing the position of each voxel 30 are (vx24, vy24, vz2
4).

(1) Voxel coordinate generation unit 101 The coordinates (v
x24, vy24, vz24) in order.

(2) Pixel coordinate generation units 111, 112, 1
13 Three pixel coordinate generation units 111, 112, 113 corresponding to the three multi-view stereo processing units 61, 62, 63, respectively.
Is provided. These pixel coordinate generation units 111, 112,
Since the functions of the 113 are the same, the first pixel coordinate generation unit 111 will be representatively described.

The first pixel coordinate generation unit 111 receives the voxel coordinates (vx24, vy24, vz24) and inputs the corresponding pixel coordinates (i2) of the output image of the first multi-view stereo processing unit 61.
1, j21) is output. Note that voxel coordinates (vx24, vy24, vz2
The relationship between 4) and the pixel coordinates (i21, j21) may be calculated each time using the mounting position information and the lens distortion information of the multi-view stereo camera 11, or may be calculated in advance for all voxel coordinates (vx24 , vy24, vz24) at the pixel coordinates (i21, j
21) may be calculated and stored in the form of a look-up table or the like, and may be called from the storage.

Similarly, the second and third pixel coordinate generators 112 and 113 also perform the second and third multi-view stereo systems 62 and 6 corresponding to the voxel coordinates (vx24, vy24, vz24).
Output the coordinates (i22, j22) and (i23, j23) of the output image of No. 3 respectively.

(4) Distance Generation Unit 114 The distance generation unit 114 inputs the voxel coordinates (vx24, vy24, vz24), and the corresponding voxel and the first, second, and third multi-view stereo cameras 11, 12. , 13 are output as distances Dvc21, Dvc22, and Dvc23. In addition, each distance Dvc21, Dvc22, Dvc23 is each multi-view stereo camera 11,
The calculation is performed using the mounting position information and lens distortion information of 12, 13.

(5) Multiview Stereo Data Storage Unit 115 The multiview stereo data storage unit 115 has storage areas 116 corresponding to the three multiview stereo processing units 61, 62, and 63,
117, 118, and three multi-view stereo processing units 6
From 1, 62 and 63, the image after the stereo processing (the luminance image Im
1, Im2, Im3, range image D1, D2, D3, reliability image Re1, Re2, Re
3), and store these input images in the corresponding storage area 1
16, 117 and 118. For example, the luminance image Im1 and the distance image D1 from the first multi-view stereo processing unit 61
The reliability image Re1 is stored in the first storage area 116.

Subsequently, the multi-view stereo data storage unit 115
Inputs pixel coordinates (i21, j21), (i22, j22), and (i23, j23) from three pixel coordinate generation units 111, 112, and 113,
Pixels corresponding to the input pixel coordinates (i21, j21), (i22, j22), and (i23, j23) from the storage areas 116, 117, and 118 respectively corresponding to the three pixel coordinate generation units 111, 112, and 113. And outputs the stereo data (luminance, distance, reliability). For example, for the pixel coordinates (i21, j21) input from the first pixel coordinate generation unit 111, the accumulated luminance image Im1, distance image D1, and reliability image of the first multi-view stereo processing unit 61 are stored. From Re1, the luminance Im1 (i21, j) of the pixel corresponding to the input pixel coordinates (i21, j21)
21), the distance D1 (i21, j21) and the reliability Re1 (i21, j21) are read and output.

The input pixel coordinates (i21, j21), (i2
(2, j22) and (i23, j23) are real number data obtained by calculation from voxel coordinates, whereas pixel coordinates of an image stored in the multi-view stereo data storage unit 115 (that is, memory addresses ) Is an integer. Therefore, the multi-view stereo data storage unit 115 stores the input pixel coordinates (i21, j2
(1), (i22, j22), (i23, j23) is converted to integer pixel coordinates by rounding down the decimal point, or input pixel coordinates (i21, j21), (i22, j22), (i23, j23) A plurality of integer pixel coordinates near each may be selected, stereo data of the plurality of integer pixel coordinates may be read and interpolated, and the interpolation result may be output as stereo data for the input pixel coordinates.

(6) Distance coincidence detecting sections 121, 122, 1
23 three distance match detection units 121, 122, 123 corresponding to the three multi-view stereo processing units 61, 62, 63, respectively.
Is provided. These distance match detection units 121, 122,
Since the functions of 123 are the same as each other, the first distance match detection unit 121 will be described as a representative.

The first distance match detecting section 121 outputs the distance D1 (i21, j21) measured by the first multi-view stereo processing section 61 output from the multi-view stereo data storage section 115.
And the voxel coordinates (vx2
4, vy24, vz24) is compared with the distance Dvc1. If the outer surface of the object 10 is in the voxel, D1
(i21, j21) should match Dvc21. Therefore, when the absolute value of the difference between D1 (i21, j21) and Dvc21 is equal to or smaller than a predetermined value, the distance match detection unit 121 determines that the outer surface of the object 10 exists in the voxel and determines the determination value. Ma21 = 1 is output, and when the absolute value of the difference between D1 (i21, j21) and Dvc21 is larger than a predetermined value, it is determined that the outer surface of the object 10 does not exist in the voxel, and the determination value Ma21 = Outputs 0.

Similarly, the second and third distance coincidence detecting sections 122 and 123 measure the distances D2 (i22, j22) and D3 (i23, j23) measured by the second and third multi-view stereo processing sections 62 and 63, respectively. )
, And determines whether or not the outer surface of the object 10 exists in the voxel, and outputs determination values Ma22 and Ma23, respectively.

(7) Voxel data generators 124 and 12
5, 126 Three voxel data generators 124, 125, corresponding to the three multi-view stereo processors 61, 62, 63, respectively.
126 are provided. These voxel data generators 12
Since the functions of 4, 125 and 126 are the same as each other, 1
The voxel data generation unit 124 will be described as a representative.

The first voxel data generation unit 124
The judgment value Ma21 from the first distance match detection unit is checked, and if Ma21 is 1, the voxel coordinates (vx24, vy2
4, vz24), when the outer surface of the object 10 is present in the voxel), the data output from the first storage area 116 of the multi-view stereo storage unit 115 for the voxel is used as the voxel data of the voxel. Accumulate as The voxel data to be stored is the voxel coordinates (vx24, vy2
4, vz24) at the pixel coordinates (i21, j21) at the luminance Im1 (i21, j2).
1) and the reliability Re1 (i21, j21), and the voxel luminance Vim1 (vx24, vy24, vz24) and the voxel reliability Vre1 (v
x24, vy24, vz24).

After the voxel coordinate generating section 101 has generated the voxel coordinates of all the voxels 30,..., 30 to be processed, the voxel data generating section 124 generates all the voxel 3
Voxel data accumulated for each of 0, ..., 30
Vim1 (vx24, vy24, vz24) and Vre1 (vx24, vy24, vz24) are output. The number of voxel data stored for each voxel is not the same, and some voxels do not store voxel data.

Similarly, the second and third voxel data generators 125 and 126 also output all voxels 30,.
0 and the second and third stereo processing units 6
Voxel data Vim2 (v
x24, vy24, vz24), Vre2 (vx24, vy24, vz24) and Vim3 (vx2
4, vy24, vz24) and Vre3 (vx24, vy24, vz24) are accumulated and output.

(8) Integrated Voxel Data Generation Unit 127 The integrated voxel data generation unit 127 integrates the voxel data from the three voxel data generation units 124, 125, and 126 for each voxel, thereby obtaining the integrated luminance of each voxel. Find Vim (vx24, vy24, vz24).

The following integration methods are available.

A. In the case of a voxel in which multiple voxel data are stored, the average of the stored multiple brightnesses is integrated into the integrated brightness Vim (vx24, v
y24, vz24). In this case, a variance value of a plurality of luminances is obtained, and if the variance value is equal to or greater than a predetermined value, it is determined that there is no data in the voxel, and for example, Vim (vx24,
4, vz24) = 0.

Alternatively, the highest one of the plurality of accumulated reliability values is selected, and the luminance corresponding to the highest reliability is set as the integrated luminance Vim (vx24, vy24, vz24). In this case, if the highest reliability is equal to or less than a predetermined value, it is determined that the voxel has no data, for example, Vim (vx24, vy24,
vz24) = 0.

Alternatively, a weight coefficient is determined from the accumulated reliability, and a value obtained by multiplying the accumulated luminance by a corresponding weight coefficient and averaging the integrated luminance Vim (vx24, vy2
4, vz24).

B. In the case of a voxel in which only one voxel data is stored, the luminance is set as an integrated luminance Vim (vx24, vy24, vz24).
In that case, if the reliability is equal to or less than a predetermined value, it is assumed that the voxel has no data, and for example, Vim (vx24, vy24, vz24) =
It may be set to 0.

C. In the case of a voxel that does not store voxel data, it is assumed that the voxel has no data.
m (vx24, vy24, vz24) = 0.

As described above, the integrated voxel data generation unit 127 outputs the integrated luminance Vim (vx24, vy24, vz
24) is calculated and sent to the modeling unit 78. The processing of the modeling unit 78 is as described above with reference to FIG.

By the way, the arithmetic unit 18 in FIG.
Similar to the difference from the arithmetic device 200, the arithmetic device 300 of FIG. 12 also includes a target surface inclination calculator, and uses the target surface inclination instead of the reliability when generating the integrated luminance. It is also possible.

FIG. 13 shows a fourth arithmetic unit 400 which can be replaced with the arithmetic unit 18 shown in FIGS. 9 and 10.
Is shown.

The arithmetic device 400 shown in FIG. 13 combines the configuration of the arithmetic device 18 shown in FIG. 10 with the configuration of the arithmetic device 300 shown in FIG. Is to be filled.
That is, according to the configuration of the arithmetic unit 300 in FIG. 12, since the processing is performed by changing the coordinates of the three axes of the voxel coordinates (vx24, vy24, vz24), the voxel size is reduced in order to create a fine three-dimensional model. Therefore, when the number of voxels is increased, the amount of calculation becomes enormous. On the other hand, according to the configuration of the arithmetic unit 18 in FIG. 10, the coordinates of the two axes of the pixel coordinates (i11, j11) need only be changed, so that the amount of calculation is smaller than that of the arithmetic unit 300 in FIG. Even if the number of voxels is increased in order to obtain a three-dimensional model, since the number of voxels to which voxel data is given is limited by the number of pixels, there is a gap between voxels to which voxel data is given, and a fine three-dimensional model There is a problem that can not be obtained.

Therefore, in order to solve the above problem, the arithmetic unit 400 shown in FIG. 13 first sets a small number of coarse voxels and performs the same pixel-oriented arithmetic processing as the arithmetic unit 18 in FIG. Then, an integrated luminance Vim11 (vx15, vy15, vz15) for the coarse voxel is obtained. Next, based on the integrated luminance Vim11 (vx15, vy15, vz15) of the coarse voxel, regarding the coarse voxel having the integrated luminance in which the outer surface of the object 10 is determined to be present, the area of the coarse voxel is reduced to a smaller area. Are divided into fine voxels having the symbol v, and only the divided fine voxels shown in FIG.
A voxel-oriented operation such as 0 is performed.

That is, the arithmetic unit 400 shown in FIG.
Is a multi-view stereo processing unit 6 having the same configuration as that already described.
Downstream of 1, 62, and 63, there are provided a pixel coordinate generation unit 131, a pixel-oriented operation unit 132, a voxel coordinate generation unit 133, a voxel-oriented operation unit 134, and a modeling unit 78 having the same configuration as that already described.

The pixel coordinate generation section 131 and the pixel-oriented operation section 132 are composed of the portion 79 (pixel coordinate generation section 64, multi-view stereo data storage section 65, voxel coordinate generation sections 71 and 72) of the arithmetic unit 18 shown in FIG. , 73, voxel data generators 74, 75, 76 and integrated voxel data generator 7
The configuration is substantially the same as 7). That is, the pixel coordinate generation unit 131 scans all pixels of the output image of the multi-view stereo processing units 61, 62, and 63 or all pixels of a partial region to be processed, similarly to the pixel coordinate generation unit 64 illustrated in FIG. Then, the coordinates (i15, j15) of each pixel are sequentially output. The pixel-oriented operation unit 132 calculates the coordinates (vx1) of the large voxel which is set by roughly dividing the space 20 based on each pixel coordinate (i15, j15) and the distance to the pixel coordinate (i15, j15).
5, vy15, vz15) and its coarse voxel coordinates (vx15, vy1
5, the integrated luminance Vim11 (vx15, vy15, vz15) for FIG.
It is obtained and output in the same manner as the arithmetic unit 18 of 0. Note that the method of obtaining the integrated luminance Vim11 (vx15, vy15, vz15) here is different from the method described above in that Vim11 (vx15
15, vy15, vz15) may be used as a simple method of distinguishing whether or not the coarse voxel has the outer surface of the object 10 or not.

The voxel coordinate generation unit 133 generates the integrated luminance Vim11 (vx15, vy15, vx15, vy15, vz15) of each coarse voxel coordinate (vx15, vy15, vz15).
vz15) and input its integrated brightness Vim11 (vx15, vy15, vz15)
Is not zero (that is, it is estimated that the outer surface of the object 10 exists), the coarse voxel is divided into a plurality of fine voxels, and the voxel coordinates (vx16, vy16, vz16) of each fine voxel are divided. Are sequentially output.

The voxel-oriented operation unit 134 is the same as the operation unit 300 shown in FIG.
112, 113, distance generation unit 114, multi-view stereo data storage unit 115, distance coincidence detection units 121, 122, 12
3, has substantially the same configuration as the voxel data generators 124, 125, 126 and the integrated voxel data generator 127). The voxel-oriented operation unit 134 obtains voxel data for each fine voxel coordinates (vx16, vy16, vz16) based on the output images from the multi-view stereo processing units 61, 62, 63, and integrates and integrates them. Brightness Vim12
(vx16, vy16, vz16) is calculated and output.

Since the processing for generating fine voxel data by the voxel-oriented operation unit 134 is performed only for voxels estimated to have the outer surface of the object 10, the outer surface of the object 10 does not exist. Useless processing for voxels is omitted, and processing time is reduced accordingly.

In the above configuration, each of the pixel-oriented operation unit 132 and the voxel-oriented operation unit 134 has a multi-view stereo data storage unit.
One multi-view stereo data storage unit is a pixel-oriented operation unit 1
32 and the voxel-oriented operation unit 134 may be configured to be shared.

By the way, the individual elements constituting the arithmetic units 18, 200, 300, and 400 shown in FIGS. 10 to 13 described above can be realized by either a pure hardware circuit or a computer program executed by a computer. Further, it can also be realized in a form in which both are combined. If it is realized by a pure hardware circuit, the modeling is completed very quickly.

FIG. 14 shows the overall configuration of the second embodiment of the virtual fitting system according to the present invention.

The virtual fitting system shown in FIG.
For example, it is suitable for performing virtual fitting in a store such as a department store, a clothing store, or a game center. That is, a stereo imaging system 6 having the same configuration as that shown in FIG. 9 is provided in a store, and the arithmetic unit 18 (or the arithmetic unit shown in FIGS. 11 to 13) having the configuration shown in FIG. Devices 200, 300 or 400) are connected. The arithmetic unit 18 includes a computer system 19
Is connected. The computer system 19 has the virtual fitting program as described above, and stores the costume database 5 storing three-dimensional models of various costumes.
2 having a controller 51 that can be operated by the user 10 who is in the stereo shooting system 6,
The display screen 50 displays the stereo shooting system 6.
It is arranged at a position where the user 10 inside can be seen.

[0158] The arithmetic unit 18 is the stereo imaging system 6
, A user's real body data is input, a standard whole body model of the user is created by the method already described, and the standard whole body model is output to the computer system 50. The arithmetic unit 18 can also output the standard whole-body model to the outside (for example, write it on a recording medium such as a CD-ROM or send it out to a communication network). The computer system 19 executes the virtual fitting program using the user's standard whole body model input from the arithmetic unit 18 and the three-dimensional models of various costumes in the costume database 52, and displays the virtual screen on the display screen 50 thereof. A virtual fitting window 500 as shown in FIG. 4 is displayed.

By operating the controller 51, the user 10 can select a costume to be worn, and change the position of the virtual camera 40 in the virtual three-dimensional space, the line of sight 41, and the zoom magnification. Further, as described above, the arithmetic unit 18 creates a number of standard whole-body models that follow the movement of the user 10 in real time or in a state close thereto in a similar manner, and sends them to the computer system 19 one after another. If the user 10 freely poses and performs motion in the stereo shooting system 6, the user's three-dimensional model in the virtual fitting window 500 displayed on the display screen 50 can also perform the same pose and perform motion. become.

FIG. 15 shows the overall configuration of an embodiment of the game system according to the present invention. This game system is
This is for a user to take in three-dimensional model data of an arbitrary article into a virtual three-dimensional space of a computer game and play.

As shown in FIG. 15, the modeling server 7
01, a computer system of a game supplier such as a game maker or a game store (hereinafter, referred to as a “game supplier system”) 702, and a computer system of a user (for example, a personal computer or a game computer, hereinafter referred to as a “user system”) ) Are communicably connected through a communication network 703 such as the Internet. The user system 704 includes at least one multi-view stereo camera 705, a controller 706 operated by a user,
And a display device 707. The user system 704 includes a game program and a stereo shooting program.

FIG. 16 shows a processing flow of the game system. The operation of the game system will be described with reference to FIGS.

(1) As shown in step S81 of FIG. 16, the user system 704 first executes a stereo photography program. Therefore, the user uses the multi-view stereo camera 704 to capture a still image of an article 709 desired to be used in the game program (for example, his / her car toy used in a car racing game).
Using multiple directions (e.g., front, back, left, right, up, down,
(Diagonally up front, back up, etc.). At this time, the stereo shooting program is displayed on the display device 707 by, for example, a shooting window 71 as shown in FIG.
Displays 0. An example window 711 showing an example when shooting from each direction is shown in the shooting window 710.
And a real image data window 712 showing the result of the user's actual shooting, and a monitor window 713 showing the video image currently output from the multi-view stereo camera 705.
, A "shutter" button 714, a "cancel" button 715, and the like. The user displays the image displayed on the monitor window 713 in the example window 7.
After adjusting the positional relationship between the multi-lens stereo camera 705 and the article 709 so as to be in the same direction as the image to be taken in 11, the user presses the “shutter” button 714 to stop the article 709 in that direction. The picture is taken.

(2) When photographing from all directions is completed, the stereo photographing program of the user system 704 is connected to the modeling server 701 via the communication network 703 and the article 709 as shown in step S82 of FIG. Is transmitted to the modeling server 701, and the modeling server 701 receives the photographed data (S92).
At this time, the stereo shooting program of the user system 704 also notifies the modeling server 701 of the identification information (game ID) of the game program that the user intends to use.

(3) As shown in steps S101 and S91, the modeling server 701 previously expressed the data format of the three-dimensional model used in each of the various game programs from the game supplier system 702. Information is received and stored. Then, upon receiving the real image data and the game ID of the article 709 from the user system 704, the modeling server 701 uses the received real image data in the data format of the received game ID using the received real image data as shown in step S93. Is created. The method of creating three-dimensional model data is basically the same as the method described with reference to FIGS. The modeling server 701 transmits the created three-dimensional model data of the article to the user system 704, and the stereo shooting program of the user system 704 receives the three-dimensional model data as shown in step S83.

(4) As shown in step S83, the stereo shooting program of the user system 704 uses the received three-dimensional model data to generate a two-dimensional image of the three-dimensional model viewed from various directions (for example, Part 3
The moving image viewed from all directions while rotating the dimensional model is rendered and displayed on the display device 707. The user looks at it and checks whether there is any problem in the received three-dimensional model data. When it is confirmed that there is no problem, the stereoscopic imaging program saves the received three-dimensional model data, and notifies the modeling server 701 of the reception.

(5) As shown in steps S94 and S95, when the modeling server 701 confirms that the user has received the three-dimensional model data, the modeling server 701 performs a billing process for collecting a fee from the user, and executes a billing process. Is transmitted to the user system 704 in step S
As shown at 85, the stereo photography program of the user system 704 receives and displays the billing result data.

(6) When the creation of the three-dimensional model data of the article 709 is completed in this way, then, in step S86
As shown by, the user executes a game program on the user system 704, and in the game program,
The three-dimensional model data of the article 709 saved earlier is used. For example, as shown in the display device 707 of FIG. 15, the user can play a car racing game using the three-dimensional model 708 of the car toy 709 of the user.

FIG. 18 shows a second embodiment of the game system according to the present invention. This game system is for a user to take three-dimensional model data of a human body such as himself or a friend into a virtual three-dimensional space of a computer game and play.

As shown in FIG. 18, the modeling server 7
21, a game supplier system 722, a user system 724, and a store system 729 are communicably connected via a communication network 723. A stereo imaging system 730 is connected to the store system 729.

The game supplier system 722 is similar to the game supplier system 722 shown in FIG.
As with the game supplier system 702 of the game system shown in FIG. 5, the format server 721 provides format information of three-dimensional models for various game programs to the modeling server 721. Store system 729 and stereo photography system 7
The camera 30 shoots the user's body with a plurality of multi-view stereo cameras and sends the actual shot data to the modeling server 721, as in the store system 5 and the stereo shooting system 6 of the virtual fitting system shown in FIG.

The user system 724 is, for example, a personal computer or a game computer, and includes a controller 727 operated by a user and a display device 7.
Games with humans (for example,
Fighting game). Further, the user system 724 includes a plurality (for example, two) of multi-view stereo cameras 725 and 726 if the user desires.
And the multi-view stereo cameras 725, 7
It is possible to install a stereo shooting program for sending the real photograph data from the modeling server 26 to the modeling server 721 and receiving the three-dimensional model data from the modeling server 721.

FIG. 19 shows a processing flow of the game system. The operation of the game system will be described with reference to FIGS.

(1) First, the user performs stereo photography of the body of a person who wants to appear in the game, for example, himself.
This is a stereo shooting system 730 installed in a store, as in the case of the virtual fitting system described above.
, But that has already been explained. Therefore, here, a case will be described as an example where the user performs photographing using the multi-view stereo cameras 725 and 726 connected to the user system 724 of the user. As shown in step S111 of FIG.
At 24, a stereo photographing program is executed, and a plurality of multi-view stereo cameras 725 and 726 arranged so as to photograph the person from different places photograph their own body. At that time, the user may select various predetermined motions used in the game (for example, punch, kick,
Various movements such as throwing, receiving, and dodging) are performed in front of the multi-view stereo cameras 725 and 726, and the moving image data captured by the multi-view stereo cameras 725 and 726 for each motion is received by the user system 724. Then, as shown in step S112, the real system data (moving image data) of each motion and the game ID of the game program to be used are transmitted from the user system 724 to the modeling server 721.

(2) As shown in steps S121 and S122, when the modeling server 721 receives the real image data of each motion of the user and the game ID, the modeling server 721 converts each frame of the real image data (moving image data) for each motion. In the processing method described with reference to FIGS. 10 to 13, three-dimensional model data of the user's body is created in the data format of the game ID, and a plurality of 3D models created from a series of many frames of the moving image of each motion. The dimensional model data is arranged continuously according to the frame order. As a result, a series of a large number of three-dimensional model data constituting each motion is formed. Then, as shown in step S123, the modeling server 721
A series of three-dimensional model data that constitutes each motion
Send to user system 724.

(3) As shown in steps S113 and S114, upon receiving a series of three-dimensional model data of each motion, the stereo shooting program of the user system 724 uses the series of three-dimensional models of each motion. Then, a plurality of animation images, each of which shows the user performing each motion from different viewpoints, are created, and the animation images are sequentially displayed on the display device 728. The user checks these animated images for any problem with the received series of three-dimensional model data of each motion.
When it is confirmed that there is no problem, the stereo shooting program saves the received series of three-dimensional model data of each motion, and notifies the modeling server 721 of the receipt.

(4) As shown in steps S124 and S125, when the modeling server 721 confirms that the user has received the three-dimensional model data, the modeling server 721 performs a billing process for collecting a fee from the user, and executes a billing process. Is transmitted to the user system 724, and as shown in step S115, the stereo photography program of the user system 724 receives and displays the charging result data.

(5) Thereafter, as shown in step S116, the user executes the game program in the user system 724, and uses the series of three-dimensional model data of each motion stored earlier in the game program. For example, in response to the user's operation of the controller 727, the user's three-dimensional image is displayed in the virtual three-dimensional space of the fighting game as shown in the display device 728 of FIG.
The dimensional model 731 performs motions of various techniques such as a straight punch, an upper cut, and a turning kick.

In the above description, each motion is constituted by a series of a large number of three-dimensional models. Instead, a three-dimensional model (FIG. 6) in which each part of the body is jointed by joints is used. Part joint model) 601
And motion data for moving the part joint model 601 in the same manner as the motion of the user. When the part joint model 601 and the motion data are used, the modeling server 721 creates the part joint model 601 by performing the processing described with reference to FIGS. The rotational accuracy of each part at each joint and the location of the part joint model 601 for determining the part joint model 601 in the same posture as the three-dimensional model created from each frame of the image are determined and used as motion data. The part joint model 601 and the motion data are transmitted to the user system 724.

FIG. 20 shows the overall configuration of the third embodiment of the game system according to the present invention. In this game system, a user takes in a virtual three-dimensional space of a game a three-dimensional model of his / her body, which operates in real time in exactly the same way as his / her own, and plays a highly realistic game. You can experience.

As shown in FIG. 20, a plurality of multi-view stereo cameras 741 to 74 are provided at different places around a predetermined space into which a user 748 should enter so that the space can be photographed.
3 are arranged. These multi-view stereo cameras 74
1 to 743 are arithmetic units 74 for three-dimensional modeling
The output of the arithmetic unit 744 is connected to a game device (for example, a personal computer, a home game computer, or a business game computer installed in a game center or the like) 745. The game device 745 has a display device 746, and the user 748 can see the screen of the display device 746. In short, this game system has
14 has a configuration substantially similar to that of the virtual fitting system shown in FIG. 4, and allows a computer system 19 shown in FIG. 14 to execute a game.

The arithmetic unit 744 operates in the same manner as described with reference to FIGS.
3 from the photographed image data (moving image data)
A series of three-dimensional model data that moves in the same manner following the motion of No. 8 in real time is created at high speed and sent to the game device 745. The game device 745 takes the series of three-dimensional model data into the virtual three-dimensional space of the game, and displays on the display device 746 a three-dimensional model 747 that moves in exactly the same manner as the actual user 748. Thereby, the user 74
8 can play the game with a reality as if he / she entered the game world.

While some embodiments of the present invention have been described above, these embodiments are merely examples for describing the present invention, and are not intended to limit the present invention to only these embodiments. Therefore, the present invention can be implemented in various modes other than the above-described embodiment. For example, the present invention can be applied not only to virtual fitting and games, but also to various uses in which a three-dimensional model can be used.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an overall configuration of a virtual fitting system according to an embodiment of the present invention.

FIG. 2 is a flowchart showing a part of the processing procedure of the virtual fitting system performed mainly by the modeling server 1;

FIG. 3 is a flowchart showing a part of the processing procedure of the virtual fitting system performed mainly by the virtual fitting server 3.

FIG. 4 is a view showing an example of a virtual fitting window displayed by a virtual fitting program on a display screen of a user system.

FIG. 5 is a flowchart showing a processing flow when a modeling server creates a standard whole body model having joints.

FIG. 6 is a view showing the configuration of a three-dimensional body model created in the course of the processing flow of FIG. 5;

FIG. 7 is a flowchart showing a processing flow of a virtual fitting program using a standard whole body model having joints.

FIG. 8 is a diagram illustrating an operation performed on the standard full-body model and the three-dimensional model of the costume performed in the process of the processing flow of FIG.

FIG. 9 is a perspective view showing a schematic overall configuration of a stereo imaging system.

FIG. 10 is a block diagram showing the internal configuration of the arithmetic unit 18.

FIG. 11 is a block diagram showing an internal configuration of a second arithmetic device 200 that can be replaced with the arithmetic device 18 shown in FIGS. 9 and 10;

FIG. 12 is a block diagram showing an internal configuration of a third arithmetic device 300 which can be replaced with the arithmetic device 18 shown in FIGS. 9 and 10;

FIG. 13 is a block diagram showing an internal configuration of a fourth arithmetic unit 400 which can be replaced with the arithmetic unit 18 shown in FIGS. 9 and 10;

FIG. 14 shows a second embodiment of the virtual fitting system according to the present invention.
FIG. 2 is a perspective view showing the overall configuration of the embodiment.

FIG. 15 is a block diagram showing an overall configuration of an embodiment of a game system according to the present invention.

FIG. 16 is a flowchart of processing of the game system in FIG. 15;

FIG. 17 is a diagram showing a shooting window.

FIG. 18 is a block diagram showing an overall configuration of a second embodiment of the game system according to the present invention.

FIG. 19 is a flowchart of processing of the game system in FIG. 18;

FIG. 20 is a perspective view showing an overall configuration of a third embodiment of the game system according to the present invention.

[Explanation of symbols]

1, 701, 721 Modeling server 2 Costume supplier system 3 Virtual fitting server 4, 704, 724 User system 5, 729 Store system 6, 730 Stereo shooting system 6B, 11, 12, 13, 705, 725, 726, 7
41, 742, 743 Multi-view stereo camera 8, 703, 723 Communication network 18, 744 Arithmetic unit 19 Computer system 500 Virtual fitting window 600 Whole body integrated model 601 Partial joint model 620 3D model of costume 709 Goods to be used in game 710 Stereo Shooting window 722 Game supplier system 745 Game device

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G06T 17/40 G06T 17/40 D 5C054 H04N 5/262 H04N 5/262 5C061 // H04N 7/18 7 / 18Z F term (reference) 2C001 BA00 BA04 BC00 BC05 CA00 CA08 CB01 CC02 5B046 AA10 CA06 EA09 FA18 GA01 HA05 JA04 KA06 5B050 AA10 BA08 BA09 BA12 BA13 CA07 CA08 DA02 EA24 EA28 FA02 FA13 FA19 5B057 AA18 BA02 DB03 DC08 DC22 DC32 5C023 AA11 AA37 AA38 CA03 DA08 5C054 AA02 CC00 DA09 EH00 FA04 FD01 FE12 GB01 GB16 HA14 HA15 5C061 AB04 AB08 AB10 AB14 AB21

Claims (8)

[Claims] 1. A system for allowing a real object to appear in a virtual three-dimensional space of a computer application used by a user, wherein the system is capable of communicating with a stereo imaging device available to the user, A real image data receiving means for receiving real image data obtained by stereo shooting of a real object from the stereo image capturing apparatus; and a predetermined data format which the computer application can take in a virtual three-dimensional space based on the received real image data. In the object 3
A system comprising: modeling means for creating a dimensional model; and three-dimensional model output means for outputting the three-dimensional model data of the object in a manner that can be provided to the user or a computer application used by the user. 2. The photographing data received from the stereo photographing device includes photographing data of a plurality of poses taken when the real object takes different poses. The system according to claim 1, wherein the three-dimensional model data of the object is created based on the real shot data of the pose so that the three-dimensional model data can take different poses or perform a motion. 3. The actual photographing data received from the stereo photographing device includes actual photographing data of a moving image photographed when the real object performs a certain motion. The system according to claim 1, wherein the three-dimensional model data of the object is created based on real image data of the image so as to perform the same motion as performed by the object. 4. The modeling means creates the three-dimensional model data so as to follow the motion performed by the real object substantially in real time during shooting by the stereo shooting device and perform the same motion. 4. The system of claim 3, wherein 5. A method for allowing a real object to appear in a virtual three-dimensional space of a computer application used by a user, the method comprising: converting a real object from a stereo image capturing device available to the user to a stereo image. Receiving the photographed actual photographed data; and, based on the received photographed actual data, the object application in a predetermined data format that the computer application can take in the virtual three-dimensional space.
Creating a three-dimensional model; and outputting the three-dimensional model data of the object in a manner that can be provided to the user or a computer application used by the user. 6. A system for allowing a real object to appear in a virtual three-dimensional space of a computer application used by a user, comprising: a stereo photographing device usable by the user; A modeling device that is communicable and can also communicate with a computer device available to the user, wherein the modeling device receives, from the stereo shooting device, shooting data of a real object in stereo. Receiving means, based on the received actual photographed data, in a predetermined data format which the computer application can capture in the virtual three-dimensional space;
A system comprising: modeling means for creating a three-dimensional model; and three-dimensional model transmitting means for transmitting three-dimensional model data of the object to a computer device usable by the user. 7. A system for allowing a real object to appear in a virtual three-dimensional space of a computer application used by a user, comprising: a computer device for the user to execute the computer application; A stereo photographing device that can be used by a user, and a modeling device that can communicate with the stereo photographing device and the computer device. The modeling device, from the stereo photographing device, captures actual photographed data of a real object in stereo. Means for receiving actual photograph data; and receiving the actual photograph data in a predetermined data format which the computer application can take in the virtual three-dimensional space.
A system comprising: modeling means for creating a three-dimensional model; and three-dimensional model transmitting means for transmitting three-dimensional model data of the object to the computer device. 8. A method for allowing a real object to appear in a virtual three-dimensional space of a computer application used by a user, comprising: a step of taking a stereo image of the real object; Based on the real image data of the object, the computer application is a virtual 3
Creating a three-dimensional model of the object in a predetermined data format that can be captured in a three-dimensional space; and inputting the three-dimensional model data of the object to a computer device capable of executing the computer application. Method.