JP2011036483A - Image generation system, control program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image generation system for obtaining an image to acquire information useful for improving skill in performance of a user, eliminating the need of attaching many sensors or markers onto the body of the user in order to generate the image, and obtaining detailed information on the user such as the expression of the user through the use of the image. <P>SOLUTION: The image generation system includes: an HMD for displaying an opposite player as a virtual object to the user; and a video camera for imaging the user conducting performance with respect to the opposite player. The system obtains a position on a virtual space corresponding to the position of the video camera in a reality space (step S104), generates the CG image of the opposite player viewed from the obtained position (step S109), and then, combines the image of the user imaged by the video camera 3 with the CG image of the opposite player (step S112). <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image generation system used for sports practice, a control program for the image generation system, and a recording medium.

  In recent years, a virtual sports system has been developed in which a virtual object placed in a virtual space and a user perform sports interactively.

  For example, Non-Patent Document 1 discloses a virtual sports system for performing batting practice. In this virtual sports system, a pitcher and a ball are expressed by CG (Computer Graphics), and the user visually observes the movement of the pitcher and swings the bat. The timing and speed at which the user swings the bat is measured by the photosensor, and the user's batting is evaluated based on the measured value.

  Non-Patent Document 2 discloses a virtual sports system that performs boxing with an opponent player represented by CG. In this virtual sports system, a haptic device is used to capture user motion data. The avatar generated based on the motion data performs boxing with the opponent player in the virtual space.

  Patent Document 1 discloses a virtual sports system for playing tennis. In this virtual sports system, a ball and an opponent player are represented by CG, and a user visually observes the movement of the ball and moves a racket to which a marker made of a light emitting diode is attached. Then, the marker is photographed by the camera, and based on the photographed image, it is determined whether the swing performed by the user is a forehand swing or a backhand swing.

JP 2005-218757 A

Proceedings of ACM Symposium on Applied Computing 2002 pp.1060-1065 ACM Computers in Entertainment, vol.4, No.3, Article No.9 (2006)

  By the way, those who work on sports have a desire to “view their performance objectively”. In an actual sport, the user can confirm what performance he / she performed according to the movement of the opponent player or the like by the photographed image of the camera. As a result, the skill of the user is improved. In other words, in motor learning, one's own skills are learned from attempts to correct or perceive one's own movement. At this time, when the practitioner obtains information on the exercise result, it is called feedback.

  However, in the virtual sports system, opponent players, balls and the like do not actually exist in the three-dimensional space. That is, the user performs a performance by visually observing the CG image to image a three-dimensional positional relationship with the opponent player and the like. Even if the images are taken together, the image obtained by the photography does not recognize the three-dimensional positional relationship between the opponent player and the user. For this reason, even if the user confirms the captured image, the user cannot obtain information useful for skill improvement.

  In the virtual sports system disclosed in Non-Patent Document 2, it is necessary to attach many sensors to the user's body in order to obtain the user's motion data. For this reason, the user cannot move as expected, and there is a possibility that the original merit of sports is significantly impaired. In addition, since it is impossible to model detailed information such as the user's facial expression by the above-described capturing, the user cannot obtain detailed information about the user from the CG image.

  The present invention has been made in view of such a situation, and an image from which information useful for improving the performance skill of the user can be obtained, and in order to generate the image, many sensors and sensing are provided on the user's body. It is another object of the present invention to provide an image generation system, a control program, and a recording medium in which detailed information about the user such as the user's facial expression can be obtained from the image without attaching an auxiliary instrument. That is, the present invention aims to effectively perform feedback of performance knowledge obtained from a video taken by a practicing exerciser, and is expressed by CG using a mixed reality (MR) technique. Providing a system for visualizing and practicing the exercise performance of the practitioner in the interaction with the other player, ball, environment, etc., including the other player expressed by the above CG, etc. It is.

  In order to achieve the above object, an image generation system according to a first aspect of the present invention includes a display device that displays a virtual object to a user, and a photographing device that photographs a user who performs a performance on the virtual object. An image generation system comprising: a shooting position acquisition unit that acquires a position in a virtual space corresponding to a position of the shooting device in real space; and a virtual object viewed from the position acquired by the shooting position acquisition unit The image processing apparatus includes: virtual image generation means for generating an image; and composite image generation means for combining the image captured by the imaging device and the image of the virtual object generated by the virtual image generation means.

  Preferably, the camera further comprises viewpoint position acquisition means for acquiring a position on the virtual space corresponding to the viewpoint position of the user in the real space, and the virtual image generation means is a virtual object viewed from the position acquired by the viewpoint position acquisition means An image of an object is generated and displayed on the display device.

  Preferably, the apparatus further comprises a motion recognizing unit that acquires information related to a user's motion based on the user's image captured by the image capturing device, and the virtual image generating unit is configured based on the information acquired by the motion recognizing unit. The movement of the object is set.

  Preferably, further comprising an evaluation unit that evaluates a user's performance with respect to the virtual object, the evaluation unit specifies a time point when the user performs a predetermined operation based on the image generated by the composite image generation unit, The performance of the user is evaluated according to the operation of the virtual object at the time.

  Preferably, the composite image generation means displays a composite image obtained for a time from a time before the user performs a predetermined operation to a time when the predetermined operation is completed to the user. To do.

  Preferably, the composite image generation means displays a composite image obtained at a time when the virtual object performs a predetermined operation to the user.

  A control program according to a second aspect of the present invention is an image generation system including a display device that displays a virtual object to a user, and a photographing device that photographs a user who performs a performance on the virtual object. A control program for acquiring a position in a virtual space corresponding to the position of the imaging device in real space, and an image of a virtual object viewed from the position acquired in the shooting position acquisition procedure A virtual image generation procedure to be generated, and a composite image generation procedure for synthesizing an image captured by the imaging device and an image of a virtual object generated in the virtual image generation procedure are executed by the image generation system. And

  Preferably, the method further includes a viewpoint position acquisition procedure for acquiring a position in the virtual space corresponding to the viewpoint position of the user in the real space, and the virtual image generation procedure is viewed from the position acquired in the viewpoint position acquisition procedure. An image of the virtual object is generated and displayed on the display device.

  Preferably, the method further includes a motion recognition procedure for acquiring information related to the user's motion based on the user's image captured by the imaging device, and the virtual image generation procedure is based on the information acquired in the motion recognition procedure. The movement of the virtual object is set.

  Preferably, the method further includes an evaluation procedure for evaluating the performance of the user with respect to the virtual object, and the evaluation procedure specifies a point in time when the user performs a predetermined operation based on the image generated in the composite image generation procedure. The performance of the user is evaluated according to the operation of the virtual object at the time.

  Preferably, in the composite image generation procedure, a composite image obtained from a time before a user performs a predetermined operation to a time when the predetermined operation is completed is displayed to the user. And

  Preferably, in the composite image generation procedure, a composite image obtained at a time when the virtual object performs a predetermined operation is displayed to the user.

  A recording medium according to the third aspect of the present invention is a computer-readable recording medium that records the control program.

  According to the present invention, a position in the virtual space corresponding to the position of the imaging device in the real space is acquired, and an image of the virtual object viewed from the acquired position is generated. Then, the image of the virtual object and the image of the user photographed by the photographing device are combined. As a result, a composite image that can recognize the positional relationship between the virtual object and the user is obtained. As a result, the user can obtain information useful for improving performance skills.

  In addition, a composite image is generated using the user image captured by the image capturing apparatus. From this, it is not necessary to attach many sensors and sensing aids to the user's body in order to generate a composite image, and the user can move as desired. In addition, since the user's facial expression and the like are expressed in the composite image, the user can obtain detailed information about the user.

It is the schematic which shows the space where the image generation system which concerns on embodiment of this invention is arrange | positioned. It is a block diagram which shows the structure of an image generation system. It is a flowchart which shows the process of the whole image generation system roughly. It is a figure which shows the example of the CG image displayed on HMD. It is a figure which shows the example of the picked-up image transmitted to PC from a video camera. It is a figure which shows the example of the synthesized image displayed on a display part. It is a flowchart which shows the detail of the process in PC. It is a flowchart which shows the detail of the process performed following the process of FIG. 7 in PC. It is a flowchart which shows the detail of the process performed following the process of FIG. 8 in PC. It is a figure which shows the transition of the motion of the other party player determined by the virtual image generation part. It is a figure which shows the time change of the position of an other party player, and the acceleration of a user's fist.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. This embodiment relates to a virtual karate system for supporting acquisition of basic skill in karate competition (non-contact rule), which is a non-contact type interpersonal game, and is a basic technique. A technique called “first upper thrust” is performed as an exercise task. This technique is a technique that pokes toward the face at the moment when the opponent player advances rapidly. In the drawings, the same or corresponding parts are denoted by the same reference numerals, and description thereof will not be repeated. FIG. 1 is a schematic diagram showing a space in which an image generation system 1 according to an embodiment of the present invention is arranged, and FIG. 2 is a block diagram showing a configuration of the image generation system 1.

  The image generation system 1 includes an HMD (Head Mounted Display) 2, a video camera 3, an acceleration sensor 4, and a PC (Personal Computer) 5.

The HMD 2 is worn on the user's (practice) 's head and allows the user to view the opponent player as a virtual object represented by CG. As this HMD2, for example, iWear VR920 ^™ (manufactured by VUZIX) can be used.

  The video camera 3 is installed in order to photograph the marker 6 representing the user and the world coordinate system. As this video camera 3, for example, Lu075cC-Io 112fps (manufactured by Lumenera) can be used. The world coordinate system is a coordinate system representing a space where the user is located, and the origin is indicated by the marker 6.

  The acceleration sensor 4 is attached to the user's arm and measures the timing of the user's bump. As this acceleration sensor 4, for example, WAA-001 (manufactured by Wireless Technology Co., Ltd.) can be used.

  In the image generation system 1 having the above-described configuration, the user and the video camera 3 for photographing the user are installed in the real space represented by the world coordinate system indicated by the marker 6 and coincide with the world coordinate system. The CG image of the opponent player is placed at the initial position determined by the exercise task in the virtual space having the virtual world coordinate system. The position and posture of this opponent player move with time according to the user's position and practice task. The image of the opponent player is generated as if it were an image taken by a camera placed at the real user's viewpoint (point A) and the position of the real video camera 3 (point B) and viewed from the point A The player image is sent to the HMD 2 and the CG image of the opponent player viewed from the point B is combined with the actual user image taken by the video camera 3 and added to the feedback image sequence.

  As a physical configuration, the PC 5 includes a control unit 50, a storage unit 51, and a display unit 52. The PC 5 can transmit and receive images to and from the HMD 2 and the video camera 3 via wired communication. Data can be transmitted and received via wireless communication.

The control unit 50 includes a CPU (Central Processing Unit) and the like, and executes processing according to a program stored in the storage unit 51. For example, an AMD Opteron ^™ processor 2.99 GHz can be used as the control unit 50.

  The storage unit 51 includes a main storage unit including a RAM (Random-Access Memory) and the like, and an external storage unit including a nonvolatile memory such as a flash memory and a hard disk. The main storage unit is used as a work area of the control unit 50 and is configured by a RAM having a capacity of 4.0 GB, for example. The external storage unit stores a program to be executed by the control unit 50 in advance, supplies data of the program to the control unit 50 in accordance with an instruction from the control unit 50, and stores data supplied from the control unit 50 To do.

  The display unit 52 is composed of a liquid crystal display device, for example, and can display an arbitrary image.

  FIG. 3 is a flowchart schematically showing processing of the entire image generation system 1. The process shown in FIG. 3 is to generate a composite image in which an image of an opponent player existing in a virtual space is superimposed on a captured image in a real space, using a mixed reality (MR) technique.

  As a premise that the process shown in FIG. 3 is performed, as shown in FIG. 1, a marker 6 representing the world coordinate system is installed at a place where a movement task is performed, and the user is placed at a position determined by the coordinate system of the marker 6. And a video camera 3 are arranged.

  First, the video camera 3 transmits an image of the marker 6 and the user taken to the PC 5 (Step S101).

  Then, the PC 5 acquires information on the position and orientation of the video camera 3 with respect to the world coordinate system based on the image of the marker 6 included in the captured image of the video camera 3 (step S103).

  Next, the PC 5 acquires the coordinates of the position in the virtual space corresponding to the position of the video camera 3 in the real space based on the information acquired in step S103 (step S104).

  Next, the PC 5 acquires the coordinates of the position in the virtual space corresponding to the viewpoint position of the user in the real space based on the marker 6 and the user image included in the captured image of the video camera 3 (step S105).

  Next, the PC 5 determines the initial position and orientation of the opponent player in the virtual space based on the coordinates in the virtual space acquired in steps S104 and S105 (step S106).

  Thereafter, the loop processing from step S107 is started. At the beginning, the video camera 3 transmits the user's captured image to the PC 5, and the acceleration sensor 4 transmits data indicating the timing of the user's bump to the PC 5 (step S102).

  The process shown in step S107 is not executed at the first loop.

  Next, the PC 5 generates a CG image of the opponent player viewed from the position in the virtual space corresponding to the user's viewpoint position (coordinates acquired in step S105) (step S108). In step S108 in the first loop, a CG image is generated according to the initial position and orientation determined in step S106.

  Next, the PC 5 generates a CG image of the opponent player viewed from the position in the virtual space corresponding to the position of the video camera 3 (the position acquired in step S104) (step S109). In step S109 in the first loop, a CG image is generated according to the initial position and orientation determined in step S106.

  Next, the PC 5 transmits the CG image generated in step S108 to the HMD 2 (step S110).

  Next, the HMD 2 displays the CG image received from the PC 5 (step S111). FIG. 4 shows an example of a CG image displayed on the HMD 2 in step S111, and the user visually observes the opponent player represented in the CG image and performs an operation such as “first upper butt”.

  The state of this user is photographed by the video camera 3, and this photographed image is transmitted to the PC 5. Further, the timing data of the user's bump is acquired by the acceleration sensor 4 and transmitted to the PC 5. FIG. 5 shows an example of a captured image transmitted from the video camera 3 to the PC 5. In the image of FIG. 5, a state in which the user wearing the HMD 2 is performing “first upper bump” is shown.

  After step S110 in FIG. 3 is executed, the PC 5 combines and stores the captured image of the video camera 3 and the CG image of the opponent player generated in step S109 (step S112).

  Next, the PC 5 determines whether or not the opponent player has reached the motion end point of the exercise task, or whether or not the user has completed the thrust (step S113).

  If it is determined that the opponent player has not reached the motion end point of the exercise task, or the user has not completed the thrust (NO in step S113), the process returns to step S107. By this return, the processing of steps S107 to S113 is executed. The processes in steps S107 to S113 are repeated until YES is determined in step S113. In each of the second and subsequent loops, the PC 5 determines the position / posture of the opponent player after the lapse of Δt time (step S107), and generates a CG image of the opponent player based on the position / posture (step S108, S109), the CG image of the opponent player generated in step S108 is transmitted to the HMD2. As a result, a movement is given to the opponent player displayed on the HMD 2, and the user performs an operation such as a karate thrust according to the movement. Then, the user's situation is photographed by the video camera 3, and the photographed image and the CG image of the opponent player generated in step S109 are synthesized and stored (step S112).

  If it is determined in step S113 that the opponent player has reached the motion end point of the exercise task or the user has completed the thrust (YES in step S113), the PC 5 stores the composition stored in step S112. The image is displayed on the display unit 52 (see FIG. 2) (step S114). FIG. 6 shows an example of the composite image displayed in step S114. The composite image shown in FIG. 6 is obtained by superimposing the opponent player's image on the captured image of FIG. The user can check the positional relationship with the opponent player, his / her performance according to the opponent player's movement, and his / her facial expression during the performance by using the composite image. In step S114, a plurality of composites stored in the time from when the opponent player starts moving forward until the player moves backward and returns to the original position, or the time from when the user starts hitting to completion is stored. Images are displayed continuously. Thereby, the moving image which shows the movement which the user and the other player performed in said time is displayed.

  Next, the PC 5 evaluates the collision performed by the user based on the composite image stored in step S112 (step S115).

  7, 8, and 9 are flowcharts showing processing in the PC 5. Hereinafter, the details of the processing in the PC 5 will be described with reference to these drawings.

  Here, as shown in FIG. 2, in the PC 5, the motion recognition unit 5a, the composite image generation unit 5b, the virtual image generation unit 5c, and the evaluation unit 5d are logically configured.

  The process illustrated in FIG. 7 is a process executed by the motion recognition unit 5a, the composite image generation unit 5b, and the virtual image generation unit 5c. S201 to S203 correspond to S107 in FIG. 3, S204 corresponds to S108, S205 corresponds to S109, S206 corresponds to S110, and S207 to S210 correspond to S112.

  Further, the process shown in FIG. 8 is a process executed by the composite image generation unit 5b following the process of FIG. S211 to S213 and S211 to S220 correspond to S113 in FIG. 3, and S214 and S221 correspond to S114.

  Further, the process of FIG. 9 is a process executed by the evaluation unit 5d following the process of FIG. 8, and corresponds to S115 of FIG.

  The motion recognition unit 5a, the composite image generation unit 5b, the virtual image generation unit 5c, and the evaluation unit 5d are stored in the storage unit 51 by the control unit 50 illustrated in FIG. It is configured by reading out to the main storage unit 51 and executing it.

  In the process of FIG. 7, first, the motion recognition unit 5a acquires user motion data (step S201). This motion data is acquired based on the timing data received from the acceleration sensor 4 and the captured image received from the video camera 3, and the motion data includes the position information of the user's fist and the timing information of the hit.

  Next, the motion recognition unit 5a sends the user's motion data to the composite image generation unit 5b and the virtual image generation unit 5c (step S202).

  Next, the virtual image generation unit 5c determines the opponent player's motion series based on the user's motion data (step S203).

  FIG. 10 is a diagram illustrating the motion transition of the opponent player determined by the virtual image generation unit 5c. The opponent player performs one of the operations of “defense posture jump”, “forward”, “feint advance”, and “reverse”, and the virtual image generation unit 5c converts the opponent player's motion into “ Probabilistic transition to the other three states centered on “defense posture jump”. The symbol P (i) (i = 0, 1, 2, 3; Σp (i) = 1) or the numerical value (1.0) attached to the arrow in the figure indicates the probability that the opponent player changes to a different action. For example, as an action following “defensive posture jump”, the forward movement is determined with the probability of p (1), the forward movement of the feint is determined with the probability of p (2), and the backward movement with the probability of p (3). Is determined.

  After step S203 in FIG. 7 is executed, the virtual image generation unit 5c views the CG image of the opponent player viewed from the viewpoint position of the user and the position of the video camera 3 according to the motion sequence determined in step S203. A CG image of the opponent player is generated (steps S204 and S205). The movement of the opponent player shown in these images is expressed based on the motion data of the karate player stored in the storage unit 51 in advance.

  Next, the virtual image generation unit 5c transmits the CG image of the opponent player viewed from the viewpoint position of the user generated in step S204 to the HMD 2 (step S206). With this process, the HMD 2 displays an image as shown in FIG.

  Next, the virtual image generation unit 5c sends the CG image of the opponent player viewed from the position of the video camera 3 generated in step S205 to the composite image generation unit 5b (step S207).

  Next, the motion recognition unit 5a sends the image captured by the video camera 3 to the composite image generation unit 5b (step S208).

  Next, the composite image generation unit 5b synchronizes the CG image of the opponent player received in step S207 and the captured image of the video camera 3 received in step S208 (step S209). By this processing, a composite image as shown in FIG. 6 is generated.

  Next, the composite image generation unit 5b stores the image combined in step S209 in the storage unit 51 (step S210).

  Next, the process of FIG. 8 executed following step S210 will be described.

  The composite image generation unit 5b determines whether or not the current time is a time T1 when the user should make a protrusion (step S211). Specifically, the composite image generation unit 5b determines whether or not the CG image of the opponent player received in step S207 indicates a state from when the opponent player starts moving forward to returning to the original position.

  In the following, the flow of processing when it is determined that the current time is at time T1 when the user should make a protrusion (YES in step S211) will be described.

  In this case, the composite image generation unit 5b performs marking on the composite image (the composite image stored in the immediately preceding step S210) stored in the storage unit 51 at the present time (step S212). This marking is performed to select a composite image to be displayed on the display unit 52 in step S214 described later.

  Then, the composite image generating unit 5b determines whether or not the current time is the time when the opponent player reaches the motion end point of the exercise task (step S213). Specifically, based on the CG image of the opponent player received in step S207, the composite image generation unit 5b determines whether the opponent player has moved forward and then moved backward to return to the original position. to decide.

  If it is determined in step S213 that the current time point is not the time point when the motion end point is reached (NO in step S213), the process returns to step S201 in FIG. As a result of this return, the processing of steps S201 to S213 is executed based on the data and the captured image newly received from the acceleration sensor 4 and the video camera 3. The processes in steps S201 to S213 are repeated until YES is determined in step S213. In each repetition, a CG image of the opponent player until reaching the motion end point is generated (steps S204 and S205), and the CG image of the opponent player generated in step S204 is displayed on the HMD2. The user visually observes the CG image and performs an operation such as a thrust, and the video camera 3 takes a picture of the user. Then, this captured image and the image of the opponent player generated in step S109 are combined and stored in the storage unit 51 (step S210). And the composite image memorize | stored in this case is produced | generated at the time T1 (time until an opponent player returns to the original position after an opponent player starts advancing) which a user should stick out (YES in step S211). ) Marking is performed in step S212.

  If it is determined in step S213 that the opponent player has reached the motion end point (YES in step S213), the composite image generation unit 5b displays the composite image that has been marked in the previous step S212. Is displayed on the display unit 52 (step S214). As a result, the composite image stored in step S210 is continuously displayed at the time T1 when the user should make a thrust. Note that in step S214, the composite image is displayed regardless of whether or not the user has made a thrust at time T1.

  Next, the case where it is determined in step S211 that the current time is not the time T1 when the user should make a collision (NO in step S211) will be described.

  In this case, the composite image generation unit 5b determines whether or not the user is making a thrust at the present time (step S215). This process is executed based on the motion data received in steps S202 and S208 and the captured image of the video camera 3.

  If it is determined that the user has not made a thrust (NO in step S215), the process returns to step S201. Due to this restoration, based on the data newly received from the acceleration sensor 4 and the video camera 3 and the captured image, the processing of steps S201 to S215 is repeated until YES is determined in S215.

When the user makes a thrust (YES in step S215), the composite image generating unit 5b determines whether or not the current time is a time t _e when the user starts the thrust (step S216). This process is executed based on the motion data received in step S202.

Then, current time, when the user is a time t _e that initiated the thrust (YES at step S216), the composite image generating unit 5b, currently, stored stored synthesized image in the storage unit 51 (in step S210 immediately before Marking is performed on the synthesized image) (step S217).

Next, the composite image generation unit 5b performs marking on the composite image stored in step S210 from the time t _e −Δ _e before the start of hitting to the current time (corresponding to t _e ) (step S218). .

  Next, the composite image generation unit 5b returns to the process of step S201. As a result of this return, the processes in steps S201 to S216 are repeated based on the data and the captured image newly received from the acceleration sensor 4 and the video camera 3.

  Then, when step S216 is executed again (when step S216 is the second time), since the time has elapsed since the user started hitting, the composite image generation unit 5b determines NO. Then, the composite image stored at the current time is marked (step S219).

Next, the composite image generation unit 5b determines whether or not the current time is the time t _f when the user completes the hit (step S220). This process is executed based on the motion data received in step S202.

  If the user has not completed hitting, and if NO is determined in step S220, the composite image generating unit 5b returns to the process of step S201. Due to this return, based on the data newly received from the acceleration sensor 4 and the video camera 3 and the captured image, the processes in steps S201 to S220 are executed until YES is determined in S220. In each repetition, the video camera 3 captures the state until the user completes the hit, and the captured image and the image of the opponent player generated in step S109 are combined and stored (step S210). ). Then, marking is performed on these synthesized images in step S219.

When the user completes the thrust (YES in step S220), the composite image generation unit 5b displays the composite image that has been marked in the previous steps S217, S218, and S219 on the display unit 52 (step S221). ). As a result, in step S221, the previous time t _e - [delta at time T2 from _e until the user completes the thrust time t _f, the composite image stored in the storage unit 51 in step S210 the user starts pushing Are displayed continuously.

  According to the above processing, a position in the virtual space corresponding to the position of the video camera 3 in the real space is acquired, and an image of the opponent player viewed from the acquired position is generated. The opponent player's image and the user's image taken by the video camera 3 are combined. As a result, a composite image capable of recognizing the positional relationship between the opponent player and the user as shown in FIG. 6 is obtained. Thereby, the user can obtain information effective in improving the skill of karate thrust.

  Further, a composite image is generated by using the user image taken by the video camera 3. From this, it is not necessary to attach many sensors and sensing aids to the user's body in order to generate the composite image, and the user can move as desired. In addition, since the user's facial expression and the like are expressed in the composite image, the user can obtain detailed information about the user.

In addition, when the user makes a hit, the composite image stored at time T2 from the time t _e −Δ _e before the hit is started to the time t _f when the hit is completed is displayed. Thereby, since the user can confirm the situation before and after he / she made a thrust, the user can consider whether or not the judgment of the thrust was appropriate. Incidentally, delta _e is set to a preferred time in order to check the status before and after the.

In addition, since the composite image stored before the time t _e −Δ _e is not displayed, it is possible to prevent the user from obtaining information that is not necessary for improving the skill.

  In addition, the time T1 from when the opponent player starts moving forward to the return to the original position is a time for the user to make a thrust, but the composite image stored at this time T1 is pushed by the user at the time T1. Displayed regardless of whether or not. For this reason, at time T, the user can confirm that he / she did not perform karate thrust, that the thrust did not reach the opponent player, and the like. Thereby, the user can confirm that the judgment and operation performed by himself / herself were incorrect.

  Next, processing executed by the evaluation unit 5d will be described with reference to FIG.

  First, the evaluation unit 5d specifies a time during which the distance between the user's fist and the opponent player is equal to or less than a predetermined value (step S222). Specifically, among the composite images stored in the storage unit 51, a composite image in which the distance between the user's fist and the opponent player is equal to or less than a predetermined value is extracted, and the time when these are stored is specified.

  Next, the evaluation unit 5d starts the forward movement of the opponent player when the acceleration of the user's fist reaches a negative peak at the time specified in step S222 (at the time of pulling back the fist that has struck, that is, at the time of squeezing). Then, it is determined whether or not the operation from the completion to the backward movement has been performed (step S223). Hereinafter, the point in time at which the acceleration of the user's fist has a negative peak is abbreviated as “a point at which the negative peak is reached”.

  FIG. 11 is a diagram showing temporal changes in the position of the opponent player and the acceleration of the user's fist. L1 to L3 shown in FIG. 11 indicate the time from when the opponent player starts moving forward until the backward movement is completed. Of these, L3 is the time from when the opponent player starts moving forward to just before starting the backward movement. In addition, L2 and L1 are divided into two equal parts from the time the opponent player starts retreating to completion, L2 is the first half time, and L1 is the second half time. It is. L4 is a time other than L1 to L3. In this time of L4, the opponent player is in a state where, for example, the feint is moving forward or the defense posture jump is performed.

  In step S223 of FIG. 9, it is determined whether the time point at which the negative peak occurs is between the times L1 to L3 and the time L4, so that the backward movement is completed after the opponent player starts moving forward. It is determined whether or not the above operations have been performed.

  When the negative peak occurs during the period from L1 to L3, it is determined that the opponent player has performed the operation from the start of the forward movement to the completion of the backward movement (YES in step S223), and step S224. Migrate to

  In this step S224, the evaluation unit 5d counts the evaluation points of the bumps made by the user.

  More specifically, the evaluation unit 5d determines that the negative evaluation point is given when the negative peak occurs at the time L3 (when the opponent player moves forward at the negative peak). As a result, 3 points are counted. In addition, when the time point of the negative peak occurs at the time of L2 (when the opponent player is performing the operation of the initial stage of retreat at the time of the negative peak), the evaluation score of 2 Count points. In addition, when the negative peak occurs at the time of L1 (when the opponent player is performing an operation at the completion stage of the reverse at the negative peak), 1 is used as the evaluation score of the collision. Count points.

  And the evaluation part 5d memorize | stores the evaluation score of the protrusion counted by step S224 in the memory | storage part 51 (step S225).

  On the other hand, when the negative peak occurs at the time of L4, it is determined in step S223 that the opponent player has not performed the movement from the start of the forward movement to the completion of the backward movement (step S223). NO), the process proceeds to step S226.

  Here, the situation determined as NO in step S223 occurs as a result of the user making a thrust when the opponent player is moving forward. Based on this, in step S226, one error value is counted as a value indicating that the timing of the hit is not appropriate.

  Then, the evaluation unit 5d stores the error value counted in step S226 in the storage unit 51 (step S227).

  Thus, the process shown in FIG. 9 ends, and the evaluation score and error value stored in the storage unit 51 in steps S225 and S227 are displayed on the display unit 52 when the user performs a predetermined operation on the PC 5. And confirmed by the user.

  According to the above processing, the time point at which the acceleration of the user's fist has a negative peak is specified based on the information related to the user's performance, and the user's performance is evaluated based on the action of the opponent player at that time. From this evaluation result, the user can determine whether or not the operation performed by the user is appropriate. In addition, the user can reach the target skill by repeating the practice until a desired evaluation is obtained.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims.

  For example, the acceleration sensor 4 may be omitted from the image generation system 1. In this case, data indicating the timing at which the user has made a thrust is acquired based on the captured image of the video camera 3. By doing so, the user does not need to attach the acceleration sensor 4 to the arm, so that the user can move as desired.

  A plurality of video cameras 3 may be installed. In this case, the process shown in FIG. 3 is executed based on the image captured by each video camera 3. As a result, captured images of the user from multiple viewpoints are obtained, and each of these captured images is used to generate a composite image in which the user and the opponent player face each other. Thereby, the positional relationship between the user and the opponent player viewed from multiple viewpoints can be confirmed.

  Further, the composite image generated by the composite image generation unit 5b may be displayed on a monitor or screen provided separately from the PC 5.

  In addition, the CG image of the opponent player generated by the virtual image generation unit 5c may be displayed on a monitor or a screen arranged in the vicinity of the user instead of the HMD2.

  Further, the PC 5 may exchange images with the HMD 2 and the video camera 3 via wireless communication.

  The image generation system 1 of the present invention can be used for sports practice, games, and rehabilitation other than karate such as boxing, tennis, and baseball. For example, when used for tennis practice, a PC 5 generates a CG image including an opponent player, a ball, and the like, and the CG image is projected on a screen. Then, the user confirms the movement of the ball or the like projected on the screen, and moves the arm holding the racket. Then, the user's situation is photographed by the video camera 3, and the photographed image and the CG image generated by the PC 5 are combined.

  The means and method for performing various processes in the image generation system 1 of the present embodiment can be realized by either a dedicated hardware circuit or a programmed computer. Here, the program may be provided by a computer-readable information recording medium such as a flexible disk or a CD-ROM. In this case, the program recorded on the computer-readable information recording medium is usually transferred to and stored in a storage unit such as a hard disk. The program may be provided as a single application software, or may be incorporated in the software of the device as one function of the device.

  Next, an experiment conducted for confirming the effect of the present invention will be described.

  In this experiment, two male students (21 years old, athletic club affiliation) were used as subjects, and the “first upper butt” practice was performed using the image generation system 1 of the above embodiment. One subject A was allowed to confirm the composite image (see FIG. 6) generated by the composite image generation unit 5b, and the other subject B was not allowed to confirm the composite image.

  Moreover, about the probability that an opponent player changes a motion, p (0) shown in FIG. 10 is set to 0.7, and p (1), p (2), and p (3) are all 0.1. Set to.

Also, in determining the "previous t _e - [delta _e time thrust is started" in the step S218 of FIG. 8, the time delta _e traced back from the time t _e the user has performed a thrust to the past, were set to 500 msec.

  In this experiment, the evaluation score and error value stored in steps S225 and S227 of FIG. 9 were output and confirmed on the PC 5 after each practice.

  Practice is 10 minutes at a time, with a frequency of 1 to 4 times a day until a score of 3 or less is obtained for an error score of 1 point or less and an average evaluation score of 2.5 points or more. I went there.

  The number of exercises performed until reaching the target score was 44 for subject A and 55 for subject B, and subject A who confirmed the composite image practiced compared to subject B who did not confirm the composite image. Was reduced by about 20%. From this, it was confirmed that the number of exercises required to reach the target skill can be reduced by checking the composite image.

  The present invention can be applied to equipment used for sports practice, rehabilitation, and computer games.

1 Image generation system 2 HMD
3 Video camera 4 Acceleration sensor 5 PC
5a Motion recognition unit 5b Composite image generation unit 5c Virtual image generation unit 5d Evaluation unit 6 Marker 50 Control unit 51 Storage unit 52 Display unit

Claims (13)

An image generation system including a display device that displays a virtual object to a user, and a photographing device that photographs a user performing performance on the virtual object,
Shooting position acquisition means for acquiring a position on the virtual space corresponding to the position of the shooting device in the real space;
Virtual image generation means for generating an image of a virtual object viewed from the position acquired by the shooting position acquisition means;
An image generation system comprising: a composite image generation unit that combines an image captured by the imaging device and a virtual object image generated by the virtual image generation unit. Further comprising viewpoint position acquisition means for acquiring a position in the virtual space corresponding to the viewpoint position of the user in the real space,
The image generation system according to claim 1, wherein the virtual image generation unit generates an image of a virtual object viewed from the position acquired by the viewpoint position acquisition unit and displays the image on the display device. Based on the user's image captured by the imaging device, further comprising motion recognition means for acquiring information on the user's motion,
The image generation system according to claim 1, wherein the virtual image generation unit sets a movement of a virtual object based on information acquired by the motion recognition unit. An evaluation means for evaluating a user's performance with respect to the virtual object;
The evaluation unit specifies a time point when the user performs a predetermined operation based on the image generated by the composite image generation unit, and evaluates the user performance according to the operation of the virtual object at the time point. The image generation system according to any one of claims 1 to 3, wherein   The composite image generation unit displays a composite image obtained for a time from a time before a user performs a predetermined operation to a time when the predetermined operation is completed to the user. The image generation system according to any one of 1 to 4.   The image according to any one of claims 1 to 5, wherein the composite image generation means displays a composite image obtained at a time when the virtual object performs a predetermined operation to a user. Generation system. A control program for an image generation system including a display device that displays a virtual object to a user, and a photographing device that photographs a user performing performance on the virtual object,
An imaging position acquisition procedure for acquiring a position on the virtual space corresponding to the position of the imaging device in the real space;
A virtual image generation procedure for generating an image of a virtual object viewed from the position acquired in the shooting position acquisition procedure;
A control program for causing an image generation system to execute a composite image generation procedure for combining an image captured by the imaging device and an image of a virtual object generated by the virtual image generation procedure. A viewpoint position acquisition procedure for acquiring a position in the virtual space corresponding to the viewpoint position of the user in the real space;
The control program according to claim 7, wherein in the virtual image generation procedure, an image of a virtual object viewed from the position acquired in the viewpoint position acquisition procedure is generated and displayed on the display device. . Further comprising a motion recognition procedure for obtaining information on the user's motion based on the user's image captured by the imaging device;
The control program according to claim 7 or 8, wherein in the virtual image generation procedure, a motion of a virtual object is set based on information acquired in the motion recognition procedure. Further comprising an evaluation procedure for evaluating a user's performance on the virtual object;
In the evaluation procedure, a point in time when the user performs a predetermined operation is specified based on the image generated in the composite image generation procedure, and the user's performance is evaluated according to the operation of the virtual object at that point. The control program according to any one of claims 7 to 9, wherein:   In the composite image generation procedure, a composite image obtained from a time before a user performs a predetermined operation until a time at which the predetermined operation is completed is displayed to the user. Item 11. The control program according to any one of Items 7 to 10.   12. The composite image generation procedure according to claim 7, wherein a composite image obtained at a time when the virtual object performs a predetermined operation is displayed to a user. Control program.   The computer-readable recording medium which recorded the control program of any one of Claims 7 thru | or 12.