JP2005034195A - Lesson support system and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To realize the lessen to meet the requests of the recipients by reducing the restriction of the lesson in the location, time and the like. <P>SOLUTION: The motions of the recipient are measured based on the outputs of a position attitude sensor mounted on the recipient and the position attitude sensor mounted on a golf club or a tennis racket, for instance. Then, the lesson is carried out displaying the instructor, the recipient, balls and the like as the virtual objects on the HMD mounted on the recipient utilizing the compound reality presentation technique. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a class support technology that enables a class such as sports to be attended with limited space and time.
[0002]
[Prior art]
There are various sports in the world now, and each sport is more specialized and sophisticated. For this reason, it is difficult for people to learn sports without taking classes from experts. Therefore, sports classes such as tennis classes and golf classes are also active.
[0003]
[Problems to be solved by the invention]
However, since this type of training is held at a specific location and according to a specific schedule, it may not be possible to take the training as expected. In order to improve sports, it is often effective to shoot the students' own forms and plays with videos, etc., and observe them objectively. In general, it is quite difficult to prepare a video camera for each person and always follow the trainee for shooting.
[0004]
In addition, instructional videos are also available on the market, but in general, sports practice such as golf swing practice cannot be performed in the room where the video is watched, and it can only be used for image training after all. .
[0005]
It has also been proposed to take a video of the student's swing, send it to the instructor, record the instruction video of the instructor, and send it back (Patent Document 1). There are still many problems, such as it takes time to return results and it is still difficult to practice while watching the video.
[0006]
The present invention has been made in view of the above-described problems of the prior art, and can reduce restrictions on the place and time of training such as sports, and can easily shoot and play back the actions of the trainee himself. A course support system and a course support method are proposed.
[0007]
[Patent Document 1]
JP-A-5-328275
[0008]
[Means for Solving the Problems]
In order to achieve the above-described object, the course support system according to the present invention includes model information for generating an instructor image as a virtual object, and class data for conducting a class predetermined for a student under the guidance of the instructor. A first sensor for detecting the head position / posture of the student, and a second sensor for detecting a position / posture of the tool operated by the student or a part of the student's body And the position and orientation detection means for detecting the position and orientation of the student from the outputs of the first and second sensors, the instructor image is generated according to the training data, model information and the position and orientation of the student, And an image generation means for displaying on the HMD to be mounted.
[0009]
The class support method according to the present invention includes a first sensor for detecting the head position and orientation of the student, and a first sensor for detecting the position and orientation of a tool operated by the student or a part of the student's body. The position and orientation detection step for detecting the position and orientation of the student from the output of the sensor 2, model information for generating an instructor image as a virtual object, and a class predetermined for the student by the instruction of the instructor In accordance with the training data and the position and orientation of the student detected in the position and orientation detection step, the image generation step for generating an instructor image and the image generated in the image generation step are displayed on the HMD worn by the student And a display step.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the drawings.
The course support system according to the present embodiment is realized by a mixed reality presentation device. The mixed reality presentation device superimposes a virtual object image (virtual object image) corresponding to the observer's viewpoint position and orientation on the real space or its image, and presents it to the observer as if it were virtual in the real space. It makes you feel as if an object exists.
[0011]
The mixed reality presentation apparatus according to the present embodiment presents a sports instructor, for example, a golf instructor, as a virtual object to a user (student), as if the instructor exists on the spot and instructs golf. This is a way for students to experience the sensation.
[0012]
FIG. 1 is a block diagram showing a schematic configuration inside the mixed reality presentation apparatus according to the present embodiment.
The CPU 101 controls the entire apparatus using data and program codes stored in the ROM 111, and also stores data and program codes (not shown) in the course progress management unit 110 described later, and data and program codes stored in the RAM 112. The image processing, numerical calculation, and the like described later are performed. Note that the locations of the above-described data and program codes are not particularly limited. For example, control program codes for the entire apparatus may be stored in the external storage device 113 or the RAM 112.
[0013]
The head position / orientation sensor 102 is attached to the user's head, and outputs a signal indicating the absolute position and orientation of the user's head (the position and orientation of the user's head in the absolute coordinate system), which will be described later. Output to the position and orientation measurement unit 103. Hereinafter, the absolute coordinate system may be referred to as a world coordinate system.
[0014]
As shown in FIG. 2, this world coordinate system is a coordinate system having an appropriate position as an origin and a z-x plane as a horizontal plane, and a gravity direction as a -y direction. Here, the origin is a position where a position / orientation measurement unit 103, which will be described later, is fixed, but the origin can be arbitrarily set if the world coordinate system is uniquely determined.
[0015]
The position / orientation measurement unit 103 identifies the position and orientation of the user's head from the above-described signals input from the head position / orientation sensor 102. On the other hand, the position and posture of the golf club held by the user in the world coordinate system are specified from a signal input from a golf club position and posture sensor 104 described later. Note that the identified head position and posture data and golf club position and posture data are output to the CPU 101.
[0016]
The golf club position / orientation sensor 104 may be the same as the head position / orientation sensor 102. When the golf club position / orientation sensor 104 is attached to the golf club, a signal indicating the position and orientation of the golf club in the world coordinate system is transmitted to the position / orientation measuring unit. To 103. The sensor can be mounted inside the grip, for example, and it is desirable to calibrate the reference of the output signal and the face orientation so that the face orientation of the club can be detected correctly.
[0017]
The position and posture of the user's head in the world coordinate system specified by the position and orientation measurement unit and the position and posture of the golf club are represented as follows.
Position of user's head at time t: h! (T)
h! (T) = (hx, hy, hz) (1)
However, h! Is a vector notation of h.
The posture of the user's head at time t:
θr Roll angle
θp Pitch angle
θy Yaw angle (azimuth angle)
The position of the golf club at time t: P! (T) (2)
P! (T) = (Px, Py, Pz)
However, P! Is a vector notation of P.
Attitude of golf club at time t:
φr Roll angle
φp pitch angle
φy yaw angle (azimuth angle)
[0018]
The user's head position / posture data and golf club position / posture data are input to the CPU 101. Here, a method in which the CPU 101 detects the movement of the golf club from these data will be described.
[0019]
The position P ′ of the hand based on the position of the user's head and the azimuth! Can be obtained by performing the following calculation (within the CPU 101).
P '! = P! × T_ (h!) × Ry_ (θy) (3)
However, T_ is the inverse transformation of the translational transformation T, and Ry_ is the inverse transformation of the rotational transformation Ry around the y axis.
[0020]
The hand position P ′ !! at time t + Δt is based on the orientation of the user's head. Moving vector d! Can be obtained as follows.
d! = P '! (T + Δt) −P ′! (T) (4)
[0021]
And the velocity vector v indicating the moving speed of the hand! Can be obtained as follows.
v! = D! / Δt (5)
[0022]
As a result, this velocity vector v! Based on the components (vx, vy, vz) in the x, y, and z directions, hand movements based on the user's head can be detected as follows.
vx> 0: moved to the right vx <0: moved to the left
vy> 0: moved up vy <0: moved down
vz> 0: moved after vz <0: moved before
[0023]
Returning to FIG. 1, the head mounted display (hereinafter referred to as “HMD”) 105 includes a CCD 105 b as an imaging device and a display unit 105 c inside, and an image synthesized by an image generation unit 106 described later (an actual image captured by the CCD 105 b And an instructor and a golf ball image (virtual object image) existing in the virtual space are displayed on the display unit 105c, so that the user who wears the HMD 105 on the head is as if the instructor and the golf ball are You can make it feel as if it actually exists.
[0024]
As is well known, there are two types of HMD 105 used in the mixed reality presentation device: an optical see-through method and a video see-through method. In this embodiment, a case where a video see-through HMD is used will be described. The HMD 105 may be an optical see-through HMD. When the HMD 105 is an optical see-through method, the CCD 105b is not provided.
[0025]
When the HMD 105 is of the video see-through method, the image generation unit 106 may perform arbitrary image processing (gamma correction, gradation conversion, etc.) on the real image input from the CCD 105b. Then, from the user's head position, posture information, club position, posture information, and instructor and golf ball three-dimensional model information stored in advance in the external storage device 113, and a course progress management unit to be described later. Based on the given instructor and the position and orientation information of the golf ball, an instructor and a golf ball image (virtual object image) suitable for the viewpoint position and orientation of the user are generated using a known computer graphics technique. Then, this image is combined with a live-action image captured by the CCD 105b or a real-image image obtained by subjecting the real-image image to arbitrary image processing, and the combined image is output to the display unit 105c.
[0026]
Note that the processing for converting the user's head position / posture into a viewpoint position / posture or generating the right-eye image and the left-eye image so that the HMD 105 can be stereoscopically viewed at the time of generating the virtual object image is based on the mixed reality presentation technology. Any known method can be used.
[0027]
On the other hand, when the HMD 105 is an optical see-through method, as described above, since the HMD 105 does not include the CCD 105c, the image generation unit 106 displays only the image of the instructor and golf ball to be presented on the video see-through HMD. To the unit 105c.
[0028]
The speaker 107 outputs the voice generated by the voice synthesis unit 108 described later. Note that what is output from the speaker 107 is not limited to sound, and for example, it is possible to output sound effects such as a collision sound between a ball and a golf club. The speaker can be configured as, for example, an earphone or a headphone worn by the user in addition to a normal speaker.
[0029]
Reference numeral 108 denotes a voice synthesizer, which generates a voice when the instructor utters a voice. Further, as described above, it is also possible to generate sound effects. Voice and sound effect data are stored in advance in the external storage device 113, for example, and are given to the voice synthesis unit 108 under the control of the course progress management unit 110.
[0030]
Although the course progress management unit 110 is described as a functional block for explanation, it is actually a part of the function realized by the CPU 101 and manages the progress of the golf course in order to realize predetermined course contents. .
[0031]
The ROM 111 stores at least a part of program codes and data necessary for the CPU 101 to control the entire apparatus. In addition, settings necessary when the apparatus is started up, character codes for displaying characters on the display unit, and the like are also stored.
[0032]
The RAM 112 has an area for temporarily storing program codes and data loaded from an external storage device 113 to be described later, and also has a work area used when the CPU 101 executes various program codes.
[0033]
The external storage device 113 includes a storage device such as an HDD, and stores various program codes and data installed from a storage medium such as a CD-ROM or a floppy (registered trademark) disk. When a shortage occurs in the work area in the RAM 112, a temporary file as a virtual work area can be created in the area in the external storage device 113.
[0034]
As will be described later, the course recording / playback unit 114 records the progress of the golf course in the external storage device 113 and can play back the recorded data.
The operation unit 115 is an input device for a user to give an instruction to the system. For example, the operation unit 115 is a remote controller provided with an arrow key or a numeric keypad.
[0035]
FIG. 3 is a diagram showing a basic overview of an MR golf course system using the mixed reality presentation apparatus having the above-described configuration. 301 is a user wearing the HMD 105, 302 is a virtual instructor visible to the user, and 303a and 303b are virtual golf balls. The user 301 can see the virtual instructor 302 and the virtual golf balls 303 a and 303 b through the HMD 105.
Since the user 302 of this system is a student of this training system, the user is hereinafter referred to as a student.
[0036]
<Overall processing>
Next, the overall processing of the mixed reality presentation device according to the present embodiment will be described using the flowchart shown in FIG.
First, when the apparatus is activated, various initialization processes such as reading of the operating system are performed. When the initialization process ends, a login screen is displayed on the display unit 105c. The login screen may be a general one having a text box for inputting a user name and a password, for example. The user inputs his / her user name and password registered in advance, and executes login using the execution key of the operation unit 115 or the like.
[0037]
In the present embodiment, the GUI operation is performed by the user using the operation unit 115. When the operation unit 115 is a device with few keys such as a remote controller, it is possible to use character input using a soft keyboard, for example, which is conventionally employed in video games.
[0038]
The user who takes the course for the first time selects and executes a “new user registration” button displayed on the login screen, for example.
In step S120, it is determined whether or not a new user registration button has been selected and executed. If so, the process proceeds to step S130, and if a normal login process is performed, the process proceeds to step S140.
[0039]
In step S130, user registration processing is performed. Here, setting of login user name and password, registration of parameters (gender, age, height, weight, arm and hand length, etc.) for user model generation described later, registration of other personal information (paid for attendance) If it is, registration of credit information etc. is performed. The registered information is stored in a user database included in the external storage device 113.
[0040]
On the other hand, in step S140, a user authentication process is performed by referring to the user database using the user name and password input on the login screen. Then, if the authentication is successful, initial information such as the contents of the training conducted so far is acquired from the user database. If authentication does not end normally, such as when the user name and password do not correspond, an error message is displayed, for example, and the process returns to step S110.
[0041]
In step S150, a training menu screen is displayed. Here, it is possible to display a different menu screen between the registered user and the newly registered user, such as displaying the “continue last time” button on the screen only for the registered user. Of course, an individual menu screen may be displayed for each user.
[0042]
In the system according to the present embodiment, there are three practice methods (an instructor model, an overlay model, and a training record / playback) as will be described later, and the training menu screen is configured so that these practice methods can be selected. Of course, as an automatic lesson, select a predetermined program (for example, beginner, intermediate, advanced course, ◎◎ shot intensive lesson, etc.), and perform the above three types of practice in each program. It doesn't matter. Here, in order to simplify the explanation, it is assumed that the user selects his / her desired one from three kinds of practice methods.
[0043]
Further, the class menu screen includes a button for instructing the end of the class. In step S160, it is determined whether the class has been instructed to end. When termination is instructed, record and update predetermined items (for example, information about the date and time of completion, information about the course history, billing information, etc.) among user information in the user database. To end the process.
[0044]
If no end instruction has been given, processing according to the selected practice method is performed in steps S170 to S190, and the user's instruction. The process returns to step S150 with the elapse of a predetermined time or the end of a predetermined lesson content as a trigger.
[0045]
Hereinafter, three kinds of practice contents realized by the system according to the present embodiment and processing for realizing the practice contents will be described.
[0046]
<Instructor model>
In this practice, the student performs the practice while observing the model operation performed by the instructor.
This will be described with reference to FIG.
The course progress management unit 110 generates or reproduces images of the instructor 302 and the golf ball 303a, which are virtual objects, using model data of a model operation stored in advance, and shows a model operation. During the operation display, the student can operate the operation unit 115 and observe the operation of the instructor from an arbitrary direction.
[0047]
The student performs an example observation by the instructor and practice using the virtual ball 303b. The explanation and advice of the virtual instructor 302 are stored in advance in the external storage device 113 as character data or voice data, and are displayed as characters on the HMD 105 as virtual object images, or converted into voice signals by the voice synthesizer 108 and converted into the speaker 107. More played.
[0048]
When the trainee performs a practice (trial hit), for example, an instruction is given from the operation unit 115. In response to this instruction, the course progress management unit 110 sets the virtual tee and the virtual golf ball to an appropriate place (for example, a place where the student can hit the ball without moving) based on the head position of the student. Display the image. Then, based on the position and orientation of the club determined by the position and orientation measurement unit based on the signal from the position and orientation sensor 104 attached to the club, the presence or absence of contact between the virtual ball and the club is detected. The initial velocity and direction of the ball are calculated according to the direction of the ball, how the ball touches the face, and the like.
[0049]
Then, according to the trajectory and speed of the swing, the orientation of the face, etc., advice based on previously stored characters and voice data is given to the student. The head speed and head position / orientation can be calculated from the output of the position / orientation sensor mounted on the grip and a preset club length. You may calculate from the value obtained in the measurement part 103. FIG.
[0050]
There are no particular restrictions on the practice content possible with this method, and everything from golf grips, stances and other basics to applications such as fade balls and bunker shots. When elements constituting the golf course such as green and bunker are necessary, the model information may be stored and displayed as a virtual image.
[0051]
In this embodiment, the position and orientation measurement unit 103 calculates the position and orientation relationship between the head and the golf club from the data of the head position and orientation sensor 102 and the golf club position and orientation sensor 104, and based on the data. The course progress management unit 110 estimates the dominant hand of the student. The dominant arm of the virtual instructor is determined according to the dominant arm of the student, but it may be possible to select right-handed or left-handed according to the preference of the student.
[0052]
<Overlay model>
Next, an overlay model will be described. This practice method makes it possible to clarify the difference between the instructor's movement and your own movement by displaying the student and the instructor as if they overlap, and to practice according to the instructor's movement. It is to make.
[0053]
The course progress management unit 110 uses the parameters for user model generation stored in the user database, or the position and orientation measurement unit 103 determines the head and position from the data of the head position and orientation sensor 102 and the data of the golf club position and orientation sensor 104. Calculate the position / posture relationship of the golf club and use the height, foot length, and hand length of the student estimated based on the data to create an instructor (virtual self-instructor) that has almost the same body shape as the student. Generate.
[0054]
Then, the virtual alternate instructor is displayed so as to overlap with the student. Accordingly, the student can recognize the movement of the virtual instructor as if he / she became a virtual instructor, and can adjust his / her movement to the movement of the virtual alternate instructor. For this reason, it is possible to imitate the instructor's movement more easily and appropriately than in the case of imitating the movement of the instructor as seen from a third person's viewpoint, as in a normal lesson.
[0055]
For example, the virtual instructor performs a swing or the like at the position of the student, and the student performs a swing or the like in accordance with the movement of the virtual instructor and practice. The explanation of the virtual alternation instructor becomes a voice signal by the voice synthesizer 108 and is reproduced from the speaker 107. The practice content that can be performed by this method is all about golf, as in <Instructor Model>.
[0056]
Note that the user's height, leg length, and hand length estimated automatically can be adjusted by the user using the operation unit 115. In addition, the angle of each joint can be adjusted according to the habit of the trainee.
Normally, the dominant arm of the virtual alternate instructor is determined according to the dominant arm of the student, but the student can also select whether the student is right-handed or left-handed.
[0057]
<Course record replay>
Next, the course recording / reproduction will be described. This process is performed by the course recording / playback unit 114.
The progress of the <Instructor Model> and <Superposition Model> described above can be recorded in the external storage device 113.
[0058]
The course recording / playback unit 114 records the progress of the course itself as well as the movements of the students. The method is shown below.
The position / orientation measurement unit 103 calculates the position / orientation relationship between the head and the golf club from the data of the head position / orientation sensor 102 and the data of the golf club position / orientation sensor 104. Collected and recorded in the external storage device 113.
[0059]
The position of the student's body from the recorded data and the student's body data (dominant arm, height, foot length, hand length, indirect angle adjustment value, etc.) stored in the user database , You can guess the posture and display the student's appearance like a virtual instructor. In other words, since the position and orientation relationship data of the head and the golf club are recorded in chronological order, if the student's figure is estimated from the position and orientation relationship data and virtual object images are generated and displayed in sequence, It is possible to reproduce the motion such as the swing as a video. In this estimation, the progress of the course may be referred to (that is, the practice of the recorded student movement may be used as an aid in estimating the student movement). ). In addition, as the texture data (face, clothes, etc.) of the student displayed as a virtual object, existing data set in advance may be used. For the face, for example, data obtained by actually photographing the student's face is used. Etc.
[0060]
Selection of data to be reproduced in the course recording / reproduction can be performed by an arbitrary method. For example, information (for example, “XX swing practice 2003-5-5” and its file path that can specify recorded data contents) can be specified. ) Is registered in the student's user database for each recording, a list of files that can be played back during the playback process is presented to the student, and any file is played back in response to a selection instruction from the operation unit 115 It can be configured as follows.
[0061]
The reproduction of the reproduced video based on the head position / posture information and the club position / posture information can be performed at any time after the collection of the position / posture information. As long as it is recorded in the device 113, the reproduced image can be viewed at any time.
[0062]
Since the reproduced video is based on the position-and-orientation data, it is possible to reproduce even the video viewed from any viewpoint, and it is also possible to move the viewpoint by instructing from the operation unit 115 during playback. . Therefore, it is possible to observe one's own swing from an arbitrary angle, and an efficient practice can be realized.
[0063]
Further, the playback speed can be changed by changing the number of virtual object images generated per unit time, and still images can be displayed by continuing to play back the same data.
[0064]
The video can be reproduced in reverse by reproducing the data in reverse order of time series, and frame-by-frame reproduction can be performed by skipping the data to be reproduced.
[0065]
<Simultaneous replay of class record>
In addition to the three practice methods described above, other practice methods can be employed. For example, you can superimpose the video of the student to be played in <Learning Record Playback> and the video of the virtual instructor appearing in <Overlay Model> so that the student can objectively move the virtual instructor and the student's own You can see the difference from movement.
[0066]
<Other embodiments>
In the above-described embodiment, only the system for performing a golf course using the virtual reality presentation apparatus of the present invention has been described, but it is possible to realize a course for any other sport. For example, instead of the club position / orientation sensor 104 in FIG. 1, a racket position / orientation sensor (the sensor itself may be the same, and the object to be attached becomes a tennis racket), and by changing the contents of the course, FIG. It is possible to realize a tennis training system as conceptually shown in FIG.
[0067]
Needless to say, the present invention can also be applied to sports that do not use a racket or other equipment or to classes other than sports. For example, when a class such as ballet or dance is performed, it is possible to learn how to move the hand or foot by attaching a position and orientation sensor to the hand or foot of the student. It can also be applied to musical performance classes.
[0068]
In the above-described embodiment, when a sports training support system is realized by using a mixed reality presentation device that uses only a sensor that detects the position and orientation of a student and a sensor that detects the position and orientation of a club or a racket. Explained. However, it is also possible to use three or more position and orientation sensors in order to detect the position and orientation of the student's body in more detail. Specifically, a position and orientation sensor may be attached to both hands and feet of the student.
[0069]
Moreover, you may implement | achieve the function equivalent to the virtual reality presentation apparatus of this invention with the system comprised from a some apparatus.
A software program that realizes the functions of the above-described embodiments is supplied directly from a recording medium or to a system or apparatus having a computer that can execute the program using wired / wireless communication. The present invention includes a case where an equivalent function is achieved by a computer executing the supplied program.
[0070]
Accordingly, the program code itself supplied and installed in the computer in order to implement the functional processing of the present invention by the computer also realizes the present invention. That is, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.
[0071]
In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS.
[0072]
As a recording medium for supplying the program, for example, a magnetic recording medium such as a flexible disk, a hard disk, a magnetic tape, MO, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-R, DVD- There are optical / magneto-optical storage media such as RW, and non-volatile semiconductor memory.
[0073]
As a program supply method using wired / wireless communication, a computer program itself that forms the present invention on a server on a computer network, or a computer that forms the present invention on a client computer such as a compressed file including an automatic installation function A method of storing a data file (program data file) that can be a program and downloading the program data file to a connected client computer can be used. In this case, the program data file can be divided into a plurality of segment files, and the segment files can be arranged on different servers.
[0074]
That is, the present invention includes a server device that allows a plurality of users to download a program data file for realizing the functional processing of the present invention on a computer.
[0075]
In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to the user, and key information for decrypting the encryption for a user who satisfies a predetermined condition is provided via a homepage via the Internet, for example. It is also possible to realize the program by downloading it from the computer and executing the encrypted program using the key information and installing it on the computer.
[0076]
In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of them and performing the processing.
[0077]
Furthermore, after the program read from the recording medium is written to a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU of the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.
[0078]
【The invention's effect】
As described above, according to the present invention, it is possible to realize a class with less restrictions on practice place and time by providing class support using mixed reality presentation technology.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a schematic configuration inside a mixed reality presentation apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram showing a world coordinate system.
FIG. 3 is a diagram illustrating an implementation image of a sports class realized in the embodiment of the present invention.
FIG. 4 is a flowchart illustrating an overall process of the mixed reality presentation apparatus according to the embodiment of the present invention.
FIG. 5 is a diagram illustrating an implementation image of a sports class realized in another embodiment of the present invention.
[Explanation of symbols]
101 CPU
102 Head position and orientation sensor
103 Position and orientation measurement unit
104 Golf club position and orientation sensor
105 HMD
105b CCD
105c Display section
106 Image generation unit
107 Speaker
108 Speech synthesis unit
110 Course Progress Management Department
111 ROM
112 RAM
113 External storage device
114 Training Record Reproduction Department
301 users (students)
302 Virtual Instructor

Claims (10)

Storage means for storing model information for generating an image of an instructor as a virtual object, and training data for performing a predetermined training for a student under the guidance of the instructor;
A first sensor for detecting a student's head position and posture;
A second sensor for detecting a position and orientation of a tool operated by the student or a part of the student's body;
Position and orientation detection means for detecting the position and orientation of the student from the outputs of the first and second sensors;
A class support system comprising: an image generation unit configured to generate an image of the instructor according to the class data, the model information, and the position and orientation of the student and display the image on an HMD worn by the student. The storage means further stores model information for generating an image of an object operated by the student directly or using the tool,
2. The image generation means also generates an image of the object according to the class data, the model information, and the position and orientation of the student, and displays the image on an HMD worn by the student. Class support system. The class support system according to claim 1, wherein the image generation unit generates an image for presenting a model operation by the instructor to the student and displays the image on the HMD. The image generation means exists as an image of the instructor at a position where the student visually recognizes an image of the instructor similar to the body shape of the student so that the image of the instructor and his / her body overlap each other. The class support system according to any one of claims 1 to 3, wherein the class support system is generated as a thing. Furthermore, it has a recording means for recording the position and orientation of the student,
The image generation means generates an image that reproduces the action of the student using the position and orientation of the student recorded in the recording means, and displays the image on the HMD worn by the student. The course support system according to any one of claims 1 to 4. From the output of the first sensor for detecting the head position and orientation of the student and the second sensor for detecting the position and orientation of the tool operated by the student or a part of the body of the student, A position and orientation detection step for detecting the position and orientation of the student;
Model information for generating an image of the instructor as a virtual object, class data for conducting a class predetermined for the student under the guidance of the instructor, and the position of the student detected in the position and orientation detection step An image generating step for generating an image of the instructor according to the posture;
And a display step of displaying the image generated in the image generation step on the HMD worn by the student. The image generation step further uses model information for generating an image of an object that the student operates directly or using the tool, according to the training data, the model information, and the position and orientation of the student. The class support method according to claim 6, wherein an image of the object is also generated. The class support method according to claim 6 or 7, wherein the image generation step generates an image for presenting the student with a model action by the instructor. The image generation step exists as an image of the instructor at a position where an image of the instructor similar to the body shape of the student is visually recognized so that the student overlaps the image of the instructor with his / her body The class support method according to any one of claims 6 to 8, wherein the class support method is generated. Furthermore, the recording step of recording the position and orientation of the student in the recording means,
The image generation step generates an image that reproduces the movement of the student using the position and orientation of the student recorded in the recording unit. Course support method described in the section.

Images