JP2002008063A - Operation regenerating method, operation regenerating device, and recording medium with operation regenerating program recorded thereon - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the operation learning efficiency. SOLUTION: A measuring means 1 measures the model operation data of other person, and a measuring means 5 measures the operation data of a user. A time memory means 8 detects a starting point and a concluding point of the regeneration of a model operation image specified by the user on the basis of the operation data of the user. Measurement point extracting means 9 and 10, a ratio computing means 11 and a time changing means 12 change the model operation data stored in a memory means 4 so that the model operation image regenerating speed coincides with the specified speed. A regeneration control means 14 forms the model operation image data as the three-dimensional image data of a human body on the basis of the model operation data after adjusting the regenerating speed. A display means 7 displays the model operation image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a virtual reality system using stereoscopic vision, and more particularly to displaying a three-dimensional image of a model motion of another person when a user learns a motion such as a sport or a dance. TECHNICAL FIELD The present invention relates to an operation reproducing method, an operation reproducing apparatus, and a recording medium on which an operation reproducing program is recorded.

[0002]

2. Description of the Related Art Conventionally, there are roughly two methods for athletes of sports such as tennis, table tennis, badminton, and baseball to learn a form. One is a method in which an athlete learns under the guidance of an instructor. In this case, the competitor can receive appropriate advice by communicating his intention to the instructor. Another method is for a competitor to learn while watching a book or a video image of the model form of the instructor.

[0003]

If the instructor is not nearby, he or she will learn while looking at a two-dimensional model form provided by books or video images.
However, more effective learning is possible when the competitor actually learns using the body than learning by understanding the two-dimensional model form with his / her head. In view of this, a video superimposition method has been proposed that supports the learning of the athlete by superimposing and displaying the instructor's stereoscopic image in accordance with the body position of the athlete in a virtual environment using stereoscopic vision (Masayuki Ihara, Shiwa Shin-ichi, Satoshi Ishibashi, "3D image superimposed display for gesture learning support", IEICE Society Conference, 1999, A-16
12).

[0004] According to this video superimposition method, the competitor can recognize the difference between the form of the competitor and the instructor, and therefore can effectively learn the form. However, this video superimposition method cannot change the speed of the pre-stored leader's model form,
Athletes cannot use learning methods such as changing the speed of the model form according to their proficiency, and the competitor's movement cannot keep up with the speed of the model form,
As a result, when trying to learn a specific operation of a series of model forms, it is difficult to focus on the specific operation, and there is a problem that the learning efficiency of the operation is not improved.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and provides an operation reproducing method, an operation reproducing apparatus, and a recording medium on which an operation reproducing program is recorded, which can improve the learning efficiency of the operation. Aim.

[0006]

According to the motion reproducing method of the present invention, three-dimensional data of each body part of another person are used as model motion data of the other person.
Model operation data measurement processing for measuring the dimensional position (step 102) and reproduction start point detection processing for detecting the reproduction start point of the model operation video specified by the user (step 1)
05), and a reproduction end point detection process for detecting the reproduction end point of the model operation video specified by the user (step 10).
6), a reproduction speed adjustment process (step 107) for changing the model operation data so that the reproduction speed of the model operation video becomes the speed designated by the user, and based on the model operation data after the reproduction speed adjustment process. A reproduction process (step 109) for generating model motion video data as stereoscopic video data of a human body and sequentially displaying the model motion video in a virtual space constructed in a virtual reality system using stereoscopic vision. is there. As described above, in the present invention, after measuring the model operation data of another person, by detecting the reproduction start point and the reproduction end point of the model operation image specified by the user, the model operation image of the other person is measured. Among them, a specific operation video from the reproduction start point to the reproduction end point can be reproduced.
Also, in the present invention, the reproduction speed specified by the user is detected, and the exemplary operation data is changed so that the reproduction speed of the exemplary operation video becomes the speed specified by the user. Thereby, the specific operation video from the reproduction start point to the reproduction end point is reproduced at the reproduction speed desired by the user. Further, as one configuration example of the operation reproduction method of the present invention, the reproduction process pauses the reproduction when the movement of the user is stopped during reproduction of the exemplary operation video, and starts the movement after the pause. When the playback is completed, the playback is restarted.

[0007] One example of the configuration of the operation reproducing method according to the present invention is that a real space including a coordinate system of a virtual space and a user in a virtual reality system is provided between a model operation data measurement process and a reproduction start point detection process. (Step 103), and an initial display process (step 10) in which model motion video data is created based on the model motion data and a plurality of model motion videos are simultaneously displayed in the virtual space.
4), wherein the reproduction start point detection processing is performed by setting a distance d between a predetermined body part of the user and the same body part of a specific image in the displayed model motion image as a predetermined threshold. When the value d is α and the state of d <α continues for a certain time or more,
The exemplary operation video in which d <α is satisfied is detected as a reproduction start point. In the reproduction end point detection processing, after the reproduction start point is detected, the exemplary operation video in which d <α is satisfied is detected as the reproduction end point. This is the processing to be performed. Further, as one configuration example of the operation reproduction method of the present invention, the reproduction speed adjustment processing includes, as information indicating a reproduction speed desired by a user, a time at which the user specifies a reproduction end point and a reproduction start point. A first extraction process for obtaining a time difference (step 701); a time when the reproduction end point is measured and a time when the reproduction start point is measured in the model operation data of another person measured by the model operation data measurement process A second extraction process (step 702) for calculating the difference between the two, a calculation process (step 703) for determining the ratio of the difference extracted in the first extraction process to the difference extracted in the second extraction process, A time change process (step 704) for changing the measurement time of the model operation data based on the ratio so that the video reproduction speed becomes the speed designated by the user. The user can specify a desired reproduction speed by moving the predetermined body part of the user to the position of the same body part of the model motion video to be specified as the reproduction end point at the speed desired to be specified as the reproduction speed.

Further, the motion reproducing apparatus of the present invention includes a first measuring means (1) for measuring the three-dimensional position of each body part of another person as the model motion data of the other person, and the model motion data of the other person. Storage means (4) for storing, second measurement means (5) for measuring a three-dimensional position of each body part of the user as user operation data, and a user-specified operation data based on the user operation data A reproduction start point detecting means (8) for detecting a reproduction start point of the model operation video, and a reproduction end point detection means (8) for detecting a reproduction end point of the model operation video specified by the user based on the user's operation data. 8), reproduction speed adjusting means (9 to 12) for changing the model operation data stored in the storage means so that the reproduction speed of the model operation video becomes the speed designated by the user, and a model after the reproduction speed adjustment. Based on operating data Reproducing means (7, 14) for creating model motion video data as stereoscopic video data of a human body and sequentially displaying the model motion video in a virtual space constructed in a virtual reality system using stereoscopic vision It is. Further, as one configuration example of the operation reproducing apparatus of the present invention, the reproducing unit pauses the reproduction when the movement of the user measured by the second measuring unit stops during the reproduction of the exemplary operation video. , When the user starts moving after this pause,
This is to resume playback.

One example of the configuration of the motion reproducing apparatus of the present invention is a positioning means (6) for positioning the coordinate system of the virtual space with the coordinate system of the real space including the user in the virtual reality system. Initial display means (7, 14) for generating model motion video data based on the changed model motion data and displaying a plurality of model motion videos in a virtual space simultaneously; A distance between a predetermined body part of the user measured by the second measuring means and the same body part of a specific image in the model operation image displayed by the initial display means, d, a predetermined threshold When the state of d <α continues for a certain period of time or more, the exemplary operation video in which d <α is satisfied is detected as a reproduction start point, and the reproduction end point detection means detects the reproduction start point after detecting the reproduction start point. , D <α And it detects the video as a reproduction end point. In addition, one of the motion reproducing apparatuses of the present invention
As a configuration example, the playback speed adjusting unit is a first extraction unit that obtains a difference between a time at which a user designates a playback end point and a time at which a playback start point is designated, as information indicating a playback speed desired by the user. (9) second extraction means (10) for obtaining a difference between the time when the reproduction end point is measured and the time when the reproduction start point is measured in the model operation data of another person measured by the model operation data measurement process. ), Calculating means (11) for calculating the ratio of the difference extracted by the first extracting means to the difference extracted by the second extracting means, and the reproduction speed of the exemplary operation video is determined by the user. A time changing means (12) for changing the measurement time of the model operation data stored in the storage means based on the ratio.

The recording medium on which the operation reproduction program of the present invention is recorded is a model operation data measurement process for measuring the three-dimensional position of each body part of the other person as the model operation data of the other, and designated by the user. A reproduction start point detection process for detecting the reproduction start point of the model operation video, a reproduction end point detection process for detecting the reproduction end point of the model operation video specified by the user, and a reproduction speed of the model operation video set by the user. A reproduction speed adjustment process for changing the model operation data so that the speed becomes the speed specified by, and model operation video data that is stereoscopic video data of a human body based on the model operation data after the reproduction speed adjustment process is created. And a reproduction process of sequentially displaying the exemplary motion video in a virtual space constructed in the virtual reality system using the computer. Further, as one configuration example of a recording medium on which the operation reproduction program of the present invention is recorded, the reproduction processing pauses when the user's movement stops during reproduction of the exemplary operation video, and after this pause, When the user starts moving, the reproduction is restarted.

[0011] One example of the configuration of a recording medium on which the operation reproduction program of the present invention is recorded is such that a coordinate system of a virtual space and a reproduction system in a virtual reality system are provided between the exemplary operation data measurement processing and the reproduction start point detection processing. Positioning processing for aligning the position of the coordinate system in the real space including the user, and initial display processing for creating model motion video data based on the model motion data and simultaneously displaying a plurality of model motion videos in the virtual space. Is executed by the computer, and the reproduction start point detection processing is performed by setting a distance d between a predetermined body part of the user and the same body part of a specific image in the displayed exemplary motion image as d, which is predetermined. When the threshold value is α and the state of d <α continues for a certain period of time or more, the exemplary operation video in which d <α is satisfied is detected as a playback start point. After the detection of the starting point, a process of detecting a model operation image d <alpha is established as a reproduction end point. Further, as one configuration example of a recording medium on which the operation reproduction program of the present invention is recorded, the reproduction speed adjustment processing includes, as information indicating a desired reproduction speed of a user, a time at which a user designates a reproduction end point and a reproduction time. First extraction processing for obtaining a difference between the times at which the start points are designated, and the time at which the reproduction end point was measured and the reproduction start point were measured among the model operation data of the other person measured by the model operation data measurement processing. A second extraction processing for obtaining a time difference, a calculation processing for obtaining a ratio of a difference extracted in the first extraction processing to a difference extracted in the second extraction processing, and a reproduction speed of the exemplary motion video is determined by a user. And a time change process of changing the measurement time of the model operation data based on the ratio so that the speed becomes the speed designated by.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS [First Embodiment] Next, an embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a motion reproducing apparatus according to a first embodiment of the present invention, and FIG. 2 is a flowchart showing a schematic operation of the motion reproducing apparatus of FIG. The present invention measures the model motion data of another person (instructor of sports or dance, etc.) (step 102 in FIG. 2) and detects the reproduction start point of the model motion video specified by the user of the motion reproducing apparatus. (Step 105), a process of detecting the reproduction end point of the model operation video specified by the user (Step 106), and a model operation such that the reproduction speed of the model operation video becomes the speed specified by the user. It is characterized by performing a process of changing data (Step 107) and a process of sequentially displaying a model motion video in a virtual space based on the model motion data after the reproduction speed adjustment (Step 109).

Further, according to the present invention, when the user's movement stops during the reproduction of the exemplary motion picture, the reproduction is paused, and when the user starts moving after the pause, the reproduction is resumed (step). 109) is performed.

The operation reproducing apparatus shown in FIG.
raphics) in a virtual reality system. The virtual reality system has a structure in which, for example, screens are arranged at the top, bottom, left, and right in addition to the screen in front, and the user is surrounded by images. Furthermore, since the user wears stereoscopic glasses such as liquid crystal shutter glasses, a high sense of reality can be obtained as if the user were in the world. Instead of using screens and stereoscopic glasses, HMD (Head Mounted D
Glasses called isplay may be used. In this case, a stereoscopic video is displayed on the glasses.

The measuring means 1 of the motion reproducing apparatus measures model motion data of another person (instructor) performing the model motion, for example, by using a motion capture. Motion capture is a method for continuously measuring the movement of the human body (movement of each joint) in a three-dimensional space by a plurality of sensors attached to the human body, and capturing the three-dimensional coordinate values of each joint point in time series. Device.

By using such a motion capture, three points of each joint point of the human body such as a shoulder, an elbow, and a knee can be obtained.
The dimensional coordinate values can be captured as motion data.
The motion capture may be any of a magnetic type, an ultrasonic type, and an optical type. The storage unit 2 stores the data of the other person in the non-operation still state measured by the measurement unit 1.

The calculating means 3 calculates a line segment connecting the adjacent joint points at the same measurement time measured by the measuring means 1. The storage unit 4 stores the model operation data measured by the measurement unit 1, the line segment between the joint points calculated by the calculation unit 3, and the measurement time of the measurement unit 1 in association with each other.

The measuring means 5 has the same mechanism as the measuring means 1, but measures three-dimensional motion data of the user of the motion reproducing apparatus by using motion capture. The body part enlarging / reducing means 6 performs alignment processing for aligning a coordinate system of a virtual space constructed in the virtual reality system with a coordinate system of a real space including a user in the virtual reality system; Perform a scale matching process to match the scale of.

The display means 7 is an apparatus for providing a virtual environment using stereoscopic vision, and comprises the screen, a projector for projecting an image on the screen, and stereoscopic glasses. Further, the display unit 7 may be configured by using an HMD instead of these devices. The time storage means 8 is a reproduction start point detection means for detecting a reproduction start point of the model operation video specified by the user based on the user's operation data measured by the measurement means 5, and a reproduction start point detection means specified by the user. And a reproduction end point detecting means for detecting a reproduction end point of the exemplary operation video.

The measuring point extracting means 9 obtains a difference between the time when the user designates the reproduction end point and the time when the user designates the reproduction start point. The measurement point extracting means 10 obtains a difference between the time when the reproduction end point is measured by the measurement means 1 and the time when the reproduction start point is measured.

The ratio calculating means 11 calculates the ratio between the difference extracted by the measuring point extracting means 9 and the difference extracted by the measuring point extracting means 10. The time changing unit 12 controls the reproduction speed of the exemplary operation video to be the speed specified by the user.
The measurement time of the model operation data stored in the storage unit 4 is changed based on the ratio. The measuring point extracting unit 9, the measuring point extracting unit 10, the ratio calculating unit 11, and the time changing unit 12
It constitutes a reproduction speed adjusting means.

The storage means 13 stores the model operation data from the reproduction start point to the reproduction end point, the line connecting the adjacent joint points for each model operation data, and the measurement time changed by the time change means 12. Store them in association with each other. The playback control unit 14 performs an initial display process of simultaneously displaying a plurality of exemplary motion images on the virtual space by the display unit 7 and, when the playback start point, the playback end point, and the playback speed are specified after the initial display process, And a reproduction process for displaying the model operation video in chronological order.

Hereinafter, the operation of the operation reproducing apparatus of the present embodiment will be described in detail with reference to FIG. First, in order to perform the calibration, the other person performing the exemplary operation takes a posture in a non-operation still state. The measurement unit 1 captures the three-dimensional coordinate values of each joint point as data of the other person in the non-operation stationary state by the motion capture attached to the other person. The data of the other person in the non-operation still state is stored in the storage means 2 (step 101 in FIG. 2).

Next, the other person performs a model operation such as sports or dance. The measuring means 1, the calculating means 3, and the storage means 4
An exemplary operation data measurement process is performed (step 102). FIG. 3 is a flowchart for explaining the exemplary operation data measurement process. First, the measurement unit 1 captures the three-dimensional coordinate values of each joint point as model motion data of the other person during the model motion by the motion capture attached to the other person (Step 201 in FIG. 3).

The calculating means 3 calculates a line segment connecting adjacent joint points at the same measurement time measured by the measuring means 1 (step 202), and the storage means 4 stores the model operation data measured by the measuring means 1. Then, the line segment connecting the adjacent joint points and the measurement time of the measuring means 1 are stored in association with each other for each model motion data (step 203). Measuring means 1,
The calculation unit 3 and the storage unit 4 perform such data measurement, line segment calculation, and data storage at regular unit time intervals. Thus, the exemplary operation data measurement processing ends.

FIG. 4 shows measurement points at each measurement time taken in the exemplary operation data measurement processing on XY coordinates. In FIG. 4, the solid line represents the position of the arm, and t to
t + 6 represents the measurement time. Note that the actual coordinate system is an XYZ coordinate system.
-Y coordinate system.

Next, the user of the operation reproducing apparatus takes a posture of a non-operation still state in order to perform calibration before use. The measurement unit 5 and the body part enlargement / reduction unit 6 perform a position alignment process and a scale alignment process (Step 1).
03). FIG. 5 is a flowchart for explaining the position alignment process and the scale alignment process.

First, the measuring means 5 fetches three-dimensional coordinate values of each joint point from the motion capture attached to the user as data of the user in a non-operating and stationary state (step 301 in FIG. 5). Subsequently, the body part enlargement / reduction means 6 is determined in advance from the data of the user in the non-operation still state captured by the measurement means 5 and the data of the other person in the non-operation still state stored in the storage means 2. The coordinate values of the reference joint points are respectively extracted, and all the coordinate values of the model operation data stored in the storage unit 4 are uniformly adjusted so that the coordinate values of both the reference joint points become the same value (step 302).

The process of step 302 is to translate all other person's joint point coordinates so that the other person's reference joint point coordinates match the user's reference joint point coordinates in the three-dimensional space. means. The reference joint point differs depending on the application and the target operation. For example, the hip joint point may be set as the reference joint point. By the process of step 302, the position of the coordinate system of the virtual space and the position of the coordinate system of the real space including the user in the virtual reality system can be matched, and the distance between the exemplary motion image and the user can be obtained. Become. This completes the positioning process.

Further, the body part enlarging / reducing means 6 calculates the ratio of the length of each body part (the body part of the user) based on the data of the user in the non-operation stationary state and the data of the other in the non-operation stationary state. Length / the length of the body part of the other person), and stores this ratio (hereinafter referred to as the enlargement / reduction ratio) (step 3).
03).

For example, when the length of the upper right arm is obtained, the Euclidean distance is calculated from the coordinate value of the sensor mounted on the right shoulder and the coordinate value of the sensor mounted on the right elbow, for example. Find the distance between the point and the right elbow joint point. The body part enlargement / reduction means 6 calculates the length for each body part for each human body, and calculates the enlargement / reduction ratio for each body part.

The body part enlargement / reduction means 6 stores the data in the storage means 4 based on the calculated enlargement / reduction ratio so that the length of each body part of the other person matches the length of the corresponding body part of the user. The stored three-dimensional coordinate values and line segments of each joint point are changed (step 304). For example, if the enlargement / reduction ratio for the upper right arm is 1.2, the body part enlargement / reduction means 6
Changes the joint point coordinate values of the right shoulder and the right elbow and the line segment corresponding to the upper right arm in the coordinate values of all the joint points stored in the storage means 4, and sets the length of the upper right arm of the other person to 1. Enlarge twice. Through the above-described processing, it is possible to display a model motion image of another user of the same scale as the user. This completes the scale matching process.

Next, the reproduction control means 14 performs an initial display process (step 104). In the initial display process, the reproduction control unit 14 creates model operation video data, which is stereoscopic image data of a human body, based on the model operation data and the line segments stored in the storage unit 4 and outputs the created model operation image data to the display unit 7. .
The display means 7 displays the model operation video data output from the reproduction control means 14.

The model operation image data output from the reproduction control means 14 is data for presenting different images to the right and left eyes. By displaying these images, the user can use stereoscopic glasses or HMD. It is possible to give a three-dimensional effect to the user and present a virtual space having a three-dimensional effect to the user.

In the initial display processing, each stereoscopic video corresponding to the model operation data at each measurement time is displayed at the same time. As a result, a series of exemplary motion images of another person, which are originally displayed in chronological order, are displayed at a time. Note that this initial display processing is continued until a reproduction processing described later is started.

The user superimposes his / her predetermined body part on the same body part of the model motion image to be designated as the reproduction start point in the series of model motion pictures, and moves the body part at the position of the reproduction start point. Stop for more than a fixed time β. The measuring means 5 and the time storage means 8 perform a reproduction start point detection process for detecting a reproduction start point in accordance with the operation of the user (step 105). FIG. 6 is a flowchart for explaining the reproduction start point detection processing.

First, the time storage means 8 stores the elapsed time T = 0.
(Step 501 in FIG. 6). Measuring means 5
Captures three-dimensional coordinate values of a predetermined body part moved by the user for designating a reproduction start point by a motion capture attached to the user (step 50).
2).

The time storage means 8 stores the distance d between the three-dimensional position of the predetermined body part measured by the measuring means 5 and the three-dimensional position of the same body part stored in the storage means 4 in the storage means 4. The calculation is performed for each data at each stored measurement time (step 503). Then, the time storage unit 8 determines whether or not there is data in which the distance d is smaller than the predetermined threshold α in the data at each measurement time stored in the storage unit 4 (step 504).

If the data having the distance d less than the threshold value α does not exist in the storage means 4, the time storage means 8 stores the data of the user 3
It is determined that the three-dimensional position of the other person is far from the three-dimensional position of the other person, and the user has not designated the reproduction start point.
Return to The data in which the distance d is less than the threshold value α
In the case where the update time is present, the time storage means 8 increases the elapsed time T by one unit time (step 505), and determines whether or not the updated elapsed time T is greater than a predetermined time β (step 506). .

If the elapsed time T is equal to or smaller than the threshold value β, the measuring means 5 and the time storage means 8 execute steps 502 to 505
Is performed again. In this way, as long as the distance d is less than the threshold value α and the elapsed time T is less than or equal to the threshold value β, the processing of steps 502 to 505 is performed for each unit time.

In the processing of step 504 after T = 1, the predetermined body part of the user measured in the immediately preceding step 502 and the same body part of another person for which d <α holds at T = 0. Only when the distance d is less than the threshold value α, the judgment is Y
ES. That is, the predetermined body part of the user is T = 0.
If it has moved to a position different from that at the time, the process returns to step 501.

When the elapsed time T becomes larger than the threshold value β (when T = β + 1), the time storage means 8
The actual time corresponding to T = 0 is stored as the time at which the user has designated the reproduction start point, and the three-dimensional coordinate value of the predetermined body part of the user measured at T = 0 is stored (step 507).

Further, the time storage means 8 stores T <0 and d <
Measurement time of the exemplary operation data for which α has been established (step 20)
1) is stored as the reproduction start point time, and the position of the same body part of the other person where T = 0 and d <α holds is stored as the reproduction start point position. (Step 508). This completes the reproduction start point detection processing.

Next, the user stops the movement of the body part at the position of the reproduction start point for a certain period of time β or more, and then designates the own body part superimposed on the model operation image as the reproduction end point. The video is moved to the position of the same body part at the speed designated as the reproduction speed, and the movement of the body part is stopped at the position of the reproduction end point for a predetermined time β or more. The measuring means 5 and the time storage means 8 perform a reproduction end point detection process for detecting a reproduction end point in accordance with the operation of the user (step 106).

The reproduction end point detection processing is substantially the same as the reproduction start point detection processing of FIG.
The description of the process of 506 is omitted. The difference from the reproduction start point detection processing is that in step 507, the time storage means 8 stores the actual time corresponding to T = 0 as the time at which the user designates the reproduction end point, and T = 0.
Storing the three-dimensional coordinate values of the predetermined body part of the user measured in step 508;
Is stored as the time of the reproduction end point, and the position of the same body part of the other person where T = 0 and d <α is satisfied is set as the position of the reproduction end point. It is to remember.

FIG. 7 is an explanatory diagram for explaining the reproduction start point / end point detection processing. Also in FIG. 7, description will be made using an XY coordinate system as in the case of FIG. Here, it is assumed that the elbow is set in advance as the predetermined body part. The time storage unit 8 performs the process of step 507 when the state in which the distance d between the user and the other person is less than the threshold value α continues for a certain time period β or more.

In the example shown in FIG. 7, the position of the user's elbow at time T1 is a distance less than the threshold α from the position of the elbow of another person measured at time t1 (t1 is a time earlier than T1). If this state continues for a certain period of time β or more, the time storage means 8
Stores T1 as the time at which the user designates the reproduction start point, and stores the coordinates (X1, Y1) as the position of the user's elbow measured at T1 (T = 0). further,
The time storage means 8 stores the time t1 as the time of the reproduction start point, and also stores the coordinates (x1, y1) as the position of the reproduction start point.

In the example of FIG. 7, the time T2 (T1 <T
The position of the elbow of the user in 2) is located at a distance less than the threshold value α from the position of the elbow of another user measured at time t2. If this state continues for a certain period of time β or more, the time storage means 8 stores T2 as the time at which the user designates the reproduction end point, and stores the coordinates (X2, Y2) of the user's elbow measured at T2. Store as position. Further, the time storage means 8 stores the time t2 as the time of the reproduction end point, and also stores the coordinates (x2, y2) as the position of the reproduction end point.

After the end of the reproduction start point / end point detection processing, the measurement point extracting means 9 and 10, the ratio calculating means 11 and the time changing means 12 determine that the reproduction speed of the model motion video of another person is the speed specified by the user. (Step 107). FIG. 8 is a flowchart for explaining the reproduction speed adjustment processing.

First, the measuring point extracting means 9 extracts the number of measuring points of the measuring means 5 between the time T1 at which the user designates the reproduction start point and the time T2 at which the user designates the reproduction end point (FIG. 8).
Step 701). The number of measurement points is (T2-T1) /
It can be obtained in unit time. Subsequently, the measurement point extracting means 10 calculates the time t1 of the reproduction start point and the time t1 of the reproduction end point.
The number of measurement points of the measurement means 1 between the two is extracted (step 702). The number of measurement points can be obtained by (t2−t1) / unit time.

Next, the ratio calculation means 11 calculates the number of measurement points extracted by the measurement point extraction means 9
A ratio divided by the number of measurement points extracted by 0 is calculated (step 703).

Then, the time changing unit 12 stores the time
Out of the data at each measurement time (the model operation data and the line segment connecting the adjacent joint points for each model operation data) and the measurement time from the playback start point time t1 to the playback end point time t2. The data is read out, and each measurement time is changed by multiplying the measurement time in each read data by the ratio calculated by the ratio calculation means 11, and the exemplary operation data and line segment read out from the storage means 4 are changed. The time is stored in the storage means 13 (step 704). Thus,
The reproduction speed adjustment processing ends.

FIG. 9 is an explanatory diagram for explaining the reproduction speed adjustment processing. In the example of FIG. 4, when time t is designated as the reproduction start time and time t + 5 is designated as the reproduction end time,
The number of measurement points extracted by the measurement point extraction means 10 is 6. At this time, if the number of measurement points extracted by the measurement point extraction unit 9 is 12, the ratio calculated by the ratio calculation unit 11 is 2. By multiplying this value by each of the measurement times t to t + 5, the measurement time of the data shown in FIG. 4 is changed as shown in FIG. If the model motion video of the hand, the arm, and the elbow in FIG. 9 is reproduced at intervals of the measurement time after the change, the reproduction speed of the model motion video is １ ／ of that in FIG.

Next, when the user wants to reproduce the model operation video in which the reproduction start point, the reproduction end point, and the reproduction speed are designated, the user can specify the predetermined body part as the reproduction start point. Overlap on the same body part.

Although not explicitly shown in FIG.
Has always measured the three-dimensional coordinate value of a predetermined body part of the user every unit time. The reproduction control means 14 determines that the position of the reproduction start point stored in the time storage means 8 and the current position of the predetermined body part measured by the measurement means 5 are within a predetermined distance (in step 108). YES), a reproduction process is performed (step 109). FIG.
FIG. 7 is a flowchart for explaining this reproduction processing.

First, the reproduction control means 14
Is read, and based on the model operation data and the line segments in the read data, model operation image data that is stereoscopic image data of a human body is created and output to the display unit 7. The display means 7 displays the model operation video data output from the reproduction control means 14. In this way, the reproduction of the model motion video of another person is started from the reproduction start point (step 901 in FIG. 10).

As described in the reproduction speed adjustment processing described above, each measurement time in the data stored in the storage means 13 is changed so that the model operation image is displayed at the reproduction speed designated by the user. I have. The reproduction control means 14 converts the data read from the storage means 13 into stereoscopic video data according to the time interval of the measurement time. For example, in the example of FIG.
Assuming that the data at measurement time 2t is displayed at time T, the data at measurement time 2 (t + 1) is displayed at time T + 2, and the data at measurement time 2 (t + 2) is displayed at time T + 4. In this way, data is sequentially displayed at the time intervals of the measurement time.

Next, the user learns a form such as a sport or a dance in accordance with the model motion video whose reproduction has been started. If the user wants to temporarily stop the model motion video, the predetermined body part is once stopped. .

The reproduction control means 14 determines whether the reproduction of the data at the reproduction end time t2 has been completed (step 9).
02) If the data reproduction is not completed, it is determined whether or not the current position of the predetermined body part measured by the measuring means 5 has stopped (step 903).

When the predetermined body part of the user is not stationary, the reproduction control means 14 continues the reproduction. Therefore, as long as the user does not stop a predetermined body part, the processing of steps 902, 903, and 905 is repeated. On the other hand, when the three-dimensional position of the predetermined body part of the user is the same position for a certain period of time or more, the reproduction control means 14 determines that the predetermined body part of the user has stopped, and reproduces the exemplary motion video. It is temporarily stopped (step 904). When the reproduction is paused, the reproduction control means 14 continues to display the exemplary operation video displayed immediately before the pause.

Next, after temporarily stopping the reproduction of the exemplary motion video, the user moves the predetermined body part which is kept still when the user wants to resume the reproduction. In step 903, it is determined that the predetermined body part of the user is not stationary, and the reproduction is stopped (step 905).
YES), the reproduction control means 14 restarts reproduction of the exemplary operation video (step 906).

When the reproduction of the data at the reproduction end time t2 is completed by repeating the above processing, the reproduction control means 14 ends the reproduction processing. As described above, according to the present invention, the user can reproduce the model motion video of another person at a desired speed, and can efficiently perform self-learning.

In the present embodiment, the body of another person is represented by a line segment connecting adjacent joint points. However, a more realistic human body may be displayed using a person model. The person model stores detailed polygon coordinate values for displaying a person as three-dimensional computer graphics and textures necessary for texture mapping.

To display a more realistic human body using the human model, the reproduction control means 14 defines the three-dimensional coordinate values stored in the storage means 4 or 13 in advance as detailed polygon coordinate values and textures. It is sufficient to generate stereoscopic video data by applying to the person model in question.

[Second Embodiment] FIG. 11 shows a second embodiment of the present invention.
FIG. 3 is a block diagram showing a configuration of an operation reproducing apparatus according to the embodiment. The configuration of the operation reproducing apparatus according to the first embodiment, except for the display unit 7, can be realized by the computer 20. This computer 20 has a CPU 21 and a ROM (Re
ad Only Memory) 22, RAM (Random Access Memory)
23) Auxiliary storage device 2 such as a floppy disk device
4. Large-capacity auxiliary storage device 2 such as a hard disk device
5, an interface device 26 serving as an interface with the display means 7, an interface device 27 serving as an interface with a keyboard (not shown), and an interface device 28 serving as an interface with a motion capture (not shown).

In the apparatus shown in FIG. 11, a program for realizing the operation reproducing method of the present invention is provided in a state of being recorded on a recording medium such as a floppy disk, a CD-ROM, a memory card, or the like. When this recording medium is inserted into the auxiliary storage device 24 of the computer 20, the program recorded on the medium is read. And the CPU 21
The read program is written in the RAM 23 or the auxiliary storage device 25, and the processing described in the first embodiment is executed according to the program. Thus, processing similar to that of the first embodiment can be realized.

[0067]

According to the present invention, exemplary motion data measurement processing for measuring the three-dimensional position of each body part of another person as exemplary motion data of another person, and reproduction of an exemplary motion image specified by the user are started. Playback start point detection processing for detecting points;
Reproduction end point detection processing for detecting the reproduction end point of the model operation video specified by the user, and reproduction speed adjustment processing for changing the model operation data so that the reproduction speed of the model operation video becomes the speed specified by the user And model motion video data, which is stereoscopic video data of a human body, based on the model motion data after the reproduction speed adjustment processing, and model motion video is created in a virtual space constructed in a virtual reality system using stereoscopic vision. In a virtual reality system using stereoscopic vision, the user reproduces a specific motion in the model motion video of another person by designating a reproduction start point and a reproduction end point. It is possible to reproduce at a desired speed by designating the reproduction speed. As a result, when the learning cannot keep up with the normal reproduction speed of the exemplary operation video, the user can change the reproduction speed of the exemplary operation video according to his / her proficiency. For example, when a specific operation is not known among the exemplary operations, the operation can be played back in a slow manner to focus on learning. On the other hand, when the user has improved, the model operation video can be reproduced at high speed. Thus, the present invention can improve the operation learning efficiency.

When the user's movement stops during the reproduction of the model operation video, the reproduction is paused, and when the user starts moving after the pause, the reproduction is resumed. It is possible to arbitrarily designate a pause / resume of the operation video. As a result, convenience for the user can be improved.

In addition, between the exemplary operation data measurement processing and the reproduction start point detection processing, a positioning processing for aligning the position of the coordinate system of the virtual space with the coordinate system of the real space including the user in the virtual reality system is performed. This makes it possible to obtain the distance between the exemplary motion image and the user. Further, when a distance between the predetermined body part of the user and the same body part of a specific image in the displayed model motion image is d, and a predetermined threshold is α, a state of d <α is maintained for a predetermined time. When the above operation is continued, the reproduction start point detection process can be easily realized by detecting the exemplary operation video in which d <α is satisfied as the reproduction start point, and after the reproduction start point is detected, d <α is satisfied. By detecting the modeled operation video as the reproduction end point, the reproduction end point detection processing can be easily realized. The user superimposes his / her predetermined body part on the same body part of the model motion video that the user wants to specify as a playback start point in a series of model motion videos, and moves the body part at the position of the playback start point for a predetermined time or more. By stopping, the reproduction start point can be designated. Further, after designating the reproduction start point, the user superimposes his / her predetermined body part on the same body part of the model motion video to be designated as the reproduction end point, and keeps the movement of the body part constant at the position of this reproduction end point. By stopping for more than the time, the reproduction end point can be designated.

A first extraction process for obtaining a difference between a time at which the user designates a reproduction end point and a time at which the user designates a reproduction start point as information indicating a reproduction speed desired by the user; A second extraction process of calculating a difference between a time at which the reproduction end point is measured and a time at which the reproduction start point is measured among the model operation data of the other person measured by the process;
A calculating process for calculating a ratio between the difference extracted in the first extraction process and the difference extracted in the second extraction process; and a process of reproducing the model motion data so that the reproduction speed of the model motion video becomes a speed designated by the user. By performing the time change process of changing the measurement time based on the ratio, it is possible to easily realize the playback speed adjustment process of adjusting the playback speed of the exemplary operation video to a speed desired by the user. The user stops the movement of his / her predetermined body part at the position of the reproduction start point for a certain period of time or more, and then specifies the own body part superimposed on the model movement video as the reproduction end point of the same model movement video. By moving to the position of the body part at a speed desired to be specified as the reproduction speed and stopping the movement of the body part at the position of the reproduction end point for a certain period of time or more, a desired reproduction speed can be specified.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an operation reproducing apparatus according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing a schematic operation of the operation reproducing apparatus of FIG. 1;

FIG. 3 is a flowchart illustrating exemplary operation data measurement processing according to the first embodiment of the present invention.

FIG. 4 is an explanatory diagram showing measurement points at each measurement time taken in an exemplary operation data measurement process on XY coordinates.

FIG. 5 is a flowchart illustrating a positioning process and a scaling process according to the first embodiment of the present invention.

FIG. 6 is a flowchart for explaining a reproduction start point / end point detection process according to the first embodiment of the present invention.

FIG. 7 is an explanatory diagram illustrating a reproduction start point / end point detection process according to the first embodiment of the present invention.

FIG. 8 is a flowchart illustrating a reproduction speed adjustment process according to the first embodiment of the present invention.

FIG. 9 is an explanatory diagram for describing a reproduction speed adjustment process according to the first embodiment of the present invention.

FIG. 10 is a flowchart illustrating a reproduction process according to the first embodiment of the present invention.

FIG. 11 is a block diagram illustrating a configuration of an operation reproducing apparatus according to a second embodiment of the present invention.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Measurement means, 2 ... Storage means, 3 ... Calculation means, 4 ... Storage means, 5 ... Measurement means, 6 ... Body part enlargement / reduction means, 7 ...
Display means, 8 time storage means, 9, 10 measurement point extraction means, 11 ratio calculation means, 12 time change means, 13
Storage means, 14 ... reproduction control means.

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Satoshi Ishibashi 2-3-1 Otemachi, Chiyoda-ku, Tokyo Nippon Telegraph and Telephone Corporation F-term (reference) 5B050 AA06 BA04 BA12 CA09 EA12 EA19 FA01 FA06 5C061 AA01 AA20 AA21 AB14

Claims (12)

[Claims] 1. An operation reproducing method for teaching an operation to a user by displaying a model motion image of another person in a virtual reality system using stereoscopic vision, wherein the model motion data of the other person is used as the model motion data of the other person. Model motion data measurement processing for measuring the three-dimensional position of each body part, reproduction start point detection processing for detecting the reproduction start point of the model motion video specified by the user, and model motion video specified by the user A reproduction end point detecting process for detecting a reproduction end point of the reproduction speed adjustment process for changing the model operation data so that a reproduction speed of the model operation video becomes a speed designated by a user; and a reproduction speed adjustment process. Model motion video data, which is stereoscopic video data of the human body, is created based on the model motion data later, and is modeled on a virtual space constructed in a virtual reality system using stereoscopic vision. Operation reproducing method characterized by comprising a playback process for displaying an operation image sequentially. 2. The motion reproduction method according to claim 1, wherein the reproduction process pauses when a user's movement stops during reproduction of the exemplary motion video, and after the pause, the user stops the reproduction. An operation reproducing method characterized by resuming reproduction when it starts moving. 3. The motion reproducing method according to claim 1, wherein the real space including a coordinate system of the virtual space and a user in a virtual reality system is provided between the model motion data measuring process and the reproduction start point detecting process. And an initial display process for creating model motion video data based on the model motion data and simultaneously displaying a plurality of model motion videos in a virtual space. The reproduction start point detection processing is performed by setting a distance between the predetermined body part of the user and the same body part of a specific video in the displayed model motion video as d, and a predetermined threshold value as α, where d is
When the state of <α continues for a certain period of time or more, the exemplary operation video in which d <α is established is detected as a reproduction start point. The reproduction end point detection processing includes the following steps: An operation reproducing method, characterized in that it is a process of detecting, as a reproduction end point, a model operation video in which α has been established. 4. The operation reproduction method according to claim 3, wherein the reproduction speed adjustment processing specifies, as information indicating a reproduction speed desired by the user, a time at which the user specifies a reproduction end point and a reproduction start point. A first extraction process for obtaining a difference between the times when the reproduction end point was measured and a time when the reproduction start point was measured in the model operation data of the other person measured by the model operation data measurement process. A second extraction process for calculating the ratio of the difference extracted in the first extraction process and the difference extracted in the second extraction process; and a reproduction speed of the exemplary motion video specified by a user. A time change process of changing the measurement time of the exemplary operation data based on the ratio so that the speed becomes the set speed. 5. An operation reproducing apparatus for instructing a user by displaying a model motion image of another person in a virtual reality system using stereoscopic vision, wherein the model motion data of the other person is used as the model motion data of the other person. First measuring means for measuring the three-dimensional position of each body part, storage means for storing the model movement data of the other person, and measuring the three-dimensional position of each body part of the user as movement data of the user A second measurement unit, a reproduction start point detection unit that detects a reproduction start point of the exemplary operation video specified by the user based on the operation data of the user, and a reproduction start point detection unit that detects the reproduction start point of the user based on the operation data of the user. A reproduction end point detecting means for detecting a reproduction end point of the designated model operation video; and a model stored in the storage means such that a reproduction speed of the model operation video becomes a speed designated by a user. Reproducing speed adjusting means for changing motion data; and model motion video data which is stereoscopic video data of a human body based on the model motion data after the reproduction speed adjustment, and is constructed in a virtual reality system using stereoscopic vision. A reproduction unit for sequentially displaying model motion images in a virtual space. 6. The motion reproducing apparatus according to claim 5, wherein the reproducing unit performs the reproduction when the movement of the user measured by the second measuring unit is stopped during the reproduction of the exemplary operation video. An operation reproducing apparatus, which pauses and resumes reproduction when a user starts moving after the pause. 7. The motion reproducing apparatus according to claim 5, wherein: a position adjusting unit that aligns a coordinate system of the virtual space with a coordinate system of a real space including a user in the virtual reality system; Initial display means for creating model motion video data based on the motion data and simultaneously displaying a plurality of model motion videos in a virtual space, wherein the reproduction start point detection means is provided by the second measurement means Assuming that the distance between a predetermined body part of the user to be measured and the same body part of a specific image in the model operation image displayed by the initial display means is d, and a predetermined threshold is α, d <α When the state of (1) or more has continued for a predetermined time or more, the exemplary operation video in which d <α has been established is detected as the reproduction start point. A model move An operation reproducing apparatus for detecting a created video as a reproduction end point. 8. The operation reproduction device according to claim 7, wherein the reproduction speed adjusting means specifies a time at which the user has specified a reproduction end point and a reproduction start point as information indicating a reproduction speed desired by the user. First extracting means for calculating a difference between the times obtained, and a difference between a time at which the reproduction end point is measured and a time at which the reproduction start point is measured in the model operation data of the other person measured by the model operation data measurement processing. Second extracting means for calculating the ratio of the difference extracted by the first extracting means and the difference extracted by the second extracting means; and a reproduction speed of the exemplary motion video specified by a user. A time changing means for changing the measurement time of the model operation data stored in the storage means based on the ratio so that the speed becomes the set speed. 9. A recording medium storing an operation reproduction program for teaching a user to perform an operation by displaying an exemplary operation image of another person in a virtual reality system using stereoscopic vision, the method comprising: Model motion data measurement processing that measures the three-dimensional position of each body part of another person as motion data, playback start point detection processing that detects the playback start point of the model motion video specified by the user, and user specification A reproduction end point detection process of detecting a reproduction end point of the model operation video, and a reproduction speed adjustment process of changing the model operation data so that the reproduction speed of the model operation video becomes a speed designated by a user. Based on the model motion data after the reproduction speed adjustment processing, model motion video data, which is stereoscopic video data of a human body, is created, and is created in a virtual reality system using stereoscopic vision. A recording medium recording the operation playback program for executing a reproduction process on a computer that sequentially displays model operation image in a virtual space to be built. 10. The recording medium on which the operation reproduction program according to claim 9 is recorded, wherein the reproduction processing pauses when a user's movement stops during reproduction of the exemplary operation video, and the reproduction is temporarily stopped. A recording medium on which an operation reproduction program is recorded, wherein reproduction is resumed when the user starts moving after stopping. 11. A recording medium on which the motion reproduction program according to claim 9 is recorded, wherein the use of the virtual space coordinate system and the virtual reality system between the model motion data measurement process and the reproduction start point detection process is performed. Positioning processing for adjusting the position of the coordinate system of the real space including the person, and initial display processing to create model motion video data based on the model motion data, and to simultaneously display a plurality of model motion videos in the virtual space. Wherein the reproduction start point detection processing includes: a distance d between a predetermined body part of the user and the same body part of a specific video in the displayed model operation video, and a predetermined threshold value α As d
When the state of <α continues for a certain period of time or more, the exemplary operation video in which d <α is established is detected as a reproduction start point. The reproduction end point detection processing includes the following steps: A recording medium storing an operation reproduction program, wherein the operation reproduction program is a process of detecting a model operation video in which α has been established as a reproduction end point. 12. The recording medium on which the operation reproduction program according to claim 11 is recorded, wherein the reproduction speed adjustment processing includes, as information indicating a reproduction speed desired by the user, a time at which the user specifies a reproduction end point. A first extraction process for calculating a difference between times at which a reproduction start point is designated; and a time when a reproduction end point is measured and a reproduction start point are measured in the model operation data of another person measured by the model operation data measurement process. A second extraction process for obtaining a difference between the extracted times, a calculation process for obtaining a ratio of the difference extracted in the first extraction process to the difference extracted in the second extraction process, and a reproduction speed of the exemplary motion video And a time changing process of changing the measurement time of the exemplary operation data based on the ratio so that the speed becomes a speed designated by a user. Recording media.