JP2017158184A - Optional viewpoint video composite display system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the following problem: cost reduction in a conventional motion training supporting system using an imaging apparatus causes an inappropriate display of a result on a trainer.SOLUTION: An image of a trainer is picked up by an imaging apparatus and posture data of the trainer is developed from the picked-up image. On the basis of the posture data, a posture model is created and the posture model is converted into a visual field image at an optional viewpoint to display the image on the trainer. A difference from another posture data which is a comparative object to the posture data of the trainer is displayed on the trainer.SELECTED DRAWING: Figure 7

Description

The present invention relates to an apparatus used for form confirmation in exercises such as sports.

In practice such as sports, checking the form of the practitioner from a viewpoint other than the practitioner is very useful for mastering, and there are several prior arts based on this viewpoint. Shown as 1.
In addition, there is a technique for estimating an area that cannot be directly captured by a camera based on the structure of a human body (Non-Patent Document 1).

JP 2011-152333 A “Posture Estimation Method from Single Image Based on Human Structure” Takashi Mitake, Shinya Yoshiji, Masaki Oshita (Kyushu University) (http://www.cg.ces.kyutech.ac.jp/paper/miru07 .pdf)

(Angle problem)
In the conventional method, the form can be examined only at the angle taken by the camera, so it was necessary to prepare multiple cameras or change the installation position of the camera as needed. That is, when a plurality of cameras are prepared, the cost increases, and there is a problem that if the installation position of the camera is changed each time, it is not convenient and complicated.

(Ease of understanding)
In addition, when an image taken by a practitioner is displayed on a large panel display standing around the HMD (head-mounted display) or the practitioner, settings such as left-right reversal processing are required, and this is not an easy-to-use method. In addition, since the video does not come from the practitioner's own viewpoint, the direction of motion does not coincide subjectively and it cannot be said that it is easy to understand intuitively.

Therefore, the present inventor made the present invention in order to solve these problems.

The practice support system of the present invention includes:
“With a photographic device to shoot the practitioner,
A posture estimation unit that estimates the posture of the trainee based on the video of the trainee photographed by the photographing device;
A posture model generation unit that generates a posture model based on the posture data estimated by the posture estimation unit;
A posture model display unit that displays the posture model generated by the posture model generation unit to the practitioner as an image from an arbitrary viewpoint. "
It is characterized by that.

Furthermore, the practice support system of the present invention provides:
“In addition to the practice support system described above,
A reference motion data holding unit that holds reference motion data that is a time-series posture data group for a practitioner to compare with herself, and a posture comparison unit,
The posture comparison unit compares the posture data estimated by the posture estimation unit with the posture data of the reference motion data held in the reference motion data holding unit, and displays the comparison result superimposed on the posture model display unit. "
It is characterized by that.

According to the present invention, it is possible to solve the angle problem and the ease of understanding problem described above, and it is possible to construct a practice support system with a minimum number of cameras.

It is a figure which shows the imaging device and practice person of this invention system. It is an image of the practitioner image | photographed with the imaging device of FIG. It is a figure which shows the imaging device and practice person of this invention system. It is an image of the practitioner image | photographed with the imaging device of FIG. It is a figure which compares a camera image and the visual field by a self viewpoint. It is a figure which shows the structure of this invention system. It is a figure which shows the outline | summary of the data processing of this invention system. It is a figure which shows the example of the image | video which this invention system outputs.

Example 1
In the system of the present invention, first, a photographing apparatus is installed at a place where a practitioner performs sports form practice. The setting device is specifically a video camera. Install the practitioner so that the practitioner does not leave the field of view of the video camera during a series of actions when the practitioner performs the form exercise.

The practitioner operates the system. The above-described photographing apparatus starts photographing, and displays the image processed based on the photographed image on the display unit worn by the practitioner. A specific example of the display unit is an HMD.
The relationship between the imaging device and the practitioner is as shown in FIGS. 1 and 3, and the imaging device is arranged around the practitioner.
The video imaged by the imaging device is determined by the relative positional relationship between the place where it is installed and the practitioner, for example, as shown in FIGS. Even in this state, the practitioner can objectively observe his / her own form, but as described in the above-mentioned problem, it is not easy to confirm the form with a different field of view.

FIG. 5 compares the difference between the image captured by the imaging apparatus and the practitioner's own visual field.
The image of the imaging device is based on the line of sight from the outside of the practitioner toward the practitioner, and the self-view is based on the line of sight going outward from the eyes on the practitioner's head. That is, the image of the image pickup device and the practitioner's self-view are essentially the opposite vectors and are difficult to replace each other.

FIG. 6 is a diagram showing the configuration of the system of the present invention. The processing of the system of the present invention will be described based on this figure.
First, the practitioner is captured as an image by the photographing unit (= imaging device). The data collection unit collects practitioner information other than video, and is, for example, a voice collection device such as a microphone when a motion sensor is attached to a specific part of the limb.

The posture estimation unit estimates the posture of the practitioner and generates posture data based on the video captured by the photographing unit (and the practitioner information collected by the data collection unit).
The 3D human body model generation unit generates model data of a three-dimensional human body model that is the same as the posture of the practitioner based on the posture data. Since this model data is generated by modeling, the appearance other than the posture may differ from the actual practitioner.

The virtual camera arrangement setting unit defines line-of-sight information (origin and direction) when the model data generated by the 3D human body model generation unit described above is used as an actual visible image. In the present embodiment, the position of the practitioner's eyes is set as the line-of-sight origin, and the direction of the line-of-sight is set as a predetermined part of interest (for example, an arm). However, other embodiments are not limited to this.
The set viewpoint image synthesis unit synthesizes a self-viewpoint image that is an image to be shown to the practitioner based on the above-described model data and line-of-sight information.
The display unit displays the aforementioned self-viewpoint image. Thereby, even without attaching an imaging device to the practitioner himself / herself, the practitioner can visually recognize an image modeled with respect to his / her posture.

The practitioner can instruct to change the field of view through an instruction unit (not shown), and can display a field of view from a desired arbitrary origin in an arbitrary direction on the display unit. Specifically, the instruction unit and the instruction method may be a dedicated small remote controller, or a voice recognition system may be used to give an instruction by a practitioner. The instruction unit and the instruction method are not particularly limited in the present invention.

(Example 2)
Next, comparison with reference motion will be described.
The reference motion is an object to be compared with the trainee's own form, for example, a model. Specifically, it takes in the form of professional athletes and instructors as motion data. The reference motion may be its own past form. In this embodiment, it is assumed to be a past form of itself.
The practitioner selects one of the reference motion data held in the reference motion data holding unit (not shown), and starts comparison with his / her own form.

FIG. 7 shows a procedure for comparing the model data of the past form with the current form of the present.
The recording unit in FIG. 6 is not used in the first embodiment, but records a series of model data of a form so that a behavior as a motion that is a continuous operation can be grasped. Based on this, the trajectory calculation unit calculates the moving direction and speed of the part where the practitioner moves. This calculation result can be notified to the practitioner via the display unit.

FIG. 8 is an example of a mode in which the difference between the reference motion data (its past form) and the current form is compared on the right arm and displayed on the display unit. The practitioner repeats the practice of the form while observing this difference, and corrects his form so that the difference disappears. As a result, you will learn your past forms anew.
In addition, as the reference motion data, a new form may be acquired by using a professional player or instructor as a model.

The system of the present invention may have only one photographing apparatus, but may have a plurality. A more accurate posture model can be generated if there are a plurality of posture models.
Also, even if there is only one shooting device, if the practitioner makes a single horizontal rotation while standing in front of the shooting device in advance, the image information necessary for generating the posture model can be obtained. High accuracy.

The present invention has been described with reference to preferred embodiments. However, the present invention is not limited to these embodiments, and many modifications can be made without departing from the spirit of the invention. Of course.

A motion training support system can be realized using mass-produced imaging devices and electronic equipment, and at least one imaging device can be used, so it is possible to reduce the cost and display the posture model generated from the captured video This makes it possible to practice efficiently and saves the man-hours and time of the practitioner.

Claims (2)

A practice support system in which a practitioner confirms the motion of a practitioner,
A photographing device for photographing a practitioner;
A posture estimation unit that estimates the posture of the practitioner based on the video of the practitioner photographed by the photographing device;
A posture model generation unit that generates a posture model based on the posture data estimated by the posture estimation unit;
A posture model display unit that displays the posture model generated by the posture model generation unit to the practitioner as a video from an arbitrary viewpoint;
A practice support system characterized by comprising: The practice support system according to claim 1,
further,
A reference motion data holding unit for holding reference motion data, which is a time-series posture data group for the practitioner to compare with himself,
A posture comparison unit;
Comprising
The posture comparison unit
Comparing the posture data estimated by the posture estimation unit with the posture data of the reference motion data held in the reference motion data holding unit;
A practice support system, wherein the comparison result is displayed superimposed on the posture model display unit.