WO2023242981A1

WO2023242981A1 - Head-mounted display, head-mounted display system, and display method for head-mounted display

Info

Publication number: WO2023242981A1
Application number: PCT/JP2022/023903
Authority: WO
Inventors: 伸和近藤; 仁秋山; 康宣橋本; 眞弓中出; 宏司中森
Original assignee: マクセル株式会社
Priority date: 2022-06-15
Filing date: 2022-06-15
Publication date: 2023-12-21

Abstract

This head-mounted display comprises a processor and a display. The processor displays actual video, which shows an action of a user, and an example video, which shows an example of an action, in a superimposed manner on the display, from a viewpoint corresponding to the angle of rotation of the neck of the user who is wearing the head-mounted display on the head.

Description

Head-mounted display, head-mounted display system, and display method of head-mounted display

The present invention relates to a head mounted display, a system using the head mounted display, and a display method in the head mounted display. Note that, hereinafter, a head mounted display may be referred to as an HMD.

Augmented reality (hereinafter sometimes referred to as AR) is known as a technology that supplements (augments) scenes and images of the real world by overlaying them with digital information. It is used in a variety of fields such as engineering, civil engineering, and retail. In order to experience AR, for example, an HMD that supports AR is used. This HMD is, for example, a device that is worn on the head and displays augmented reality images (hereinafter sometimes referred to as AR images) on a goggle-like display. For example, this device is equipped with a plurality of sensors such as a camera and a position measurement sensor, a CPU that performs image processing, a battery, and the like.

This type of technology is sometimes used to assist users in acquiring skills and learning. Patent Document 1 discloses a learning support system including a glasses-type device and a control system. The glasses-type device includes a display section and an imaging section (camera) that captures a visual field image of the learner, and is worn by the learner. The control system is connected to the glasses-type device via a network and controls the functions of the glasses-type device. The calculation unit of the control system superimposes a model video, which is a video of the instructor's work movement that serves as a model for the learner's work movement, on the visual field image captured by the camera and displays it on the display unit. The display contents of the model video are dynamically changed according to the characteristics of the work movements of the learners involved.

This type of technique is also sometimes used for training. Patent Document 2 discloses a technique in which a model video is displayed on a display of a user terminal together with a user's training video, and the model video and the user's actual training video can be compared on the display.

JP 2020-144233 Publication Japanese Patent Application Publication No. 2020-195573

However, especially in exercise systems that involve the whole body, when practicing while superimposing the model and self-image, it is difficult to see the difference between the model and your own movements (correction points) in real time from the front or from a viewpoint or angle that has been set so that it can be observed in advance. Sometimes cases occur that are difficult to observe. Here, it is thought that it is effective to change the display angle appropriately, but it is considered that Patent Document 1 and Patent Document 2 do not specifically disclose a technique for changing the display angle.

SUMMARY OF THE INVENTION The present invention provides a head-mounted display, a head-mounted display system, and a head-mounted display system that allow the user to easily switch to an angle where the user can easily see the difference between the model and the automatic production when displaying a self-image and a model image in a superimposed manner. The purpose of this invention is to provide a display method for a head-mounted display.

According to the first aspect of the present invention, the following head mounted display is provided. That is, the head mounted display includes a processor and a display. The processor superimposes and displays a self-image showing the user's movement and a model image showing the model movement on the display from a viewpoint corresponding to the rotation angle of the neck of the user wearing the head-mounted display on the head.

According to the second aspect of the present invention, the following head mounted display system is provided. That is, the head mounted display system includes a distribution device and a head mounted display. The distribution device distributes a model video showing a model operation. The head mounted display includes a processor and a display. The processor superimposes and displays the user's self-image showing the user's actions and the model image distributed from the distribution device on the display from a viewpoint corresponding to the rotation angle of the neck of the user wearing the head-mounted display on his head.

According to the third aspect of the present invention, the following display method for a head mounted display is provided. That is, the display method of the head-mounted display is a method using a processor. In this method, a self-image showing a user's movement and a model image showing a model movement are superimposed and displayed on a display at a viewpoint corresponding to the rotation angle of the neck of a user wearing a head-mounted display on his head.

According to the present invention, there is provided a head-mounted display, a head-mounted display system, and a head-mounted display system that allow a user to easily switch to an angle where the user can easily see the difference between the model and the automatic production when displaying a self-image and a model image in a superimposed manner. , a display method for a head-mounted display is provided.

FIG. 2 is a block diagram illustrating an example of the hardware configuration and functions of an HMD. FIG. 3 is a diagram for explaining an overview of superimposed display from a viewpoint above the user. FIG. 2 is a diagram for explaining an overview of superimposed display from behind the user. FIG. 6 is a diagram for explaining an example of rotation of an image and an observed image. FIG. 6 is a diagram for explaining an example of rotation of an image and an observed image. 12 is a flowchart for explaining an example of processing related to superimposed display. FIG. 3 is a diagram for explaining an example of initial settings. FIG. 6 is a diagram illustrating an example of setting a viewpoint direction based on a rotation angle of the neck. FIG. 6 is a diagram illustrating an example of setting a viewpoint direction based on a rotation angle of the neck. FIG. 7 is a diagram illustrating an example of a change in the tempo of a model video when the model video is made to follow the tempo of a practitioner. 12 is a flowchart illustrating an example of a process for causing a model video to follow the tempo of a practitioner. It is a diagram showing an example of the configuration of an HMD system. It is a flowchart for explaining an example of processing of an HMD system. It is a figure showing an example of a display of HMD.

Examples of embodiments of the present invention will be described below with reference to the drawings. The same components are denoted by the same reference numerals throughout all the figures, and redundant explanation may be omitted. According to the first embodiment, when a self-image showing a user's action and a model video showing a model action are superimposed and displayed as augmented reality, the viewpoints of the self-image and the model video are An HMD 1 that can be easily changed and displayed is provided. Therefore, it is expected that the user wearing the HMD 1 can efficiently learn the movements of the model video by moving the user's own image to match the model video while appropriately changing the viewpoint.

<First embodiment>
A first embodiment will be described with reference to FIGS. 1 to 6. First, an example of the hardware configuration of an HMD will be described with reference to FIG. As shown in FIG. 1, the HMD 1 includes a main control section 11, a storage section 21, a sensor section 31, a communication processing section 41, a video processing section 51, an audio processing section 61, an operation input section 71, Equipped with These components are connected via a data bus for exchanging each data. The HMD 1 also includes a battery (not shown) that serves as a power source.

The main control unit 11 (processor) functions as a main processor, and is configured using, for example, a CPU (Central Processing Unit). Note that the main control unit 11 may be a main body that executes predetermined processing, and may be configured using another semiconductor device such as a GPU (Graphics Processing Unit), for example.

The storage unit 21 is used to store data, and the storage unit 21 includes a program storage unit 22, a data storage unit 23, and a program function unit 24. The program storage section 22 and the data storage section 23 can be configured using, for example, a ROM (Read Only Memory), a flash memory that stores initial setting information, and the like. The program storage unit 22 stores programs used for processing of the HMD 1, and the data storage unit 23 stores data used for processing of the HMD 1. The program function section 24 is configured using a RAM (Random Access Memory), and the main control section 11 loads a program into the program function section 24 and executes it.

The sensor unit 31 includes, for example, a GPS receiving sensor (GPS receiving unit 32) that can be used to acquire position information, a geomagnetic sensor 33, an acceleration sensor 34, a gyro sensor 35, and a sensor that can detect the distance to an object. It can be configured using a distance sensor 36, a human sensor 37, etc., and can be used to grasp data such as the condition of the wearer and the positions of surrounding objects. However, the sensors listed here are just examples, and only need to be able to execute a predetermined process, and the listed sensors may be omitted as appropriate, or other types of sensors may be included.

The communication processing unit 41 is configured to include an interface used for communication, and includes, as an example, a LAN communication unit 42 and a telephone network communication unit 43. The LAN communication unit 42 can be configured using, for example, a wireless LAN interface that is an interface for wireless LAN communication. The telephone network communication unit 43 can be configured using, for example, a telephone network interface that is an interface with a telephone network. However, the configuration of the communication processing unit 41 described here is only an example, and it is sufficient that the HMD 1 can execute a predetermined process, and can be changed as appropriate. The communication processing unit 41 may be configured using, for example, a short-range communication interface that is an interface for short-range communication. Furthermore, if communication by telephone is not required, the telephone network communication section 43 may be omitted.

The video processing unit 51 is used for video processing, and includes an imaging unit 52 and a display unit 53 (display). The imaging unit 52 is configured to acquire real images, and is configured by, for example, a camera. The display unit 53 is configured as an appropriate display that outputs video. The display unit 53 can display an image acquired by a camera, for example.

Furthermore, the video processing section 51 includes a virtual mirror image display section 56. The virtual mirror image display section 56 is used to display the own image and the model image in a superimposed manner, and includes an HMD angle detection section 56a, a rotation processing section, and a superimposition display section 56c. The HMD angle detection unit 56a is a program used to detect the angle of the HMD 1 based on data from the sensor unit 31 and the like. The rotation processing unit (3D rotation processing unit 56b) is a program used to perform three-dimensional rotation processing on the self-image and the model image to be displayed in a superimposed manner. The superimposed display section 56c is a program used to display the own image and the model image in a superimposed manner on the display. These programs are stored in the storage section 21 and executed by the main control section 11 as appropriate. Note that the processing of the virtual mirror image display section 56 will be explained in detail later.

The audio processing section 61 is used for inputting and outputting audio, and includes an audio input section 62 and an audio output section 63. The audio input section 62 is used for audio input, and can be configured using a microphone. The microphone can be provided as appropriate so that the wearer's voice can be input when the HMD 1 is worn. The audio output unit 63 is used for audio output, and can be configured using a speaker, for example. For example, the speaker can be provided close to the ear of the wearer when the HMD 1 is worn.

The operation input unit 71 is configured for the user to input operation details, and can be configured as appropriate using buttons, a touch panel display, etc., as an example.

Next, with reference to FIG. 2, the video seen by the user wearing the HMD will be described. FIG. 2A is a diagram for explaining an overview of superimposed display from a perspective above the user. FIG. 2B is a diagram for explaining an overview of superimposed display from behind the user as a viewpoint.

As shown in FIGS. 2A and 2B, the HMD 1 superimposes a self-image and a model image on the display from a viewpoint corresponding to the orientation of the user's head. Therefore, when the user wearing the HMD 1 rotates his or her neck, viewpoints are displayed in a superimposed manner according to the angle of rotation of the neck.

Here, if the user turns his or her head in the left-right direction by rotating his or her head in the left-right direction, or if the user turns his or her head in the front direction, a mirror image display is performed. For example, if it is determined that the user's head is turned to the left, the self-image from the left viewpoint and the model image are displayed superimposed, indicating that the user's head is turned to the right. If it is determined that the user's own image viewed from the right side and the model image are superimposed and displayed. Furthermore, if it is determined that the user's head is directed toward the front, the user's own image viewed from the front and the model image are displayed in a superimposed manner.

Furthermore, when the user turns his or her head in the vertical direction to face upward, the self-image viewed from above (directly above in this example) and the model image are superimposed and displayed.

The HMD 1 acquires the orientation of the user's head when the main control unit 11 executes the HMD angle detection unit 56a. Here, an example of a method for obtaining the orientation of the user's head will be specifically described.

For example, the HMD angle detection unit 56a uses the configuration of the sensor unit 31 to determine the rotation angle of the user's head. The HMD angle detection unit 56a can determine the rotation angle with respect to the front side, for example, by combining the detection results of the geomagnetic sensor 33 that detects direction and the acceleration sensor 34 that detects tilt. Further, the HMD angle detection unit 56a may use, for example, the angular velocity detected by the gyro sensor 35 to determine the rotation angle with respect to the front side.

The HMD angle detection unit 56a may determine the rotation angle of the head based on head tracking, for example, based on data from a tracking camera placed in the usage environment. In this case, the HMD 1 acquires data from the tracking camera by wireless communication (for example, short-range wireless communication), for example.

Note that the method of determining the rotation angle described here is just an example, and other known methods (for example, other known head tracking techniques) may be used.

Next, with reference to FIG. 3, the difference between the superimposed self-image and the model image will be explained. 3A and 3B are diagrams for explaining an example of rotation of an image and an observed image.

As shown in FIGS. 3A and 3B, the user can check the superimposed display of the self-image and the model image from various viewpoints by appropriately rotating the neck and changing the direction of the head. In the example in Figure 3, the user who performs a certain action confirms that the difference between the self-image seen from the front and the model image is small by turning his head to the front, and then turns his head while maintaining his posture. By rotating and turning the head to the left, you can confirm that there is a large difference between the self-image seen from the left and the model image. That is, in the case of this example, as shown in FIG. 3A, the HMD 1 executes the rotation processing unit (3D rotation processing unit 56b), sets the vertical rotation axis, and converts the self-image and the model image into this image. Rotate around the rotation axis to display the self-image viewed from the left side and the model image superimposed. As a result, as shown by the dotted line in Figure 3B, although we could not confirm that the difference was large from the frontal perspective, side observation revealed that the body was leaning forward compared to the model image. Can be easily confirmed.

Therefore, the user can easily check the difference between the model video and the model video from various viewpoints by simply rotating his or her head. Can be learned efficiently.

Next, with reference to FIG. 4, the process performed for superimposed display will be described in detail. FIG. 4 is a flowchart for explaining an example of processing related to superimposed display.

As shown in FIG. 4, a user (practitioner) who starts practicing first makes initial settings to specify a viewpoint angle to be observed and a virtual mirror surface to be displayed as a set (S401). Here, an example of initial settings will be described with reference to FIG. 5.

As shown in FIG. 5A, in the initial settings, the user selects the viewpoint of the object to be mirror image displayed from among the predetermined viewpoint directions. In this example, the front side and both left and right viewpoints are turned ON and are subject to mirror image display, and the left and right rear viewpoints are turned OFF and are not subject to mirror image display. In this example, the predetermined viewpoint directions are shown as front (0°), 90° left, 90° right, 135° left, and 135° right. The angle may be changed as appropriate. Further, although not illustrated, a setting may be made in which the viewpoint from above is the subject of mirror image display.

Further, in the initial settings, the user makes settings to associate the rotation angle of the neck with the angle of the viewpoint for displaying the mirror image. This setting will be explained with reference to FIG. 5B. In this example, in a fixed coordinate system, the rotation angle is 0° when the user is pointing his or her head straight ahead, and a mirror image of the front viewpoint is displayed when the rotation angle is between 60° to the left and 60° to the right. , settings are made to display a mirror image of the left viewpoint when the rotation angle is 60° to the left and 110° left, and to display a mirror image of the right viewpoint when the rotation angle is 60° to the right. ing. When practicing the model video, a mirror image is displayed corresponding to the rotation angle of the user's neck.

In this example, when the rotation angle is between 110° and 150° left, the left rear viewpoint is displayed as a mirror image, and when the rotation angle is between 110° and right 150°, the right rear viewpoint is displayed. Settings are also made to display a mirror image, but settings related to viewpoint directions that are turned OFF and are not subject to mirror image display may be omitted. Furthermore, although omitted in this example, if settings are made such that the viewpoint from above is the subject of mirror image display, settings regarding the direction of the viewpoint from above may be made based on the rotation angle in the vertical direction. .

In addition, in the initial settings, the angle is set so that the model image is displayed superimposed on the self-image from the front view of the user's head (that is, the third-person perspective, which is a view from the front of the user's face). May be done. For example, when the orientation of HMD 1 is 45° left to 45° right, the front viewpoint is set to be displayed, and when the orientation of HMD 1 is 45° left to 135° left, the left viewpoint is set to be displayed. Settings may also be made. Similarly, when the orientation of the HMD 1 is between 45° to the right and 135° to the right, settings may be made to display the right viewpoint. Further, the number of viewpoints that can be the target may be taken into consideration. , the angle of the viewpoint from which the user's head is viewed from the front may be set. That is, by narrowing the angular range and increasing the number of possible viewpoints, the viewpoint for viewing the user's head from the front may be more appropriately set.

Incidentally, when rotating the neck, there may be cases where the user cannot easily switch to the desired viewpoint direction. Therefore, as shown in FIG. 5C, when the orientation angle of the HMD 1 is θ1 and the angle of the viewpoint for mirror image display is θ2, settings may be made based on the relationship θ1×α=θ2. Here, α can be an appropriate value (coefficient) larger than 1. By setting such a relationship, it is possible to suppress the rotation angle of the user's neck when switching to a desired viewpoint direction, and it is possible to easily switch the viewpoint direction.

The data initialized in S401 is stored in the storage unit 21. The main control unit 11 then performs processing by referring to the initialized data.

When the initial settings are completed and the user starts practicing (S402), the HMD 1 (specifically, the main control unit 11) plays back the model video and first compares it to the self-video from the front viewpoint in a fixed coordinate system. The model video is displayed in a superimposed manner on the display unit 53 (S403).

Then, the main control unit 11 detects the orientation of the user's head (in other words, the rotation angle of the neck) using the HMD angle detection unit 56a (S404). The main control unit 11 also determines whether the detection is due to an original operation (S405). That is, the main control unit 11 determines whether or not a predetermined operation is performed.

Specifically, in S405, the main control unit 11 determines whether the user's action detected in S404 is related to the action registered in the storage unit 21 in advance. With this, for example, when registering a gesture as an operation command in advance, the operation command and the original motion can be distinguished and processed. In addition, in making this determination, the main control unit 11 can use, for example, data of the self-image. In S405, if it is determined that the user's motion is not the original motion (that is, if it is determined that the user's motion is an operation command), the process returns to S404 (S405-N). On the other hand, if the main control unit 11 determines that the user's motion is the original motion (that is, determines that the user's motion is not an operation command), it performs the process of S406 (S405-Y). .

The main control unit 11 executes the rotation processing unit 56b to perform rotation processing on the superimposed image, and displays a mirror image in the set direction (S406). That is, the main control unit 11 rotates the self-image and the model image to perform superimposed display of the viewpoint direction corresponding to the orientation of the user's head. At this time, the main control unit 11 performs the process by referring to the data initialized in S401. During the user's practice, the HMD 1 repeats the processes from S404 to S406 in real time to perform superimposed display in an appropriate viewpoint direction desired by the user.

Note that, as an example, the model video can be stored in the storage unit 21 in advance, and the main control unit 11 can perform processing using the model video stored in the storage unit 21. The model video may be stored in the storage unit 21 at the initial setting timing, for example. On the other hand, the main control unit 11 may acquire the model video via the communication processing unit 41 and process it.

For example, the self-image is generated based on data from a sensing device worn by the user or a camera that photographs the user. Here, the photographing camera is appropriately placed in the environment in which the user practices. For example, the HMD 1 can use the communication processing unit 41 to acquire data acquired by a sensing device or a photographing camera through wireless communication such as short-range wireless communication. Note that the type of sensing device includes a known sensor that can measure a user's motion.

The self-image and the model image can be three-dimensional skeleton images, and can be generated using an appropriate program. This skeletal image can be generated, for example, by representing joints with nodes and constructing edges between the nodes. On the other hand, the self-image and the model image may be real images. Further, superimposed display may be performed in which one video is a real video and the other video is a skeleton video.

Additionally, the HMD 1 may perform a process of estimating the entire skeletal image by performing appropriate estimation process based on data from a sensing device or a photographing camera. For example, in a case where the user's movements are captured in a limited manner due to the arrangement of sensing devices and shooting cameras, the HMD 1 generates an entire skeletal image through estimation processing based on the acquired data, and displays the generated skeletal image. It's okay.

Further, horizontal reversal processing may be performed on the own image and the model image that are displayed in a superimposed manner.

As described above, according to the present embodiment, the HMD 1 superimposes and displays a self-image showing the user's movement and a model image showing the model movement, An HMD 1 is provided that superimposes and displays a self-image and a model image from a viewpoint corresponding to the rotation angle of the neck on a display.

(1) Furthermore, the HMD 1 of this embodiment may be configured to be able to switch between a predetermined viewpoint display mode and a tracking display mode. Here, the predetermined viewpoint display mode is a display mode in which a self-image and a model image from a predetermined viewpoint are displayed in a superimposed manner according to the rotation angle of the user's neck, as described above. Therefore, for example, as described above, if the front side and both left and right viewpoints are set to ON in the initial settings in S401, in the predetermined viewpoint display mode, the , a mirror image display of any one of the front-side viewpoint, left-side viewpoint, and right-side viewpoint is performed.

On the other hand, the tracking display mode is a display mode that performs superimposed display in which the viewpoints of the self-image and the model image are continuously changed in accordance with the rotation of the user's neck. Therefore, in the follow-up display mode, regardless of the contents of the initial setting in S401, a mirror image display that follows the rotation angle of the user's neck is performed, and a superimposed display of a viewpoint viewing the user's head from the front is performed. For example, when a user rotates his or her head from the front side to the left, a mirror image is displayed from a viewpoint that follows the angle of rotation of the user's neck, regardless of the initial settings.

Note that switching between the predetermined viewpoint display mode and the tracking display mode can be performed using an appropriate method. The HMD 1 may be configured to be switchable between a predetermined viewpoint display mode and a tracking display mode, for example, by a user operation via the operation input unit 71. Furthermore, the HMD 1 may be configured to switch between the predetermined viewpoint display mode and the tracking display mode by recognizing a predetermined gesture for switching the display mode. Here, information regarding gestures recognized as operation commands is stored in advance in the storage unit 21, as an example. For example, when switching the display mode, the HMD 1 uses information about the self-image obtained from the sensing device or the camera and the information stored in the storage unit 21 to execute a gesture that is an operation command. recognize.

(2) Furthermore, the HMD 1 of this embodiment may be configured to be able to switch between a playback display mode and a fixed display mode. Here, the reproduction display mode is a display mode in which a model video is reproduced and displayed. On the other hand, the fixed display mode is a mode in which the model video is displayed in a fixed manner. While practicing in the playback display mode, the user can switch to the fixed display mode at an appropriate time (for example, when performing an action that he/she feels unfamiliar with), fix the model image, and then rotate his or her head to take the appropriate viewpoint. By displaying the image in a superimposed manner, it is possible to easily check the difference from the model image at this timing. Note that switching between the playback display mode and the fixed display mode may be performed by using the operation input section 71 or by using gestures, as in the above description.

(3) Furthermore, the HMD 1 of this embodiment may perform superimposed display regarding a plurality of viewpoints simultaneously. As an example, a symmetrical display may be performed; for example, when the user rotates his or her head to the left, in addition to displaying a display corresponding to the rotation of the user's neck, if the user rotates the head by the same angle to the right. A display may be performed when the image is rotated. That is, when the user rotates his or her head in one direction in the left or right direction, the HMD 1 superimposes and displays the self-image and the model image from the viewpoint corresponding to that direction on the display, and furthermore, when the user rotates his or her head in the other direction by the same amount. A process may be performed in which the self-image and the model image from viewpoints in opposite directions are displayed in a superimposed manner on the display when it is assumed that the image has been rotated. Note that the positions where each superimposed display is performed are different positions on the display.

In addition, as a superimposed display regarding multiple viewpoints, the HMD 1 superimposes and displays a self-image from a front viewpoint and a model video on the display, and further displays a self-image and a model video from a rear viewpoint. It may be displayed superimposed on top. Here, each superimposed display is placed at a different position on the display.

In this way, by performing bilaterally symmetrical superimposed display or superimposed display of the front and rear viewpoints, it is possible to simultaneously check the difference between the self-image and the model image from each viewpoint. Note that the display mode in which superimposed display regarding one viewpoint is performed and the display mode in which superimposed display regarding a plurality of viewpoints is performed simultaneously may be switchable. Here, the HMD 1 may be configured to be able to switch between the respective display modes using the same operation input unit or gesture as described above.

(4) The HMD 1 of the present embodiment may be configured to be able to set an area to be displayed in a superimposed manner on the display, and the area to be displayed in a superimposed manner is set by a user operation using the operation input unit 71 at the time of initial setting, for example. It's okay. Here, as an example, an area for displaying a superimposed viewpoint on the left side of the front may be set on the left side of the display, and an area for superimposing displaying a viewpoint on the right side of the front may be set on the right side of the display. Further, for example, if the HMD 1 is of a glasses type, respective areas may be set for the left and right displays. Further, areas for superimposing display of the front viewpoint and upper viewpoint may be similarly set on the display, and these areas are set at the same positions as areas for superimposing display of the left and right viewpoints, for example. It's okay.

(5) Furthermore, according to the HMD 1 of this embodiment, it is also possible to change the tempo (BPM) of a part of the model video and play it back. Next, changing the tempo will be explained.

When a user practices along with a model video, it is thought that by playing the model video at a slower tempo for parts that are difficult to perform or parts that the user thinks they have not mastered, the user will be able to learn more quickly. It will be done. Therefore, the HMD 1 may be configured to be able to set a portion of the model video to be played back at a slower tempo.

Note that the portion to be played back at a slower tempo may be set as appropriate, but as an example, it may be set using the operation input unit 71 or gestures, similar to the above explanation. Furthermore, the HMD 1 may be configured to automatically set the portion where the tempo is slowed down using an evaluation model that evaluates the degree of learning of the model image. The evaluation model can be generated by machine learning using the content practiced by the user and the content of the model, for example, as a model for evaluating whether the content of the model is reproduced. . Further, the portion to be played back at a slower tempo may be set, for example, at the initial setting timing in S401, or may be set during practice.

Furthermore, as shown in FIG. 6A, the model video may follow the tempo of the practitioner (user). That is, the model image may be played back in accordance with the tempo of the user's movements. In the case of FIG. 6A, since the user stands still in the part indicated by A to check his posture, the tempo of the model video in this part slows down to follow the user's tempo. Further, the portions indicated by B are easy portions or previously learned portions, and as the user practices at a faster tempo, the tempo of the model video follows suit and becomes faster. Further, the part indicated by C is a complicated part, and as the user practices in slow motion, the tempo of the model video follows and slows down. Here, in the parts indicated by A and C, the user can practice efficiently by rotating his or her head and checking the superimposed display from a viewpoint where it is easy to see the difference from the model video. can. In this way, by making the model video follow the user's tempo, it is possible to practice efficiently. Next, this process will be explained in detail with reference to FIG. 6B.

First, the HMD 1 (specifically, the main control unit 11) determines whether the practitioner has started a practice movement (S601). Here, the HMD 1 determines, for example, the start of the practice movement of the practitioner by comparing the movement with the movement in the model image (that is, by determining whether the movement matches the movement in the model image). do. When the HMD 1 detects the start of a practice motion, the HMD 1 detects the start position of the practitioner (that is, the position corresponding to the start of practice in the model video) and the tempo of the practitioner. Furthermore, the HMD 1 activates the model video and outputs the model video from the position corresponding to the start of practice (S602).

The HMD 1 monitors the tempo of the practitioner who is practicing (S603). Then, the HMD 1 determines a change in the tempo of the practitioner (S604), and when the tempo of the practitioner changes, changes the tempo of the model video to match the tempo of the practitioner (S605). By repeating the processes of S603 to S605, the HMD 1 causes the tempo of the model video to appropriately follow the tempo of the practitioner.

Note that playback of a model video without changing the tempo, playback of a model video that changes the tempo of a set portion, and playback of a model video that follows the tempo of the practitioner can be switched as appropriate. For example, at the start of practice, the HMD 1 can be configured to be switchable by using the operation input unit 71 or by making a gesture, as described above.

(6) In addition, in the HMD 1 of the present embodiment, when the difference between the self-image and the model image that are superimposed and displayed is equal to or greater than a threshold, a difference occurs between the self-image and the model image from the viewpoint that is superimposed and displayed. You may also perform processing to notify the user of the current status. By recognizing this notification, the user can easily confirm that the difference from the model video is large. Here, the manner of notification is not particularly limited, and as an example, notification may be performed by audio output using an audio output unit. Further, a notification to that effect may be displayed on the display.

<Second embodiment>
Next, a second embodiment will be described with reference to FIGS. 7-8. Functions similar to those in other embodiments are denoted by the same reference numerals, and descriptions similar to those in other embodiments may be omitted. In the second embodiment, a head mounted display system (sometimes referred to as an HMD system 703) including a distribution device 701 and an HMD 702 will be described.

The distribution device 701 is a device that distributes a model video. As shown in FIG. 7, in this embodiment, the distribution device 701 is a server, and the server acquires a model video shot at a remote location via a network (NW in FIG. 7). Then, the server performs a service of distributing the model video to the user side, and the HMD 702 displays the model video obtained from the server.

In this embodiment, the HMD 702 may be configured to have the same functions as in the first embodiment, and can perform the same superimposed display as in the first embodiment. Further, the HMD 702 can perform superimposed display of self-images and model images from various viewpoints in the same manner as in the first embodiment.

In this embodiment, a smartphone (abbreviation for smartphone) acquires a model video distributed from a distribution device 701, and the HMD 702 acquires the model video via the smartphone. Furthermore, in the present embodiment, the HMD 702 can generate a self-image of the user by using the camera of the smartphone as a camera that photographs the user's movements.

The HMD system 703 can be changed as appropriate. In the above description, an example has been described in which the server is the distribution device 701, but an HMD system 703 in which the distribution device 701 is a smartphone may also be provided. That is, the smartphone may acquire the model video via appropriate communication and distribute the model video. Further, an HMD system 703 may be provided in which the smartphone stores a model video and the HMD 702 acquires the model video stored in the smartphone. Furthermore, an HMD system 703 may be provided in which a model video is directly delivered to the HMD 702 from a delivery device 701 (for example, a server) without using a smartphone.

Alternatively, the distribution device 701 may acquire the own video and the model video, generate data to be displayed in a superimposed manner, and distribute the generated data. The HMD 702 may perform superimposed display based on this data. In the example of FIG. 8, in response to the start of practice, the distribution device 701 (in this example, the server) converts the self-video and the model video into 3D, and displays the data to be superimposed (the self-video and the model video). superimposed data) is generated (S801). Then, the generated data is continuously transferred to the HMD 702 (S802). On the HMD 702 side, the same processing as described above (that is, the content explained using FIG. 4) is performed (S803 to S806).

Although a server and a smartphone have been described as examples of the distribution device 701, any other appropriate device may be used as the distribution device 701.

Although the embodiments of the present invention have been described above, it goes without saying that the configuration for realizing the technology of the present invention is not limited to the above embodiments, and various modifications are possible. For example, the embodiments described above have been described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these belong to the scope of the present invention. Further, the numerical values, messages, etc. that appear in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different values are used.

It is sufficient that a predetermined process can be executed. For example, the programs used in each process example may be independent programs, or a plurality of programs may constitute one application program. Furthermore, the order in which each process is performed may be changed.

Some or all of the functions of the present invention described above may be realized by hardware, for example, by designing an integrated circuit. Alternatively, the functions may be realized in software by having a microprocessor unit, CPU, etc. interpret and execute operating programs for realizing the respective functions. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together. Moreover, a part or all of each function may be realized by a server. Note that the server only needs to be able to execute functions in cooperation with other components via communication, and may be, for example, a local server, a cloud server, an edge server, a network service, etc., and its form does not matter. Information such as programs, tables, files, etc. that realize each function may be stored in a memory, a recording device such as a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. However, it may also be stored in a device on a communication network.

Appropriate information indicating the viewpoint to be superimposed may be displayed on the display. For example, when superimposing a viewpoint regarding 90 degrees to the left, text information such as "90 degrees to the left" may be displayed on the display together with this superimposed display.

The HMD (1, 702) has a display mode in which the user's own image and a model image from a viewpoint with the largest difference are superimposed and displayed regardless of the rotation of the user's neck, and a display mode in which the user's neck is The display mode may be configured to be able to switch between a display mode in which the self-image and the model image are superimposed and displayed at a viewpoint corresponding to the rotation. Here, the HMD (1, 702) may be configured to be able to switch between these display modes using the operation input unit 71 or gestures, as described above.

The HMD (1, 702) may be of a glasses type or a goggle type. In addition, the HMD (1, 702) can be equipped with a smartphone as appropriate, uses the display screen of the smartphone as a display (display part 53), and has an out-camera of the smartphone (i.e., installed on the opposite side from the display screen). The configuration may also be such that a camera (a camera that can be used) can be used as a camera (the imaging unit 52).

In addition to superimposed display, the HMD (1, 702) may have a configuration that can display on the display an image in which the model image and the own image are linked (synchronized) as shown in FIG. Here, the HMD (1, 702) may be configured to be able to switch between this display and the superimposed display; for example, as explained above, the HMD (1, 702) may be configured to be able to switch between this display and the superimposed display using the operation input unit 71 or gestures. may be done.

Appropriate operations such as switching the display mode may be performed by voice input using the voice input unit 62 and appropriate voice recognition. In this case, the voice data used for voice recognition is stored in advance in the storage section 21, and the main control section 11 performs voice recognition by referring to the voice data stored in advance in the storage section 21.

1 HMD (head mounted display)
11 Main control unit (processor)
53 Display section (display)
56 Virtual mirror image display section 56a HMD angle detection section 56b 3D rotation processing section 56c Superimposition display section

Claims

A head mounted display comprising a processor and a display,
The processor includes:
superimposing and displaying a self-image showing a user's movement and a model image showing a model movement on the display from a viewpoint corresponding to a rotation angle of the neck of the user wearing the head-mounted display on the head;
A head-mounted display characterized by:
The head mounted display according to claim 1,
The processor includes:
When the user rotates his or her head in a left-right direction, the self-image and the model image from a front view of the user's head are displayed in a superimposed manner on the display.
A head-mounted display characterized by:
The head mounted display according to claim 2,
The processor includes:
When the user rotates his or her head in a vertical direction, the self-image and the model image from a viewpoint viewing the user from above are displayed in a superimposed manner on the display.
A head-mounted display characterized by:
The head mounted display according to claim 1,
Equipped with a storage section,
The storage unit includes:
storing data indicating the angle of the viewpoint according to the rotation angle of the user's neck in the left-right direction;
The processor includes:
superimposing and displaying the self-image and the model image at a viewpoint according to the rotation angle of the user's neck with reference to the data;
A head-mounted display characterized by:
The head mounted display according to claim 4,
The said data is
Set based on the relationship θ1×α=θ2, where θ1 is the rotation angle of the user's neck, θ2 is the angle in the direction of the viewpoint, and α is a coefficient larger than 1.
A head-mounted display characterized by:
The head mounted display according to claim 1,
The head mounted display is
a tracking display mode that is a display mode that performs superimposed display in which the viewpoints of the own image and the model image are changed in accordance with the rotation of the user's neck;
It is possible to switch between a predetermined viewpoint display mode, which is a display mode in which the self-image at a predetermined viewpoint and the model image are superimposed and displayed according to the rotation angle of the user's neck;
A head-mounted display characterized by:
The head mounted display according to claim 1,
The processor includes:
Playing a part of the model video at a slower tempo than other parts,
The part where the tempo of the model video is slowed down is as follows:
set by the user, or set by the processor using an evaluation model that evaluates the degree of mastery of the model video;
A head-mounted display characterized by:
The head mounted display according to claim 1,
The processor includes:
reproducing the model image in accordance with the tempo at which the user moves;
A head-mounted display characterized by:
The head mounted display according to claim 1,
The processor includes:
When the user rotates his or her head in one of the left and right directions, the self-image and the model image from a viewpoint corresponding to the direction of rotation are displayed in a superimposed manner on the display as a first superimposed display;
Displaying the self-image and the model image from a viewpoint corresponding to the direction of rotation in a superimposed manner on the display as a second superimposed display, assuming that the user rotates his or her head in the other direction by the same amount;
The processor includes:
performing the first superimposed display and the second superimposed display at different positions on the display;
A head-mounted display characterized by:
The head mounted display according to claim 9,
The processor includes:
a first area in which a superimposed display is displayed when the user rotates his or her head in one of the left and right directions;
a second area for displaying a superimposed image when the user rotates his or her head in the other direction in the left and right directions, is set on the display;
A head-mounted display characterized by:
The head mounted display according to claim 1,
The processor includes:
If the difference between the superimposed self-image and the model image is greater than a threshold, a notification will be sent to the effect that a difference has occurred.
A head-mounted display characterized by:
The head mounted display according to claim 1,
The head mounted display is
a playback display mode that is a display mode in which the model video is played back and displayed;
It is possible to switch between a fixed display mode, which is a display mode in which the model image is displayed in a fixed manner;
A head-mounted display characterized by:
a distribution device;
head mounted display,
Equipped with
The distribution device includes:
Distributing model videos showing model actions,
The head mounted display is
a processor;
display and
Equipped with
The processor includes:
superimposing and displaying a self-image showing the user's actions and the model image distributed from the distribution device on the display at a viewpoint corresponding to a rotation angle of the neck of the user wearing the head-mounted display on the head;
A head-mounted display system characterized by:
A display method for a head-mounted display using a processor, the method comprising:
superimposing and displaying a self-image showing the user's movement and a model image showing the model movement on a display at a viewpoint corresponding to a rotation angle of the neck of the user wearing the head-mounted display on the head;
A display method for a head-mounted display characterized by:
A program that causes a processor to execute the head-mounted display display method according to claim 14.