JP7432275B1 - Video display device, video display method, and program - Google Patents

Abstract

The present invention provides a video display device that can display a self-image that can be compared with a reference video without preparing a shooting environment similar to that of the reference video. A video display device 1 includes a storage unit 11 that stores a reference video that is a video of an imitation target whose motion is to be imitated by a user, and a first self-video that is a video of the user's motion. a skeleton recognition unit 13 that performs skeletal recognition of the user's first self-image, and a skeletal recognition unit 13 that performs skeletal recognition of the second self-image, which is an image of a three-dimensional object that moves according to the user's movements. Using the results of , the relative positional relationship between the reference video camera that shoots the reference video and the imitation target becomes the same as the relative positional relationship between the video viewpoint of the 3D object and the 3D object. The display unit 15 includes a generation unit 14 that generates a reference video and a second self-video. [Selection diagram] Figure 1

Description

The present invention relates to a video display device and the like that displays a video of a motion to be imitated whose motion is to be imitated by a user, and a video corresponding to the user's motion.

Conventionally, in order to learn movements such as those in surgery, learning involves combining and displaying a reference video, which is a video of the movement of the target to be imitated by the learner, and a self-video, which is a video of the learner's movements. Support devices are known (for example, see Patent Document 1). By referring to such a display, the learner can train to perform the same motion as the motion to be imitated.

Japanese Patent Application Publication No. 2014-071443

However, in conventional learning support devices, in order to make the reference video and self-video comparable, it is necessary to ensure that the relative positional relationship between the camera that is shooting and the subject is the same for both videos. was there. For example, if the reference video is a first-person video shot with a head-mounted camera worn by the teacher performing the action to be imitated, the self-video is also shot with a head-mounted camera worn by the learner. It needed to be a first-person video. In this way, the learner also needs to prepare a shooting environment similar to that of the reference video, which poses the problem of cost and time.

The present invention has been made to solve the above problems, and provides a video display device etc. that can display a self-image that can be compared with a reference video without having to prepare a shooting environment similar to that of the reference video. The purpose is to provide.

In order to achieve the above object, a video display device according to one aspect of the present invention includes a storage unit that stores a reference video that is a video of an action to be imitated whose action is to be imitated by a user; an image acquisition unit that acquires a first self-image, a skeleton recognition unit that recognizes the user's skeleton included in the first self-image, and a three-dimensional object that corresponds to the imitation target and that responds to the user's movements. 3. The second self-image, which is an image of a three-dimensional object that moves, is determined using the results of skeleton recognition by the skeleton recognition unit, and the relative positional relationship between the reference image camera that shoots the reference image and the imitation target; A generation unit that generates a video so that the viewpoint of the video of the dimensional object and the relative positional relationship with the three-dimensional object is the same, and a display unit that displays the reference video and the second self-video. The relative positional relationship between the user camera and the imitation target is different from the relative positional relationship between the self-image camera that captures the first self-image and the part of the user corresponding to the imitation target.

With such a configuration, it is possible to display a second self-image that can be compared with the reference image without preparing a shooting environment similar to that of the reference image. For example, it becomes possible to generate and display a second self-image from the user's first-person perspective from a first self-image taken by a camera facing the user, such as a laptop computer, a tablet terminal, or a smartphone.

Further, in the video display device according to one aspect of the present invention, the generation unit may generate a second self-image in which the line of sight direction as a result of skeleton recognition by the skeleton recognition unit is changed by a set angle. .

With such a configuration, for example, it is possible to generate a second self-image from a first-person viewpoint of the user from a first self-image taken by a camera facing the user.

Further, in the video display device according to one aspect of the present invention, the reference video is a video from the viewpoint of the imitator who moves the imitation target, and the generation unit generates a second self-video that is the video from the user's viewpoint. may be generated.

With such a configuration, it is possible to generate a second self-image from a first-person viewpoint, regardless of the relative positional relationship between the user and the self-image camera that captures the first self-image.

Further, in the video display device according to one aspect of the present invention, the imitation target includes an operation target whose shape changes, the first self-image includes the user's hand, and the generation unit A second self-image may be generated that includes a three-dimensional object as an operation target whose shape changes according to the user's hand gesture included in the image.

With such a configuration, the three-dimensional object to be manipulated can be manipulated by gestures without using a controller or the like.

Further, in the video display device according to one aspect of the present invention, the imitation target includes an operation target whose shape changes, and the generation unit further includes a reception unit that receives an instruction from a controller operated by a user. A second self-image including a three-dimensional object of an operation target whose shape changes according to an instruction received by the user may be generated.

With such a configuration, the three-dimensional object to be manipulated can be manipulated by using the controller. Therefore, for example, if the reference image is an image of a surgical robot, the user can operate the three-dimensional object to be operated using a controller similar to the controller used to operate the surgical robot. , the user will be able to manipulate three-dimensional objects in an environment similar to a real-world surgical robot.

Further, in the video display device according to one aspect of the present invention, the display unit may combine the reference video and the second self-video and display the composite image.

With such a configuration, the reference video and the second self-video can be easily compared, and the user can learn to make his/her own actions similar to the actions to be imitated in the reference video. .

Further, in the image display device according to one aspect of the present invention, the self-image camera and the display device on which the reference image and the second self-image are displayed are configured to move from the self-image camera to the object to be photographed in the optical axis direction. The display device may be arranged so that the direction and the direction of the line of sight looking directly at the display device are opposite to each other.

With such a configuration, for example, a second self-image is generated from a first self-image taken by a camera facing the user such as a laptop computer, a tablet terminal, or a smartphone, and the second self-image and the reference The video can now be displayed to the user.

Further, a video display method according to one aspect of the present invention includes: a storage unit that stores a reference video that is a video of an action to be imitated whose action is to be imitated by a user; a video acquisition unit; a skeleton recognition unit; A video display method that is processed using a generation unit and a display unit, the video acquisition unit acquiring a first self-image that is an image of a user's movement; a second self-image, which is a three-dimensional object corresponding to the imitation target and which is an image of a three-dimensional object that moves in response to the user's movements; Using the results of skeleton recognition, the relative positional relationship between the reference video camera that shoots the reference video and the imitation target, and the relative positional relationship between the video viewpoint of the 3D object and the 3D object are determined. and a step in which the display unit displays the reference video and the second self-video, and the display unit displays the relative positional relationship between the reference video camera and the imitation target, and the first self-video. The relative positional relationship between the self-image camera that photographs the self-image and the part of the user corresponding to the imitation target is different.

According to a video display device or the like according to one aspect of the present invention, it becomes possible to display a self-image that can be compared with a reference video without preparing a shooting environment similar to that of the reference video.

A block diagram showing the configuration of a video display device according to an embodiment of the present invention A diagram illustrating an example of use of a computer that implements a video display device according to the embodiment. A diagram showing an example of a reference video in the same embodiment A diagram showing an example of the first self-image and skeleton recognition results in the same embodiment. A diagram showing an example of the result of hand skeleton recognition in the same embodiment. A diagram showing an example of a second self-image in the same embodiment A diagram showing an example of display of first and second self-images in the same embodiment. A diagram illustrating an example of operation of an operation target using a gesture in the same embodiment. A diagram illustrating an example of operation of an operation target using a gesture in the same embodiment. Flowchart showing the operation of the video display device according to the embodiment A block diagram showing another configuration of the video display device according to the embodiment A perspective view showing an example of a controller in the same embodiment A diagram showing an example of the configuration of a computer in the same embodiment.

Hereinafter, a video display device and a video display method according to the present invention will be described using embodiments. Note that in the following embodiments, components and steps denoted by the same reference numerals are the same or equivalent, and a repeated explanation may be omitted. The video display device according to the present embodiment uses a first self-video shot in a different shooting environment from a reference video, which is a video of the motion of the imitation target whose motion is to be imitated by the user. A second self-image of the three-dimensional object is generated so that the relative positional relationship with the dimensional object is similar to the shooting environment of the reference image, and the reference image and the second self-image are displayed.

FIG. 1 is a block diagram showing the configuration of a video display device 1 according to this embodiment. The video display device 1 according to the present embodiment includes a storage section 11, a video acquisition section 12, a skeleton recognition section 13, a generation section 14, and a display section 15. Note that the video display device 1 may be realized by a computer 900, as shown in FIG. 2 and the like, as an example. In this embodiment, this case will mainly be explained.

The storage unit 11 stores a reference video that is a video of an action to be imitated whose action is to be imitated by the user. The user is a learner who learns movements while referring to reference videos. The motions that the user learns may be, for example, the motions of surgery, the motions of work in a factory, the motions of work in nursing care, hotels, etc., the motions of cooking, etc. It may be an action for creating a work such as a craft, it may be an action for sports, it may be an action for calligraphy, or it may be an action for tying a rope. , or other operations. The imitation target may be, for example, a part of the imitator's body, or may be an object that is operated by the imitator. The imitator may be, for example, a teacher of a user who is a learner, and may be someone who is skilled in the movement that the learner is learning. Further, the part of the imitator's body may include, for example, the imitator's hand. Further, the object to be moved by the imitator may be, for example, the hand of a surgical robot or forceps, or it may be a tool owned by the imitator, such as forceps, a scalpel, tweezers, scissors, or a brush. You can. The reference video is usually a video taken by a camera, but may also be a CG (Computer Graphics) video corresponding to the video taken by a camera. The reference video may be, for example, a video from the viewpoint of the imitator who moves the imitation target, that is, a video from the first-person viewpoint of the imitator. In this case, the reference video may be, for example, a video shot with a head-mounted camera worn by the imitator. In this embodiment, as shown in FIG. 3, a case will be mainly described in which a reference image including an imitation target 21, which is a forceps of a surgical robot, is stored in the storage unit 11.

In the storage unit 11, for example, the entire reference video may be stored, or a part of the reference video may be stored. As an example, when the video display device 1 displays a reference video while receiving it from the outside, a portion of the latest received reference video that is part of the reference video is stored in the storage unit 11, and it is readable. The information may be output and displayed, and may also be sequentially overwritten. Information other than the reference video may be stored in the storage unit 11. For example, information on a three-dimensional object may be stored in the storage unit 11, a second self-image generated by the generation unit 14 may be stored in the storage unit 11, and a second self-image acquired by the image acquisition unit 12 may be stored in the storage unit 11. One self-image may be stored in the storage unit 11.

The process by which information is stored in the storage unit 11 does not matter. For example, information may be stored in the storage unit 11 via a recording medium, information transmitted via a communication line or the like may be stored in the storage unit 11, or, Information input via a device such as a camera may be stored in the storage unit 11. The storage unit 11 is preferably implemented by a nonvolatile recording medium, but may also be implemented by a volatile recording medium. The recording medium may be, for example, a semiconductor memory or a magnetic disk.

The video acquisition unit 12 acquires a first self-video that is a video of the user's actions. The image acquisition unit 12 may be, for example, an optical device such as a camera that photographs an image, or may be one that acquires an image photographed by an optical device such as a camera. In this embodiment, a case will be mainly described in which the video acquisition unit 12 receives a video shot by the camera 901, which is a self-video camera. The first self-image may be, for example, an image of the user's hand movement. The first self-image may be, for example, an image of the user's palm, an image that also includes the user's arm from the wrist to the elbow, or an image that also includes the user's arm from the elbow to the shoulder. Furthermore, the image may also include the user's shoulders and torso. When the second self-image is generated by changing the line-of-sight direction by a set angle, the first self-image is preferably photographed in a predetermined manner. For example, the first self-image may be photographed with the self-image camera facing the user. Further, when a second self-image from the user's viewpoint is generated, a first self-image that shows the direction of the user's line of sight, for example, a first self-image that also includes the user's head is acquired. It is preferable that

In addition, the relative positional relationship between the reference video camera that shoots the reference video and the imitation target, and the relative position between the self-image camera that shoots the first self-video and the part of the user corresponding to the imitation target. It shall be different from the relationship. The part of the user that corresponds to the imitation target is not particularly limited, but may be, for example, the user's palm or the part of the user's hand from the elbow. As an example, the reference video may be a first-person perspective video of the person to be imitated, and the first self-video may be a video captured by a camera 901 facing the user 30, as shown in FIG. . In this embodiment, this case will mainly be explained. Furthermore, in this embodiment, as an example, as shown in FIG. When the display device 902 is arranged so that the direction toward the user 30 and the direction of the user's line of sight looking straight at the display device 902 are opposite, that is, a laptop computer with a camera, a tablet terminal with a camera, a camera The following will mainly describe cases in which a first self-image is photographed, a reference image and a second self-image are displayed, etc. using a smartphone with an attached camera. Note that the camera 901 may be, for example, a built-in camera of the computer 900. The built-in camera of the computer 900 may be, for example, a facing camera of a notebook computer, a tablet terminal, or an in-camera of a smartphone.

The skeleton recognition unit 13 recognizes the user's skeleton included in the first self-image. For example, the skeleton recognition unit 13 may detect a person or a part of a person in a certain frame of the first self-image, and perform skeleton recognition on the detected person or part of the person. Further, the skeleton recognition unit 13 may perform the processing on each of a plurality of frames included in the first self-image. That is, the skeleton recognition process may be performed repeatedly. Note that skeleton recognition may be performed for all frames of the first self-image, or skeleton recognition may be performed for discrete frames. The part of the person may be, for example, the upper body of the person, a portion of the arm including the palm of the hand, or a portion of the palm of the person. When the generation unit 14 generates the second self-image from the user's viewpoint, it is preferable that skeleton recognition including the user's head is performed. This skeleton recognition method is already known, and detailed explanation thereof will be omitted. For example, the skeleton recognition unit 13 may identify the user 30 included in the first self-image and recognize the skeleton 31 of the user 30, as shown in FIG. 4, for example. For example, as shown in FIG. 4, the skeleton 31 corresponds to node shapes such as round shapes corresponding to joints, body ends such as fingertips and head, and body parts such as arms that connect them. It may also include a linear link figure. FIG. 5 is a diagram showing the skeleton 31 recognized for the hand 32 of the user 30. In this embodiment, a case will be mainly described in which a three-dimensional object is operated using the skeleton 31 of the hand 32 recognized by the skeleton recognition unit 13.

The generation unit 14 generates a second self-image, which is an image of a three-dimensional object that corresponds to the imitation target and moves in response to the user's movements, using the results of skeleton recognition by the skeleton recognition unit 13. do. The three-dimensional object corresponding to the imitation target is preferably a three-dimensional object with a shape similar to the imitation target, but for example, a three-dimensional object with a shape that is a simplified version of the imitation target may also be used. good. The position and orientation of the three-dimensional object included in the second self-image may be, for example, the position and orientation according to the recognition result of the skeleton of the part of the user that corresponds to the imitation target; It may move in response to changes in time series of the recognition results of the skeleton of the user's part. In addition, the generation unit 14 makes sure that the relative positional relationship between the reference video camera that shoots the reference video and the imitation target is the same as the relative positional relationship between the viewpoint of the video of the three-dimensional object and the three-dimensional object. Assume that the second self-image is generated so that The viewpoint of an image of a three-dimensional object may be, for example, a viewpoint when rendering a three-dimensional object placed in a three-dimensional virtual space into a two-dimensional image. In addition, the relative positional relationship between the direction of the optical axis of the reference video camera and the imitation target is the same as the relative positional relationship between the direction of the line of sight of the image of the three-dimensional object and the three-dimensional object. Two self-images may be generated. By doing so, the first self-image can be converted into the second self-image shot in the same shooting environment as the reference image. For example, if the reference video is a first-person viewpoint video, a second self-video that is a first-person viewpoint video is created from a first self-video shot by the camera 901 facing the user 30 as shown in FIG. can be generated. Note that the viewpoints and directions of the line of sight are the same, for example, they may be strictly the same, or they may be the same within a predetermined error range.

The results of skeletal recognition show the skeletal structure recognized in a two-dimensional image, but the length of the human body is measured from the shoulder to the elbow, from the elbow to the wrist, and from the wrist to the base of each finger. The length of each part is roughly determined, such as the length of the head, the length of each finger, and the width of the hand, and the range of motion of each joint is also determined. Therefore, by considering them, the generation unit 14 estimates the three-dimensional position and orientation of the user's 30 skeleton included in the first self-image based on the result of skeleton recognition by the skeleton recognition unit 13. Can be done. For example, the generation unit 14 may specify the position and optical axis direction of the self-image camera that captured the first self-image and the user's skeleton 31 in the three-dimensional space based on the skeleton recognition result. good.

For example, in the skeleton 31 of the hand 32 shown in FIG. 5, even if the hand 32 rotates in the direction of the arrow B1, the length of the double arrow A1 does not change, and even if the hand 32 rotates in the direction of the arrow B2, the length of the double arrow A1 does not change. , the length of the double-headed arrow A2 does not change. Note that the double-headed arrow A1 is a double-headed arrow whose end points are the joint at the base of the index finger and the joint at the base of the little finger, and the double-headed arrow A2 is a double-headed arrow whose end points are the joint at the wrist and the joint at the base of the middle finger. It is a double-headed arrow. Therefore, in the skeleton recognition results, the generation unit 14 determines the angle of the hand in the directions of the arrows B1 and B2 and the change in the angle according to the lengths of the double arrows A1 and A2 of the hand 32 and changes in the lengths. You can know.

The direction of rotation in the direction of arrow B1 may be determined based on, for example, a change in the distance between the joints of each finger. For example, when rotating in the direction of arrow B1, if the distance between the joints on the side closer to the wrist increases, but the distance between the joints on the side closer to the fingertips decreases, the wrist side becomes the camera. If the distance between the joints on the side closer to the wrist becomes smaller, whereas the distance between the joints on the side closer to the fingertips increases, the fingertips will move closer to the camera. It may be determined that the object has rotated in the direction. The determination may be made in the same manner for the blade rotated to either side in the direction of arrow B2. For example, when rotating in the direction of arrow B2, the distance between the joints of the little finger becomes larger, but if the distance between the joints of the thumb becomes smaller, the little finger side rotates in the direction closer to the camera. If it is determined that the distance between the joints of the little finger has become smaller while the distance between the joints of the thumb has become larger, it may be determined that the thumb side has rotated in a direction closer to the camera.

In addition, in the skeleton recognition results shown in FIG. 4, for example, the positions of the shoulders, elbows, and wrists in the plane direction perpendicular to the optical axis direction of the camera are the same as the positions of the shoulders, elbows, and wrists in the first self-image. It can be specified accordingly. Regarding the position of the elbow and wrist relative to the shoulder in the optical axis direction of the camera, for example, in the first self-image, if the length from the shoulder to the elbow is shorter than the original length from the shoulder to the elbow, It can be estimated that the elbow is moving closer to the camera side accordingly. The same applies to the elbow to wrist. In this way, the generation unit 14 may estimate the three-dimensional position and posture of the skeleton of the user's 30 arm and palm.

The generation unit 14 may generate the second self-image using the three-dimensional position and orientation of the skeleton 31 of the user 30 estimated in this way. For example, the generation unit 14 may generate a second self-image in which the line-of-sight direction as a result of skeleton recognition by the skeleton recognition unit 13 is changed by a set angle. Specifically, the generation unit 14 specifies, based on the result of skeleton recognition, a viewpoint and line-of-sight direction corresponding to the position and optical axis direction of the self-image camera, and the user's skeleton 31 in the three-dimensional virtual space. However, the viewpoint and line of sight direction may be changed as determined in advance. In this way, the generation unit 14 may change the line of sight direction and also change the viewpoint. More specifically, the generation unit 14 changes the line of sight direction from the user's 30 hand toward the shoulder to the line of sight direction from the user's 30 shoulder to the hand, as in the first self-image shown in FIG. A second self-image, that is, a second self-image from a first-person perspective may be generated. Note that, since the positional relationship between the viewpoint and the line-of-sight direction and the skeleton 31 is relative, the generation unit 14 changes the skeleton of the user 30 in the three-dimensional virtual space, for example, instead of changing the line-of-sight direction. You may let them. The change in the user's 30 skeleton may be, for example, a rotation of the user's 30 skeleton, and may further include movement. In this way, for example, even if the angle of the skeleton is changed in the three-dimensional virtual space, it may be considered that the viewing direction is changed.

Further, the generation unit 14 may arrange a three-dimensional object based on the position and posture of the user's 30 skeleton in the three-dimensional virtual space, for example. For example, in a three-dimensional virtual space, the generation unit 14 uses the skeleton of the user 30 from the elbow to the hand, and the longitudinal direction is along the direction from the elbow to the wrist of the user 30, and the tip is the point of the palm of the user 30. The three-dimensional object of the forceps may be arranged so as to correspond to the position. Further, the generation unit 14 uses, for example, the skeleton of the palm of the user 30, and the longitudinal direction is along the direction of the arrow A2 in FIG. 5, and the angle around the longitudinal direction is in the direction of the arrow A1 in FIG. The three-dimensional object of the forceps may be arranged so that the position changes accordingly and the tip is located at the tip of the user's 30 middle finger. This three-dimensional object is stored in the storage unit 11, for example, and may be read out and used. The three-dimensional object corresponds to the imitation target, for example, and may be a tool such as forceps or a scalpel, or may include a hand. When the three-dimensional object includes a hand, the generation unit 14 may arrange, in the three-dimensional virtual space, a three-dimensional hand object having a shape corresponding to the hand skeleton indicated by the result of skeleton recognition.

Note that here, a case has been described in which a three-dimensional object is arranged according to the user's 30 skeleton after changing the relative positional relationship between the user's 30's skeleton and the viewpoint or line of sight direction, but the order is reversed. It may be. For example, after arranging a three-dimensional object according to the skeleton of the user 30, the relative positional relationship between the three-dimensional object and the viewpoint or direction of line of sight may be changed.

Furthermore, the generation unit 14 may generate, for example, a second self-image that is an image from the user's 30 viewpoint. In this case, the generation unit 14 identifies the user's skeleton 31 including the position of the head of the user 30 in the three-dimensional virtual space based on the result of skeleton recognition, and determines the position of the head of the user 30. may be taken as a viewpoint, and the direction from the viewpoint toward the hand of the user 30 may be specified as the line-of-sight direction. Then, similarly to the above description, the generation unit 14 may arrange the three-dimensional object in the three-dimensional virtual space based on the position and orientation of the user's 30 skeleton.

For example, a three-dimensional object corresponding to the right hand of the user 30 and a three-dimensional object corresponding to the left hand of the user 30 may be arranged in the three-dimensional virtual space. In this case, a three-dimensional object corresponding to the right hand is placed based on the result of skeleton recognition regarding the right hand of the user 30, and a three-dimensional object corresponding to the left hand is placed based on the result of skeleton recognition regarding the user's left hand. may be done. FIG. 6 is a diagram showing an example of the second self-image including the three-dimensional objects 33a and 33b generated in this way. The three-dimensional objects 33a and 33b may correspond to the right and left hands of the user 30, respectively. Note that the three-dimensional objects 33a and 33b may be referred to as a three-dimensional object 33 when not particularly distinguished. Further, in the second self-image, the area other than the three-dimensional objects 33a and 33b may be transparent, for example.

The generation unit 14 may generate a two-dimensional image by rendering the three-dimensional object 33 based on the viewpoint and line-of-sight direction in the three-dimensional virtual space. This two-dimensional image is a two-dimensional image of the three-dimensional object 33 viewed from the viewpoint in the line-of-sight direction in the three-dimensional virtual space. For example, the generation unit 14 changes the position and orientation of the three-dimensional object 33 in the three-dimensional virtual space according to the results of skeleton recognition repeatedly performed by the skeleton recognition unit 13, and renders the three-dimensional object 33 after the change. The resulting two-dimensional image may be generated repeatedly. The second self-image may be composed of a plurality of two-dimensional images generated in this way, for example.

Note that the size of the three-dimensional object 33 included in the second self-image varies depending on the size of the three-dimensional object in the three-dimensional virtual space, the distance from the viewpoint to the three-dimensional object, the angle of view at the time of rendering, etc. It will be decided. As an example, the size of a 3D object in a 3D virtual space and the angle of view at the time of rendering are predetermined values, and the distance from the viewpoint to the 3D object is determined from the self-image camera. It may be determined according to the distance to the user's part corresponding to . The distance from the self-image camera to the part of the user corresponding to the imitation target may be determined, for example, depending on the size of the part of the user corresponding to the imitation target included in the first self-image. In this case, the user can adjust the size of the three-dimensional object 33 included in the second self-image by, for example, changing the distance between the self-image camera and the user. The user may also be able to adjust predetermined values, for example. Further, as an example, the generation unit 14 generates a three-dimensional object 33 in the three-dimensional virtual space so that the size of the three-dimensional object 33 included in the second self-video is the same as the size of the imitation target included in the reference video. The size of the dimensional object, the angle of view during rendering, etc. may be automatically adjusted. This automatic adjustment may be performed only once, for example, at the start of generation of the second self-image, or may be performed repeatedly during generation of the second automatic image. The size of the three-dimensional object 33 and the size of the imitation target may be, for example, strictly the same or may be the same within a predetermined error range. . Further, the imitation target area included in the reference video may be specified by pattern matching, segmentation, or the like, for example.

The display unit 15 displays the reference video stored in the storage unit 11 and the second self-video generated by the generation unit 14. It is preferable that the display unit 15 displays both images so that they can be compared. The display unit 15 may, for example, combine and display the reference video and the second self-video. In this case, for example, as shown in FIG. 7, the imitation target 21 and the three-dimensional objects 33a and 33b may be displayed together. In addition, in FIG. 7, the three-dimensional objects 33a and 33b are shown with broken lines in order to make it possible to distinguish between the imitation target 21 and the three-dimensional objects 33a and 33b. Further, for example, the second self-image may be synthesized on the near side, that is, on the upper side of the reference image. As described above, the area other than the three-dimensional object 33 of the second self-image to be combined with the reference image may be transparent. Further, the opacity of the area of the three-dimensional object 33 in the second self-image to be combined with the reference image may be, for example, 100% or less than 100%. If the opacity of the second self-image is greater than 0% and less than 100%, that is, if the second self-image is semi-transparent, the user can assume that the three-dimensional object 33 and the imitation target 21 overlap. Even if you do, you will be able to see both.

Note that the display unit 15 may display both videos while switching them in a time-sharing manner, for example. More specifically, the display unit 15 may repeat displaying the reference video for a first time and displaying the second self-video for the next second time. In this case, the display unit 15 may display the reference video divided for each first time in order while interposing the display of the second self-video for the second time. The first and second times are not particularly limited, but may be, for example, times within a range of 0.1 seconds to 1 second. Even in this case, the user will be able to view both videos. Note that the display while switching between the reference image and the second self-image is such that the opacity of the second self-image synthesized in front of the reference image is set to 0% for the first time, and then the second self-image is displayed. It can also be thought of as repeating the display at 100% for the time 2. In this case, in the second self-image, the area other than the three-dimensional object 33 may be opaque (for example, a single color such as white). Further, in this case, instead of switching the opacity between 0% and 100%, the opacity may be changed continuously from 0% to 100%. For example, the opacity may be changed continuously within a range from 0% to 100%, such as a sine wave, a sawtooth wave, a triangular wave, or the like.

Note that the display unit 15 may or may not include a display device (for example, a liquid crystal display, an organic EL display, etc.) that performs these displays. Furthermore, the display target may be displayed in another device. In that case, the display unit 15 may transmit the video to be displayed to the outside of the device. Further, the display unit 15 may be realized by hardware or by software such as a driver that drives a display device.

Further, when the imitation target includes an operation target whose shape changes, the generation unit 14 may generate a second self-image including the three-dimensional object of the operation target, for example. An operation object whose shape changes is one whose shape changes in response to an operation by an operator. This object to be operated has an opening/closing part, such as scissors, forceps, tweezers, tongs, etc., and the shape of the opening/closing part changes between the open state and the closed state. good. In this case, the first self-image may include the hand 32 of the user 30. Then, the generation unit 14 may generate a second self-image including a three-dimensional object of the operation target whose shape changes according to the gesture of the hand 32 of the user 30 included in the first self-image. The gesture may be, for example, the shape of the hand 32 of the user 30, or the movement of the hand 32. The former gesture is static, and the latter gesture is dynamic. For example, the generation unit 14 may identify the gesture of the hand 32 of the user 30 from the first self-image itself, and identify the gesture of the hand 32 of the user 30 using the result of skeleton recognition in the first self-image. may be specified. The dynamic gesture may be identified using, for example, a change in the shape of the hand 32 or a change in the shape of the skeleton 31 in the first self-image.

When the object to be operated is a forceps having an opening/closing part and a static gesture is specified, the generation unit 14 generates a gesture when the hand 32 is open, as shown in FIG. 8A, for example. In particular, when the tip of the index finger and the tip of the thumb of the hand 32 are separated, a second self-image including a three-dimensional object 33 of forceps with an open opening is generated, as shown in FIG. 8B. When the tip of the index finger and the tip of the thumb of the hand 32 are in contact, a second self-image including the three-dimensional object 33 of forceps with the opening/closing part closed may be generated. Note that, depending on the gesture, the opening/closing part of the three-dimensional object 33 of the forceps may change into two states, that is, an open state and a closed state, and the extent to which the opening/closing part is open may also change. It may be changed to include. In the latter case, the generation unit 14 uses, for example, the result of skeleton recognition in the first self-image to create a state in which the tip of the index finger and the tip of the thumb of the hand 32 are close to the open state (FIG. 8A). The extent to which the opening/closing portion of the three-dimensional object 33 is open may be changed according to the obtained degree by obtaining the degree of whether the opening/closing portion of the three-dimensional object 33 is close to the closed state (FIG. 8B). In this case, for example, the degree of opening/closing of the three-dimensional object 33 is adjusted such that the closer the tip of the index finger and the tip of the thumb of the hand 32 are to a closed state, the closer the opening/closing part of the three-dimensional object 33 is to a closed state. May be changed. Further, if the object to be operated is a forceps having an opening/closing part and a dynamic gesture is specified, the generation unit 14 generates, for example, the tip of the index finger and the thumb from a state where the hand 32 is open. When the tip of the forceps changes to the state in which they are in contact with each other, a second self-image is generated that includes the three-dimensional object 33 of the forceps whose opening/closing part changes from the open state to the closed state, and the tip of the index finger of the hand 32 and the second self-image are generated. A second self-image containing the three-dimensional object 33 of the forceps whose opening/closing part changes from a closed state to an open state when the tips of the thumbs change from a state where they are in contact to a state where both tips are separated. may be generated. By doing so, the shape of the three-dimensional object 33 included in the second self-image can be changed using the gesture of the hand 32 of the user 30 without using a controller or the like. Note that the position and posture of the three-dimensional object 33 may be changed according to the result of skeleton recognition by the skeleton recognition unit 13, for example, the position and posture of the hand 32.

Next, the operation of the video display device 1 will be explained using the flowchart of FIG.

(Step S101) The display unit 15 starts displaying the reference video stored in the storage unit 11. After this, the display section 15 continues to display the reference video.

(Step S102) The video acquisition unit 12 determines whether to acquire the first self-video. If the first self-image is to be acquired, the process proceeds to step S103; otherwise, the process of step S102 is repeated until it is determined that the first self-image is to be acquired. Note that the video acquisition unit 12 may, for example, periodically determine to acquire the first self-video.

(Step S103) The video acquisition unit 12 acquires a first self-video. Note that this acquisition of the first self-image may be, for example, acquisition of one frame that constitutes the first self-image. The image acquisition unit 12 may receive the first self-image from the camera 901, for example.

(Step S104) The skeleton recognition unit 13 performs skeleton recognition of the user 30 included in the first self-image. This skeleton recognition may be performed, for example, on one frame that constitutes the first self-image.

(Step S105) The generation unit 14 identifies the gesture of the hand 32 of the user 30 using the first self-image or the result of skeleton recognition. Static gestures may be identified using, for example, one frame or the result of skeleton recognition performed on one frame. Dynamic gestures may be identified using, for example, a plurality of consecutive frames or the results of skeleton recognition performed on a plurality of consecutive frames.

(Step S106) The generation unit 14 generates a second self-image including the three-dimensional object 33 using the skeleton recognition result and the identification result of the gesture of the hand 32 of the user 30. Generation of this second self-image may be, for example, generation of one frame that constitutes the second self-image. Further, the shape of the three-dimensional object 33 included in the second self-image may correspond to the specified gesture of the hand 32 of the user 30. Furthermore, in this second self-image, the relative positional relationship between the reference image camera and the imitation target is the same as the relative positional relationship between the viewpoint of the second self-image and the three-dimensional object 33. It may be generated as follows.

(Step S107) The display unit 15 displays the generated second self-image together with the reference image. For example, the second self-video and the reference video may be combined and displayed. In this way, user 30 is able to view both the reference video and the second self-video. Then, the process returns to step S102.

Note that the order of processing in the flowchart of FIG. 9 is an example, and the order of each step may be changed as long as the same result can be obtained. Further, in the flowchart of FIG. 9, the process is terminated by turning off the power or by an interrupt to terminate the process.

Next, the operation of the video display device 1 according to this embodiment will be explained using a specific example. In this specific example, the reference video is assumed to be a video of a situation where the imitation target 21, which is a forceps, is being operated by a surgical robot, as shown in FIG. Further, as shown in FIG. 2, the video display device 1 is realized by a computer 900 that is a notebook computer, and a camera 901 built in the computer 900 is used to display a first image including the hand 32 of the user 30. A self-video of the applicant shall be taken.

First, assume that the user 30 operates the computer 900 and inputs an instruction to display a reference video and a second self-video. Then, in response to the instruction, the display unit 15 starts displaying the reference video stored in the storage unit 11 on the display device 902 (step S101). The video acquisition unit 12 also acquires the first self-video of the user 30 taken by the camera 901, and passes it to the skeleton recognition unit 13 (steps S102, S103). As shown in FIG. 4, the skeleton recognition unit 13 recognizes the skeleton 31 of the user 30 in the first self-image taken by the camera 901 facing the user 30, and converts the recognized skeleton 31 into a generation unit. 14 (step S104). Upon receiving the recognition result skeleton 31, the generation unit 14 identifies the hand gesture of the user 30 using the shape of the hand portion of the skeleton 31 (step S105). In addition, the generation unit 14 uses the skeleton recognition results and the gesture identification results to create a position and posture that corresponds to the skeleton 31 of the user 30 in the three-dimensional virtual space. A second self-image is generated by arranging shaped three-dimensional objects 33a, 33b and rendering the three-dimensional objects 33a, 33b to be an image from the first-person viewpoint of the user 30, and passing it to the display unit 15. (Step S106). For example, a second self-image as shown in FIG. 6 will be generated. Upon receiving the second self-image, the display unit 15 synthesizes and displays the second self-image with the reference image (step S107). As a result, the user 30 can view the reference video and the second self-video displayed on the display device 902 of the computer 900, as shown in FIG. 2. In this way, by repeatedly acquiring the first self-image, recognizing the skeleton, identifying gestures, generating the second self-image based on them, and displaying the reference image and the second self-image, The user 30 can move the three-dimensional objects 33a and 33b of the forceps according to the movement of his/her hand so as to follow the movement of the forceps included in the reference video. Then, the user 30 can perform training to operate the three-dimensional objects 33a and 33b in the same way as the imitation target included in the reference video.

As described above, the video display device 1 according to the present embodiment generates a second self-image that can be compared with the reference video without preparing a shooting environment similar to that used when the reference video was shot. and display it. Therefore, the user 30 can train to perform the same action as the imitation target while comparing the reference video and the second self-video. For example, even if the reference video is from a first-person perspective, the user 30 can now shoot his or her own video using a face-to-face camera such as a laptop computer, tablet terminal, or smartphone, or a head-mounted camera. Since there is no need to prepare one, there is an advantage that the cost and time required for this can be reduced. Furthermore, when changing the shape of the three-dimensional object 33 as the operation target in accordance with the hand gesture of the user 30, there is no need to use a controller or the like to change the shape of the three-dimensional object 33. , training can be performed with a simple configuration.

Note that in this embodiment, a case has been described in which the shape of the three-dimensional object 33 is changed using the gesture of the hand 32 of the user 30, but this is not necessarily the case. The user 30 may change the shape of the three-dimensional object 33 by operating the controller. In this case, the video display device 1 may further include a reception unit 16 that receives instructions from the controller 4 operated by the user 30, as shown in FIG. For example, the receiving unit 16 may receive instructions from the controller 4 by wire or wirelessly. Note that the reception unit 16 may or may not include a device (for example, a communication device) for performing reception. Further, the reception unit 16 may be realized by hardware or by software such as a driver that drives a predetermined device.

The controller 4 operated by the user 30 may be one shown in FIG. 11, for example. The controller 4 shown in FIG. 11 has a button 4a, and when the user 30 presses the button 4a, for example, an instruction to close the opening/closing part of the three-dimensional object 33 is sent to the video display device 1. May be sent. In this case, the generation unit 14 may generate a second self-image including a three-dimensional object of the operation target whose shape changes according to the instruction received by the reception unit 16, for example. More specifically, when the user 30 presses the button 4a, the generation unit 14 generates a 3-shaped button with the opening/closing part closed in response to an instruction to close the opening/closing part received by the reception unit 16. A second self-image including the dimensional object 33 may be generated. Note that if the user 30 has not pressed the button 4a, a second self-image including the three-dimensional object 33 in the shape of an open opening/closing part may be generated. By doing so, the three-dimensional object 33 of the operation target can be operated using the controller 4. Therefore, for example, when the reference image is an image of a surgical robot, the user 30 operates the three-dimensional object 33 to be operated using the controller 4 that is similar to the controller used to operate the surgical robot. The user 30 can also operate the three-dimensional object 33 in an environment similar to a surgical robot in a real environment.

Further, in the present embodiment, the case where the video display device 1 according to the present embodiment is realized by a notebook computer with a camera, a tablet terminal with a camera, or the like has been mainly described, but this need not be the case. For example, the video display device 1 according to the present embodiment may be realized by a desktop personal computer to which an external camera is connected. In recent years, cameras are often placed around the display of desktop computers to face the user viewing the display for purposes such as web meetings. may be used as a display for displaying the reference image and the second self-image.

Further, in this embodiment, the three-dimensional object may be changed. For example, the imitation target may be changed in the reference video. Specifically, it is conceivable that the imitation target is changed from forceps to a scalpel. In such a case, the three-dimensional object may also be changed in the second self-image in accordance with the change in the imitation target in the reference image. This change may be made manually, for example. When a three-dimensional object is changed manually, for example, the user may be able to change the three-dimensional object by a hand gesture. In this case, the three-dimensional object may be changed, for example, by the user performing a hand-sweeping gesture. Further, the change of the three-dimensional object may be performed automatically. In this case, for example, the generation unit 14 may specify the type of imitation target included in the reference video, and arrange a three-dimensional object according to the specified type in the three-dimensional virtual space. The type of imitation target may be determined by, for example, pattern matching or object recognition.

Further, in the above embodiment, the case where the video display device 1 is a stand-alone device has been described, but the video display device 1 may be a stand-alone device or a server device in a server-client system. In the latter case, the video acquisition section and the display section may acquire the video or display the video via the communication line.

Furthermore, in the above embodiments, each process or each function may be realized by being centrally processed by a single device or a single system, or may be realized by being distributedly processed by multiple devices or multiple systems. This may be realized by

In addition, in the above embodiment, the information exchange performed between each component is performed by one component, for example, when the two components that exchange the information are physically different. This may be done by outputting information and receiving the information by another component, or by one component if the two components passing that information are physically the same. This may be performed by moving from a phase of processing corresponding to the component to a phase of processing corresponding to the other component.

In the above embodiments, information related to processing executed by each component, for example, information accepted, acquired, selected, generated, transmitted, or received by each component. Information such as threshold values, formulas, addresses, etc. used by each component in processing may be held temporarily or for a long period of time in a recording medium (not shown), even if not specified in the above description. Further, the information may be stored in the recording medium (not shown) by each component or by a storage unit (not shown). Further, each component or a reading unit (not shown) may read information from the recording medium (not shown).

In addition, in the above-described embodiment, if the information used in each component, for example, information such as threshold values, addresses, various setting values, etc. used by each component in processing, may be changed by the user, the above-mentioned Even if it is not specified in the description, the user may or may not be able to change the information as appropriate. If the information can be changed by the user, the change is realized by, for example, a reception unit (not shown) that receives change instructions from the user, and a change unit (not shown) that changes the information in accordance with the change instruction. You can. The acceptance of the change instruction by the reception unit (not shown) may be, for example, acceptance from an input device, information transmitted via a communication line, or information read from a predetermined recording medium. .

Further, in the above embodiment, when two or more components included in the video display device 1 have a communication device, an input device, etc., even if the two or more components physically have a single device, or may have separate devices.

Furthermore, in the embodiments described above, each component may be configured by dedicated hardware, or components that can be realized by software may be realized by executing a program. For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. At the time of execution, the program execution section may execute the program while accessing the storage section or recording medium. Note that the software that implements the video display device 1 in the above embodiment is the following program. In other words, this program causes a computer that has access to a storage unit that stores a reference image, which is an image of the movement of the imitation target whose movement is to be imitated, to the first self-image, which is the image of the user's movement. a skeletal recognition unit that recognizes the user's skeleton included in the first self-image, a 3D object corresponding to the imitation target, and an image of the 3D object that moves in response to the user's movements. The second self-image is determined using the results of skeleton recognition by the skeleton recognition unit, the relative positional relationship between the reference image camera that shoots the reference image and the imitation target, the viewpoint of the image of the 3D object, and the 3D image. It functions as a generation unit that generates images so that the relative positional relationship with the object is the same, and a display unit that displays the reference video and the second self-image, and generates images that have the same relative positional relationship with the object, and functions as a display unit that displays the reference video camera and the imitation target. The relationship and the relative positional relationship between the self-image camera that captures the first self-image and the portion of the user corresponding to the imitation target are different programs.

Note that in the above program, the functions realized by the program do not include functions that can only be realized by hardware. For example, functions that can only be realized by hardware such as an interface card in an acquisition unit that acquires information, a display unit that displays information, etc. are not included in the functions that are realized by the program.

Further, this program may be executed by being downloaded from a server or the like, and the program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, etc.) is read out. It may be executed by Further, this program may be used as a program constituting a program product.

Further, the number of computers that execute this program may be one or more. That is, centralized processing or distributed processing may be performed.

FIG. 12 is a diagram showing an example of the configuration of a computer 900 that executes the above program to realize the video display device 1 according to the above embodiment. In FIG. 12, a computer 900 stores a camera 901, a display device 902, a keyboard 903, a pointing device 904 such as a touch pad or a mouse, an MPU (Micro Processing Unit) 911, and programs such as a boot-up program. a ROM 912 connected to the MPU 911 and temporarily storing instructions of application programs and providing temporary storage space; a storage unit 914 storing application programs, system programs, and data; and a LAN or WAN and a communication module 915 that provides connection to the network. Note that the MPU 911, ROM 912, etc. may be connected to each other via a bus. Further, the storage unit 914 may be, for example, a hard disk or an SSD (Solid State Drive). Further, the camera 901, display device 902, keyboard 903, pointing device 904, etc. may be, for example, devices built into the computer 900, or may be external devices.

A program that causes the computer 900 to execute the functions of the video display device 1 according to the embodiment described above may be loaded into the RAM 913 at the time of execution. Note that the program may be loaded directly from the storage unit 914 or the network, for example.

The program does not necessarily need to include an operating system (OS) that causes computer 900 to execute the functions of video display device 1 according to the above embodiment, a third party program, or the like. A program may include only those portions of instructions that call appropriate functions or modules in a controlled manner to achieve desired results. How computer 900 operates is well known and detailed explanation will be omitted.

Further, the above embodiments are illustrative examples for concretely implementing the present invention, and do not limit the technical scope of the present invention. The technical scope of the present invention is indicated by the claims, not the description of the embodiments, and includes changes within the literal scope and equivalent meaning of the claims. is intended.

1 Video Display Device 11 Storage Unit 12 Video Acquisition Unit 13 Skeleton Recognition Unit 14 Generation Unit 15 Display Unit 16 Reception Unit

Claims (9)

a storage unit that stores a reference video that is a video of an action to be imitated whose action is to be imitated by the user;
a video acquisition unit that acquires a first self-video that is a video of the user's actions;
a skeleton recognition unit that recognizes the user's skeleton included in the first self-image;
A second self-image, which is an image of a three-dimensional object that corresponds to the imitation target and moves in response to the user's movements, is converted into the reference image using the result of skeleton recognition by the skeleton recognition unit. generation so that the relative positional relationship between a reference video camera that shoots the image and the imitation target is the same as the relative positional relationship between the viewpoint of the video of the three-dimensional object and the three-dimensional object; Department and
a display unit that displays the reference video and the second self-video;
the first self-image is acquired while displaying the reference image;
The reference video is a video taken by a camera or a CG video corresponding to a video taken by a camera,
a relative positional relationship between the reference video camera and the imitation target; a relative positional relationship between the self-image camera that captures the first self-image and a portion of the user corresponding to the imitation target; is a different video display device. The image display device according to claim 1, wherein the generation unit generates a second self-image in which the line of sight direction as a result of skeleton recognition by the skeleton recognition unit is changed by a set angle. The reference video is a video from the perspective of an imitator who moves the imitation target,
The video display device according to claim 1, wherein the generation unit generates a second self-video that is a video from the user's viewpoint. The imitation target includes an operation target whose shape changes,
The first self-image includes the user's hand,
The generation unit generates a second self-image including a three-dimensional object of an operation target whose shape changes according to a hand gesture of the user included in the first self-image. Video display device. The imitation target includes an operation target whose shape changes,
further comprising a reception unit that receives instructions from a controller operated by the user,
The video display device according to claim 1, wherein the generation unit generates a second self-image including a three-dimensional object of an operation target whose shape changes according to an instruction received by the reception unit. 6. The video display device according to claim 1, wherein the display section combines and displays the reference video and the second self-video. The self-image camera and the display device on which the reference image and the second self-image are displayed are arranged such that the self-image camera faces the object to be photographed in the optical axis direction, and the display device faces directly. The video display device according to any one of claims 1 to 5, wherein the video display device is arranged so that the direction of line of sight is opposite to that of the video display device. Processed using a storage unit in which a reference video, which is a video of an imitation target motion whose motion is to be imitated by a user, is stored, a video acquisition unit, a skeleton recognition unit, a generation unit, and a display unit. A video display method,
a step in which the video acquisition unit acquires a first self-video that is a video of the user's actions;
a step in which the skeleton recognition unit recognizes the skeleton of the user included in the first self-image;
The generation unit generates a second self-image, which is an image of a three-dimensional object that corresponds to the imitation target and moves according to the user's motion, using the result of the skeleton recognition, The relative positional relationship between the reference video camera that shoots the video and the imitation target is the same as the relative positional relationship between the viewpoint of the video of the three-dimensional object and the three-dimensional object. step and
The display unit displays the reference video and the second self-video,
the first self-image is acquired while displaying the reference image;
The reference video is a video taken by a camera or a CG video corresponding to a video taken by a camera,
a relative positional relationship between the reference video camera and the imitation target; a relative positional relationship between the self-image camera that captures the first self-image and a portion of the user corresponding to the imitation target; The image display method is different. A computer that can access a storage unit that stores a reference video that is a video of the motion of the imitation target whose motion is to be imitated by the user,
a video acquisition unit that acquires a first self-video that is a video of the user's actions;
a skeleton recognition unit that recognizes the user's skeleton included in the first self-image;
A second self-image, which is an image of a three-dimensional object that corresponds to the imitation target and moves in response to the user's movements, is converted into the reference image using the result of skeleton recognition by the skeleton recognition unit. generation so that the relative positional relationship between a reference video camera that shoots the image and the imitation target is the same as the relative positional relationship between the viewpoint of the video of the three-dimensional object and the three-dimensional object; Department,
functioning as a display unit that displays the reference video and the second self-video;
the first self-image is acquired while displaying the reference image;
The reference video is a video taken by a camera or a CG video corresponding to a video taken by a camera,
a relative positional relationship between the reference video camera and the imitation target; a relative positional relationship between the self-image camera that captures the first self-image and a portion of the user corresponding to the imitation target; is a different program.

Images