JP7412826B1 - Video compositing device, video compositing method, and program - Google Patents

Abstract

An object of the present invention is to provide a video synthesis device that can prevent a reference video and a self-video from appearing to be fused, and allows a user who is a learner to easily follow the motion of an imitation target. A video synthesis device 1 includes a storage unit 11 that stores a reference video that is a video of an imitation target whose motion is to be imitated by a user, and a storage unit 11 that stores a reference video that is a video of a motion of an imitation target whose motion is to be imitated by a user; The composition unit includes a video acquisition unit 12 that acquires a self-image that is a video, a composition unit 13 that combines the reference video and the self-video to generate a composite video, and a video output unit 14 that outputs the composite video. Step 13 combines the reference video and self-video so that the ratio of the combination of both images changes continuously and repeatedly over time. [Selection diagram] Figure 1

Description

The present invention relates to a video synthesis device and the like that combines and outputs a reference video that is a video of an action to be imitated and a self-video that is a video of a user's action to be performed.

Conventionally, in order to learn movements such as those for surgery, a reference video, which is a video of the movement of the imitation target that the learner is imitating, and a self-video, which is a video of the learner's movements, are alternately displayed. Learning support devices are known (for example, see Patent Document 1). By referring to such a display, the learner can train to perform the same motion as the motion to be imitated.

Japanese Patent Application Publication No. 2014-071443

However, when the reference video and the self-video are alternately switched and displayed, it feels like the two videos are fused. Therefore, there is no problem if the learner can properly follow the movements of the imitation target included in the reference video, but if this is not the case, the distinction between the reference video and the self-image becomes unclear, and the learner is unable to follow the imitation target's movements. There is a problem in that it becomes difficult to follow the movements of For example, it may be difficult for the learner to easily understand which self-image is.

The present invention has been made in order to solve the above-mentioned problems, and can prevent the reference video and self-video from feeling like they are fused, and allows the user, who is a learner, to easily follow the behavior of the imitator. The purpose of the present invention is to provide a video compositing device and the like for the purpose of the present invention.

In order to achieve the above object, a video synthesis device according to one aspect of the present invention includes a storage unit that stores a reference video that is a video of an action to be imitated whose action is to be imitated by a user, and an object to be moved by the user. a video acquisition unit that acquires a self-video that is a video of an action target; a synthesis unit that combines the reference video and the self-video to generate a composite video; and a video output unit that outputs the composite video. The combining unit combines the reference video and the self-video so that the combination ratio of the reference video and self-video continuously and repeatedly changes over time.

With such a configuration, it is possible to synthesize the reference image and the own image such that the ratio of synthesis between the reference image and the own image changes continuously and repeatedly over time. In this way, since the reference image and the self-image are not alternately displayed, it is possible to prevent the user from feeling as if the reference image and the self-image have been fused, and it is possible to easily understand the self-image. As a result, it becomes possible to easily follow the movements of the imitator.

Further, in the video composition device according to one aspect of the present invention, the composition unit may compose the reference video and the own video so that the change in the composition ratio of the reference video and the own video over time becomes a sine wave.

With such a configuration, for example, when compositing is performed such that the ratio of one video changes between 0 and 100%, there are periods in which only one video is mainly displayed, and both videos are Since there are periods in which the images are displayed overlapping, the user can easily distinguish between the two images, and can easily match the motion object that he or she is moving to the motion of the imitation target.

Further, the video synthesis device according to one aspect of the present invention further includes a distance acquisition unit that acquires the distance between the imitation target and the motion target in the composite video, and the composition unit determines that the farther the distance is, the more self-image is displayed. The synthesis may be performed as follows.

With such a configuration, when the distance between the imitation target and the action target is large, more self-images are displayed, thereby making it possible to move the action target so that the distance between them becomes smaller. Furthermore, when the distance between the two is small, more imitation targets are displayed, making it possible to confirm the movements of the imitation targets in more detail.

Further, the video synthesis device according to one aspect of the present invention further includes a background changing unit that changes the background area of the video to be changed, which is at least one of the reference video and the self-video, to transparent or a predetermined monochrome color, and The unit may synthesize the image changed by the background changing unit with respect to the image to be changed.

With such a configuration, it is possible to make the imitation target and the motion target more visible in the composite video. In a composite video in which a reference video and a self-video, each including a background, are combined, for example, if the imitation target overlaps the background area of the self-video, or the motion target overlaps the background area of the reference video, the imitation target overlaps with the background area of the reference video. However, such situations can be improved by changing the background area of at least one of the images to transparent or monochrome.

In addition, in the video synthesis device according to one aspect of the present invention, a specifying unit that identifies a point of interest in a specific target video that is at least one of a reference video and a self-video; The image forming apparatus may further include a blurring processing section that performs a process of blurring an area, and the compositing section may synthesize the video after the blurring processing performed by the blurring processing section for the specific target video.

With this configuration, the blurring process is performed on areas other than the points of interest, so it is possible to make the points of interest more likely to attract attention.

Furthermore, in the video synthesis device according to one aspect of the present invention, the identifying unit may identify a point of interest that is a moving point.

With such a configuration, it becomes possible to automatically specify a point of interest.

The video synthesis device according to one aspect of the present invention further includes a reception unit that receives point-of-interest information indicating a point of interest from a user, and the identification unit specifies the point of interest according to the point-of-interest information received by the reception unit. You may.

Such a configuration allows the user to specify a point of interest.

Further, a video synthesis method according to an aspect of the present invention includes: a storage unit that stores a reference video that is a video of an action to be imitated whose action is to be imitated by a user; a video acquisition unit; a synthesis unit; The video synthesis method includes a step in which the video acquisition section acquires a self-image that is a video of an operation target that is an object to be operated by a user; and a step of generating a composite video by combining the reference video and the self-video, and a step of outputting the composite video by the video output unit, and in the step of generating the composite video, the ratio of the combination of the reference video and the self-video is determined. The two images are combined in such a way that the images change continuously and repeatedly over time.

According to the video synthesis device or the like according to one aspect of the present invention, it is possible to prevent the reference video and the self-video from appearing to be fused, and the user, who is a learner, can easily follow the motion to be imitated. Be able to do things.

A block diagram showing the configuration of a video synthesis device according to an embodiment of the present invention A diagram showing an example of the usage status of the video synthesis device according to the embodiment A diagram showing an example of a reference video in the same embodiment A diagram showing an example of a self-image in the same embodiment. A diagram showing an example of a change in the synthesis ratio in the same embodiment. A diagram showing an example of a change in the synthesis ratio in the same embodiment. A diagram showing an example of a change in the synthesis ratio in the same embodiment. A diagram showing an example of a change in the synthesis ratio in the same embodiment. A diagram showing an example of a composite image in the same embodiment A diagram showing an example of a composite image in the same embodiment Flowchart showing the operation of the video synthesis device according to the embodiment A block diagram showing another example of the configuration of the video synthesis device according to the embodiment A block diagram showing another example of the configuration of the video synthesis device according to the embodiment A block diagram showing another example of the configuration of the video synthesis device according to the embodiment A diagram showing multiple areas on a video in the same embodiment A diagram showing an example of the configuration of a computer in the same embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS A video synthesis device and a video synthesis method according to the present invention will be described below using embodiments. Note that in the following embodiments, components and steps denoted by the same reference numerals are the same or equivalent, and a repeated explanation may be omitted. The video compositing device according to the present embodiment combines the reference video and the own video such that the ratio of the combination of the two images continuously and repeatedly changes over time.

FIG. 1 is a block diagram showing the configuration of a video synthesis device 1 according to this embodiment. FIG. 2 is a diagram showing a situation in which a user 30, who is a learner, is learning the motion of an imitation target using the video synthesis device 1. The video synthesis device 1 according to this embodiment includes a storage section 11, a video acquisition section 12, a composition section 13, and a video output section 14. Note that, as an example, the video synthesis device 1 may be realized by a computer 900 as shown in FIG. 13, etc., which will be described later, or may be realized by dedicated hardware. In this embodiment, the former case will mainly be explained.

The storage unit 11 stores a reference video that is a video of an action to be imitated whose action is to be imitated by the user. The user is a learner who learns movements while referring to reference videos. The motions that the user learns may be, for example, the motions of surgery, the motions of work in a factory, the motions of work in nursing care, hotels, etc., the motions of cooking, etc. It may be an action for creating a work such as a craft, it may be an action for sports, it may be an action for calligraphy, or it may be an action for tying a rope. , or other operations. The imitation target may be, for example, a part of the imitator's body, or may be an object that is operated by the imitator. The imitator may be, for example, a teacher of a user who is a learner, and may be someone who is skilled in the movement that the learner is learning. Further, the part of the imitator's body may include, for example, the imitator's hand. Further, the object to be moved by the imitator may be, for example, a tool owned by the imitator, such as forceps, a scalpel, tweezers, scissors, or a brush. The reference video is usually a video taken by a camera, but may also be a CG (Computer Graphics) video corresponding to the video taken by a camera.

The reference video may be, for example, a video from the viewpoint of the imitator who moves the imitation target, that is, a video from the first-person viewpoint of the imitator. In this case, the reference video may be, for example, a video shot with a head-mounted camera worn by the imitator. In the present embodiment, a case will be mainly described in which the reference video shown in FIG. 3 including the imitation target 41, which is a forceps used by a person to be imitated, is stored in the storage unit 11.

In the storage unit 11, for example, the entire reference video may be stored, or a part of the reference video may be stored. As an example, when the video synthesis device 1 displays a reference video while receiving it from the outside, a part of the latest received reference video that is a part of the reference video is stored in the storage unit 11, and it is readable. The information may be output and displayed, and may also be sequentially overwritten. Information other than the reference video may be stored in the storage unit 11. For example, the self-image acquired by the image acquisition unit 12 may be stored in the storage unit 11.

The process by which information is stored in the storage unit 11 does not matter. For example, information may be stored in the storage unit 11 via a recording medium, information transmitted via a communication line or the like may be stored in the storage unit 11, or, Information input via a device such as a camera may be stored in the storage unit 11. The storage unit 11 is preferably implemented by a nonvolatile recording medium, but may also be implemented by a volatile recording medium. The recording medium may be, for example, a semiconductor memory or a magnetic disk.

The video acquisition unit 12 acquires a self-video that is a video of the motion of an object to be operated by the user. The image acquisition unit 12 may be, for example, an optical device such as a camera that photographs an image, or may be one that acquires an image photographed by an optical device such as a camera. In this embodiment, a case will be mainly described in which the video acquisition unit 12 receives a video captured by the camera 901. The camera 901 that takes a self-image may be a camera worn by the user 30, such as a head-mounted camera, or a camera installed in the shooting environment with a support such as a tripod. . The action object corresponds to the imitation object. Usually, the action target and the imitation target are of the same type. Therefore, the operation target may be, for example, a part of the user's body or an object that is operated by the user. The user's body part may include, for example, the user's hand. Further, the object operated by the user may be, for example, a tool owned by the user, such as forceps, a scalpel, tweezers, scissors, or a brush. For example, when the imitation target is a forceps, it is preferable that the motion target is also a forceps.

The self-video may be, for example, a video from the viewpoint of the user who operates the action object, that is, a video from the user's first-person viewpoint. In this case, the self-image may be, for example, an image taken by a camera 901 worn on the head of the user 30, as shown in FIG. 2. It is preferable that the optical axis of camera 901 faces the object of operation. In this embodiment, a case will be mainly described in which a self-image shown in FIG. 4 including an action object 51, which is a forceps used by the user 30, is acquired by the image acquisition unit 12. In addition, the relative positional relationship between the reference video camera that shoots the reference video and the imitation target is the same as the relative positional relationship between the self-video camera that shoots the self-video and the motion target, or Preferably, it is close. Further, it is preferable that each frame included in the reference video and each frame included in the self-video have the same size, for example. The frames having the same size may mean that the number of pixels in the vertical direction and the number of pixels in the horizontal direction of the frames are the same.

The combining unit 13 combines the reference video and the own video to generate a composite video. During this compositing, the compositing unit 13 combines the reference video and the self-video so that the ratio of compositing the images continuously and repeatedly changes over time. The ratio of combining the reference video and the self-video may be the ratio of the display of the reference video and the self-video displayed in the composite video. This ratio may be, for example, the opacity or transparency of one image. In this embodiment, a case will be mainly described in which the ratio of composition of the reference image and the self-image is the ratio of the opacity of the self-image. In this case, the self-image may be combined with the front side, that is, the upper side of the reference image. For example, the combination ratio of both videos may continuously change in the time direction, for example, the combination ratio may continuously change throughout the entire combination ratio in the time direction (for example, (See FIGS. 5A and 5B), or the change in the synthesis rate may be continuous in at least a portion of the synthesis rate in the time direction (for example, see FIG. 5C). Further, the repetition of the synthesis ratio in the time direction may be, for example, periodic repetition or non-periodic repetition. The period of change in the periodic combination ratio is not particularly limited, but may be, for example, about 0.5 seconds to 3 seconds. It is preferable that the faster the movement of the imitation target, the shorter the period. Furthermore, the ratio of compositing the reference video and the own video may vary between 100:0 and 0:100, or between 95:5 and 5:95, for example. , the composite video may be changed so that there is no period during which the composite video is one of the videos. In this embodiment, the former case will mainly be explained. Furthermore, the continuous change in the composition ratio in the time direction may or may not be a smooth change. In the former case, the time change in the rate of synthesis may be time-differentiable.

For example, the synthesizing unit 13 may synthesize the reference image and the own image so that the change in the synthesis ratio of the reference image and the own image over time becomes a sine wave. In this case, the opacity of the self-image that is synthesized in front of the reference image (that is, the proportion of synthesis) may change over time as shown in FIG. 5A. In this case, there is a period when the reference image is the main image in the composite image (a period when the opacity of the self image is around 0%), and a period when the self image is the main image in the composite image (a period when the opacity of the self image is around 100%). 2), the user can view only the reference video or only his own video. Therefore, the user can easily distinguish between the imitation target included in the reference video and the motion target included in the self-video. On the other hand, in other periods, the composite video includes both the reference video and the self-video, so the user sees the imitation target included in the reference video and the action target included in the self-video together. You can also do that. Therefore, the user can easily move the motion target included in the self-video in accordance with the motion of the imitation target included in the reference video. That is, the user can make the motion target move by following the motion of the imitation target.

For example, the combining unit 13 may specify the ratio of combining the reference video and the self-video when performing the combination, and combine the reference video and the self-video at the measured ratio. For example, if the change over time in the composition ratio of the reference video and self-video is shown in FIG. 5A, the combining unit 13, when combining both videos at time t, The opacity of the video may be specified, and both videos may be combined so that the opacity of the self-image to be combined on the near side of the reference video corresponding to time t becomes the specified value. The time change in the ratio of composition of the reference video and self-video is shown by, for example, a graph, a table, a function, etc., and the composition unit 13 uses these to synthesize the reference video and self-video when performing composition. You may also specify a percentage of When the composition ratio is shown in FIG. 5A, the function represented by the opacity (%) of the self-image at time t may be, for example, represented by the following equation. In the following equation, P is the period of opacity.
Opacity of self-image (%) = 50 x (sin (2πt/P-π/2) + 1)

6 and 7 are diagrams showing an example of a composite video. 6 is a composite image generated with opacity at time T1 in FIG. 5A, and FIG. 7 is a composite image generated with opacity at time T2 in FIG. 5A. In the composite image shown in FIG. 6, since the opacity of the self image is lower than 50%, the imitation target 41 is displayed more clearly than the action target 51. On the other hand, in the composite image shown in FIG. 7, since the opacity of the self image is higher than 50%, the action object 51 is displayed more clearly than the imitation object 41.

Further, the combining unit 13 may combine the reference video and the own video so that the change over time in the combination ratio of the reference video and the own video becomes a wave other than a sine wave, for example. For example, the synthesizing unit 13 may synthesize the reference image and the self-image so that the change over time in the synthesis ratio of both images becomes a triangular wave as shown in FIG. 5B, or as shown in FIG. 5C. Both images may be combined to create a sawtooth waveform. Further, for example, when each frame included in the reference video and each frame included in the own video are different in size, the combining unit 13 adjusts the size of one video to the size of the other video. Synthesis may also be performed.

The video output section 14 outputs the composite video generated by the composition section 13. Here, this output may be displayed on a display device (for example, a liquid crystal display or an organic EL display), or may be transmitted via a communication line to a device that displays a composite image. Note that the video output unit 14 may or may not include a device that performs output (for example, a display device, a communication device, etc.). The display device on which the video output by the video output unit 14 is displayed may be, for example, a display placed in the environment where the user 30 is, as shown in FIG. It may also be a head mounted display. Further, the video output unit 14 may be realized by hardware, or may be realized by software such as a driver that drives these devices.

Next, the operation of the video synthesis device 1 will be explained using the flowchart of FIG. 8.
(Step S101) The synthesis unit 13 determines whether to synthesize the reference video and the own video. If combining is to be performed, the process proceeds to step S102; otherwise, the process of step S101 is repeated until it is determined that combining is to be performed. Note that the combining unit 13 may determine to perform the combining at each display interval of frames included in each video. For example, when the frame rate is A (fps), the combining unit 13 may determine to perform combining every 1/A (second).

(Step S102) The compositing unit 13 reads the frame of the reference video to be composited from the storage unit 11. Note that the reference video frame to be read out T seconds after the start of the compositing process may be, for example, a frame located T seconds from the beginning.

(Step S103) The video acquisition unit 12 acquires a frame of the self video. The video acquisition unit 12 may acquire the latest self-video frame from the camera 901, for example.

(Step S104) The combining unit 13 specifies the ratio of combining the reference video and the composite video. For example, if the composition rate changes in the time direction as shown in FIG. 5A, and T seconds have elapsed since the composition process was started, the composition unit 13 The corresponding composition ratio may also be specified.

(Step S105) The synthesis unit 13 synthesizes the frame of the reference image read out in step S102 and the frame of the self-image acquired in step S103 at the ratio specified in step S104, thereby creating a frame of the synthesized image. generate.

(Step S106) The video output unit 14 outputs the composite video frame generated in step S105. Then, the process returns to step S101. In this way, each frame of the reference video and self-video is combined and output, thereby performing combination of both videos.

Note that the order of processing in the flowchart of FIG. 8 is an example, and the order of each step may be changed as long as the same result can be obtained. For example, each process of steps S102 to S104 may be executed in an order other than that shown in FIG. 8. Further, in the flowchart of FIG. 8, the process is terminated by turning off the power or by an interrupt to terminate the process.

Next, the operation of the video synthesis device 1 according to this embodiment will be explained using a specific example. In this specific example, it is assumed that the reference video is a video of a situation where the imitation target 41, which is a forceps, is being used by an expert. The reference video is, for example, the video shown in FIG. Further, as shown in FIG. 2, it is assumed that a self-image of an action object 51, which is a forceps used by the user 30, is photographed using a camera 901 worn on the head of the user 30. The self-image is, for example, the image shown in FIG. 4.

First, when the user 30 operates the video compositing device 1 to start the process of compositing the reference video and self-video, the compositing unit 13 selects the first frame of the reference video stored in the storage unit 11. is read out (steps S101, S102). Further, the video acquisition unit 12 receives a frame of the self-video shot by the camera 901 via wire or wirelessly (step S103). Furthermore, the combining unit 13 specifies the combining ratio at that time (step S104). In this specific example, it is assumed that the self-image is synthesized on the near side of the reference image using the opacity of the self-image shown in FIG. 5A. Note that it is assumed that the opacity of the self-image is 0% at the start of the compositing process. Then, the synthesizing unit 13 generates a frame of a synthesized image in which the self-image whose opacity is 0% is synthesized on the near side of the reference image, and sends it to the image outputting unit 14 (step S105). In other words, a composite video frame containing only the reference video is generated. Upon receiving the composite video frame, the video output unit 14 outputs the received composite video frame to the display device 902 (step S106). In this way, the user 30 can view the composite video.

Note that the ratio of compositing the reference video and the self video changes as shown in FIG. 5A. Therefore, in the composite image, the proportion of self-image gradually increases, and at time T1, the composite image shown in FIG. 6 is displayed, and at time T2, the composite image shown in FIG. 7 is displayed. That will happen. Furthermore, after a composite image containing only self-images is displayed, the proportion of self-images in the composite image gradually decreases. In this way, the proportion of the self-image in the composite image changes periodically within the range from 0% to 100% according to the sine wave. The user 30 can learn by moving the motion target so that the motion is similar to the motion of the imitation target while viewing the composite video displayed on the display device 902.

As described above, according to the video composition device 1 according to the present embodiment, it is possible to composite the reference video and the own video such that the ratio of composition of both videos changes continuously and repeatedly over time. . Therefore, since the reference video and the self-video are not alternately displayed, it is possible to prevent the user from feeling as if the reference video and the self-video are fused. As a result, the user viewing the composite video can easily distinguish between the imitation target 41 in the reference video and the motion target 51 in the self-video, and moves the motion target to follow the motion of the imitation target. It is possible to increase the effectiveness of learning using reference videos. Additionally, if the proportion of the combination of both images is changed periodically, it is possible to easily predict when the reference image will be displayed as the main image or when the self image will be displayed as the main image. As a result, it becomes easier to identify the imitation target or action target in the composite video.

Note that the video synthesis device 1 according to the present embodiment further includes a distance acquisition unit 21 that acquires the distance between the imitation target and the motion target in the composite video, for example, as shown in FIG. For example, composition may be performed such that the greater the distance acquired by the distance acquisition unit 21, the more self-images are displayed. The distance between the imitation target and the motion target in a composite video is, if a composite video is generated by combining a reference video read out at a certain point with a self-image acquired at that time, This is the distance between the imitation target and the motion target in the composite video. This distance may be, for example, the distance between the imitation target and the motion target in the frame of the synthesized image that was synthesized immediately before, and in order to obtain this distance, the reference image frame and the self-image frame at the time of synthesis are It may be the distance between the imitation target and the motion target in the frame of the composite video that is synthesized with the above, and the distance between the position of the motion target in the frame of the reference video at the time of composition and the position of the motion target in the latest self-video frame. It may be distance.

The imitation target or action target area in a video frame may be specified by, for example, object detection in the frame, segmentation, pattern matching, or the like. Further, the position of the imitation target or the position of the motion target may be, for example, a representative position of a region of the imitation target or motion target specified in the frame. The representative position of a certain area may be, for example, the position of the center of gravity of the area, or the position of the center of a bounding box surrounding the area.

For example, the combining unit 13 combines the reference video and the self-video so that when the acquired distance is larger than the threshold TH, more self-images are displayed than when the acquired distance is smaller than the threshold TH. May be synthesized. Note that the combining unit 13 combines the reference video and the self-video so that, for example, when the acquired distance is equal to the threshold TH, more self-videos are displayed than when the acquired distance is smaller than the threshold TH. may or may not be synthesized. Combining so that more self-images are displayed may mean, for example, combining so that the average value of the proportion of self-images in the composite video over a predetermined period in the time direction becomes larger. When the rate of synthesis changes periodically, the predetermined period may be a period corresponding to an integral multiple of one cycle of the rate of synthesis. In this case, when the acquired distance is greater than the threshold TH, more self-images are displayed when the composite image is displayed than when the acquired distance is smaller than the threshold TH. Become. As an example, if the acquired distance is greater than the threshold TH, composition may be performed such that the average ratio of self-images in the composite image exceeds 50%. When the distance between the imitation target and the action target is large, the action target is usually moved so that the position of the action target approaches the position of the imitation target, rather than adjusting the action of the action target to the action of the imitation target. It turns out. Therefore, by compositing so that more of the motion targets are displayed than when the distance between the imitation target and the motion target is small, it becomes possible to easily move the motion targets. On the other hand, when the distance between the imitation target and the motion target is small, the motion target is usually moved so that the motion of the motion target matches the motion of the imitation target. Therefore, by compositing so that more imitation targets are displayed than when the distance between the imitation target and the motion target is large, the motion can be easily imitated.

For example, if the acquired distance is greater than the threshold TH, the combining unit 13 combines the reference image and the self-image according to the opacity of the self-image that changes over time as shown in FIG. 5D. You may do so. Furthermore, when the acquired distance is smaller than the threshold TH, the synthesis unit 13 synthesizes the reference image and the self-image according to the opacity of the self-image that changes over time as shown in FIG. 5A. You may go. By doing this, the reference video and self-video are combined so that when the acquired distance is greater than the threshold TH, more self-images are displayed than when the acquired distance is smaller than the threshold TH. You will be able to do this.

Furthermore, as shown in FIG. 10, for example, the image synthesizing device 1 according to the present embodiment makes the background region transparent or a predetermined monochrome color for the image to be changed, which is at least one of the reference image and the self image. The image processing apparatus may further include a background changing section 22 that changes the background, and the synthesizing section 13 may synthesize the image changed by the background changing section 22 with respect to the image to be changed. The video to be changed may be, for example, only the reference video, only the self-video, or both videos. If the video to be changed is only the reference video, the compositing unit 13 may combine the reference video with the background changed and the self-video without the background changed. Further, when the video to be changed is only the self-image, the compositing unit 13 may combine the reference video whose background has not been changed and the self-image after the background has been changed. Further, when the video to be changed is a reference video and a self-video, the combining unit 13 may combine the reference video after the background has been changed and the self-video after the background has been changed.

The background changing unit 22 may, for example, specify an area to be imitated or an action target, and specify an area other than the specified area as the background area. The area to be imitated or to be acted upon may be specified as described above. In addition, areas of objects (for example, internal organs or simulated organs) that are subject to predetermined processing (for example, suturing processing performed by the imitation target or motion target, which are forceps) depending on the imitation target or motion target are also displayed as non-background images. Good too. Then, the background changing unit 22 may change the identified background area to, for example, transparent or a predetermined single color. In addition, if both the reference video and self-video are the videos to be changed, and the background areas of both videos are changed to a single color, the background color of the reference video and the background color of the self-video will be changed. may be different from By doing this, when the composite video is displayed, it becomes possible to easily understand which video is being displayed as the main one using the color of the background. Furthermore, software that makes the background area of an image transparent or a specific monochromatic color is known. The background changing unit 22 may use such software to change the background area to transparent or monochrome in each frame of the video to be changed.

In this way, by changing the background area of at least one of the images to be transparent or monochrome, it is possible to make the imitation target or the action target more visible in the composite image. For example, by displaying a transparent or monochrome background area of the self-image in the area to be imitated, or by displaying a transparent or monochrome background area of the reference image in the area to be imitated, the user can You will be able to easily identify the target.

Furthermore, as shown in FIG. 11, the video synthesis device 1 according to the present embodiment also includes a reception unit 23 that receives attention point information indicating a point of interest from a user, and a specific part that is at least one of a reference video and a self-video. The compositing unit 13 further includes a specifying unit 24 that specifies a point of interest in the target video, and a blurring processing unit 25 that performs a process of blurring an area other than the specified point of interest in the target video. As for the video, the video after blurring processing by the blurring processing unit 25 may be combined. The specific target video may be, for example, only the reference video, only the self-video, or both videos.

For example, as shown in FIG. 12, the reception unit 23 may receive point-of-interest information that is at least one of a plurality of region identifiers that respectively identify predetermined regions 61 to 65 in a video frame. In this case, the area identified by the area identifier that is the attention point information may be the attention point. The region identifiers may be, for example, "center," "upper left," "upper right," "lower left," and "lower right," which identify the regions 61 to 65, respectively. Furthermore, the receiving unit 23 may receive point-of-interest information that is a result of voice recognition, for example. In this case, the user can specify the point of interest by voice. It goes without saying that the number and positions of points of interest that can be specified in the video may be other than those shown in FIG. 12.

For example, the reception unit 23 may receive information input from an input device (for example, a keyboard, mouse, touch panel, etc.), or may receive information transmitted via a wired or wireless communication line. Note that the accepting unit 23 may or may not include a device for accepting requests (for example, an input device, a communication device, etc.). Further, the reception unit 23 may be realized by hardware or by software such as a driver that drives a predetermined device.

The specifying unit 24 may specify the point of interest according to the point of interest information received by the reception unit 23, for example. In this case, the specifying unit 24 may, for example, specify the point of interest which is an area identified by the area identifier which is the point of interest information received by the reception unit 23. Specifically, when point-of-interest information with the region identifier "center" is received, the specifying unit 24 may specify the region 61 identified by the region identifier "center" as the point of interest. In the case where attention point information is accepted and the specific target video is both a reference video and a self-image, for example, the attention point in both videos may be the same or may be different. good. In the latter case, for example, attention point information may be accepted for each video.

Further, the specifying unit 24 may specify the point of interest without using the received point of interest information. In this case, the specifying unit 24 may specify, for example, a point of interest that is a moving point in the video to be specified. To find a point of interest that is a moving part of a video, for example, extract feature points from each of two frames separated by a predetermined time interval in the video, and calculate the per unit time of the corresponding feature points in those two frames. It may be a change in position, that is, a location where the distance exceeds a threshold value. As an example, a location in the video where the magnitude of the optical flow vector exceeds a predetermined threshold may be set as a location of interest that is a location in the video that has movement. This threshold value may be set, for example, so that the imitation target or the motion target is included in the attention location. Note that the shape of the point of interest may be determined in advance, such as a rectangular shape or an elliptical shape, for example. Furthermore, for example, the size of the point of interest may also be determined in advance. In this case, the specifying unit 24 may specify, for example, a point of interest that is an area of a predetermined size and a predetermined shape that includes a moving portion in the video to be specified. In the case where a point of interest is specified without using point of interest information, and when the video to be identified is both a reference video and a self-video, for example, the points of interest in both videos may be different. . When specifying a point of interest without using the received point-of-interest information, the video synthesis device 1 does not need to include the reception unit 23.

The blurring processing unit 25 performs a process of blurring areas other than the point of interest specified by the specifying unit 24 in the specific target video. As an example, the specifying unit 24 passes the video to be identified and an area indicating the point of interest in the video to the blurring processing unit 25, and the blurring processing unit 25 uses the received video and the point of interest to You may perform processing to blur areas other than the above. The process performed by the blurring processing unit 25 may be, for example, a process known as a blurring process in image processing software. For example, this process may be a process of replacing the color of a certain pixel with the average value of surrounding colors in a predetermined range centered on that pixel.

In this way, by performing the blurring process on areas other than the point of interest, it is possible to make it easier for the user to focus on the point of interest. For example, if a reference video that has not been blurred and a self-video are combined, and both videos are included in the composite video, the background of the self-video may be displayed in the area to be imitated, or the background of the self-video may be displayed in the area to be imitated. The background of the reference video may be displayed in the area, and in such a situation, it may be difficult for the user to easily identify the imitation target or action target. On the other hand, if blurring is performed on an area other than the point of interest in at least one of the videos, for example, the blurred area of the self-image is displayed in the area to be imitated, or the area where the action is targeted is displayed. By displaying the blurred area of the reference video on the screen, the user can easily identify the imitation target or action target.

Further, in this embodiment, the case where the reference video, self-video, and composite video are two-dimensional single videos has been mainly described, but these videos may be stereo videos. In this case, the video output unit 14 may output a composite video, which is a stereo video, to a head-mounted display worn by the user 30.

Further, in the above embodiment, the case where the video synthesis device 1 is a standalone device has been described, but the video synthesis device 1 may be a standalone device or a server device in a server/client system. In the latter case, the reception unit, acquisition unit, and output unit may accept input, acquire information, and output information via a communication line.

Furthermore, in the above embodiments, each process or each function may be realized by being centrally processed by a single device or a single system, or may be realized by being distributedly processed by multiple devices or multiple systems. This may be realized by

In addition, in the above embodiment, the information exchange performed between each component is performed by one component, for example, when the two components that exchange the information are physically different. This may be done by outputting information and receiving the information by another component, or by one component if the two components passing that information are physically the same. This may be performed by moving from a phase of processing corresponding to the component to a phase of processing corresponding to the other component.

In the above embodiments, information related to processing executed by each component, for example, information accepted, acquired, selected, generated, transmitted, or received by each component. Information such as threshold values, formulas, addresses, etc. used by each component in processing may be held temporarily or for a long period of time in a recording medium (not shown), even if not specified in the above description. Further, the information may be stored in the recording medium (not shown) by each component or by a storage unit (not shown). Further, each component or a reading unit (not shown) may read information from the recording medium (not shown).

In addition, in the above-described embodiment, if the information used in each component, for example, information such as threshold values, addresses, various setting values, etc. used by each component in processing, may be changed by the user, the above-mentioned Even if it is not specified in the description, the user may or may not be able to change the information as appropriate. If the information can be changed by the user, the change is realized by, for example, a reception unit (not shown) that receives change instructions from the user, and a change unit (not shown) that changes the information in accordance with the change instruction. You can. The acceptance of the change instruction by the reception unit (not shown) may be, for example, acceptance from an input device, information transmitted via a communication line, or information read from a predetermined recording medium. .

Further, in the above embodiment, when two or more components included in the video synthesis apparatus 1 have a communication device, an input device, etc., even if the two or more components physically have a single device, or may have separate devices.

Furthermore, in the embodiments described above, each component may be configured by dedicated hardware, or components that can be realized by software may be realized by executing a program. For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. At the time of execution, the program execution section may execute the program while accessing the storage section or recording medium. Note that the software that implements the video synthesis device 1 in the above embodiment is the following program. In other words, this program causes a computer that has access to a storage unit that stores a reference video that is a video of the motion of the target to be imitated, the motion of which the user is to imitate, to perform the motion of the target to be imitated. It functions as a video acquisition unit that acquires a self-image that is a video, a composition unit that combines the reference video and the self-video to generate a composite video, and a video output unit that outputs the composite video. It may be a program that combines both images so that the ratio of the combination with the video changes continuously and repeatedly over time.

Note that in the above program, the functions realized by the program do not include functions that can only be realized by hardware. For example, functions that can only be realized by hardware such as a modem or an interface card in an acquisition unit that acquires information, an output unit that outputs information, etc. are not included in the functions that are realized by the program.

Further, this program may be executed by being downloaded from a server or the like, and the program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, etc.) is read out. It may be executed by Further, this program may be used as a program constituting a program product.

Further, the number of computers that execute this program may be one or more. That is, centralized processing or distributed processing may be performed.

FIG. 13 is a diagram illustrating an example of the configuration of a computer 900 that executes the above program to realize the video synthesis apparatus 1 according to the above embodiment. In FIG. 13, a computer 900 includes an MPU (Micro Processing Unit) 911, a ROM 912 for storing programs such as a boot-up program, and a ROM 912 connected to the MPU 911 to temporarily store instructions of an application program. It includes a RAM 913 that provides space, a storage unit 914 that stores application programs, system programs, and data, and a communication module 915 that provides connection to a LAN, WAN, etc. Note that the MPU 911, the ROM 912, and the like may be interconnected by a bus together with the camera 901, the display device 902, the keyboard 903, and the pointing device 904 such as a touch pad or a mouse. Further, the storage unit 914 may be, for example, a hard disk or an SSD (Solid State Drive). Further, the camera 901, display device 902, keyboard 903, pointing device 904, etc. may be external devices, or may be devices built into the computer 900, for example.

A program that causes the computer 900 to execute the functions of the video synthesis apparatus 1 according to the embodiment described above may be loaded into the RAM 913 at the time of execution. Note that the program may be loaded directly from the storage unit 914 or the network, for example.

The program does not necessarily need to include an operating system (OS) or a third party program that causes the computer 900 to execute the functions of the video synthesis apparatus 1 according to the embodiment described above. A program may include only those portions of instructions that call appropriate functions or modules in a controlled manner to achieve desired results. How computer 900 operates is well known and detailed explanation will be omitted.

Further, the above embodiments are illustrative examples for concretely implementing the present invention, and do not limit the technical scope of the present invention. The technical scope of the present invention is indicated by the claims, not the description of the embodiments, and includes changes within the literal scope and equivalent meaning of the claims. is intended.

1 Video synthesis device 11 Storage unit 12 Video acquisition unit 13 Synthesis unit 14 Video output unit 21 Distance acquisition unit 22 Background change unit 23 Reception unit 24 Specification unit 25 Processing unit

Claims (9)

a storage unit that stores a reference video that is a video of an action to be imitated whose action is to be imitated by the user;
a video acquisition unit that acquires a self-image that is a video of the action of an action target that is an object to be operated by the user;
a synthesizing unit that synthesizes the reference image and the self-image to generate a composite image;
a video output unit that outputs the composite video,
The combining unit is a video combining device that combines the reference video and the own video such that a combination ratio of the reference video and the own video continuously and repeatedly changes over time. 2. The video combining device according to claim 1, wherein the combining unit combines the reference video and the own video so that a change over time in a ratio of combining the reference video and the own video becomes a sine wave. further comprising a distance acquisition unit that acquires a distance between the imitation target and the motion target in the composite video,
The image synthesizing device according to claim 1, wherein the synthesizing unit performs the synthesizing so that the greater the distance, the more of the self-image is displayed. further comprising a background changing unit that changes a background area of the video to be changed, which is at least one of the reference video and the self-video, to transparent or a predetermined monochromatic color;
3. The video compositing device according to claim 1, wherein the compositing unit combines the video changed by the background changing unit with respect to the video to be changed. a specifying unit that specifies a point of interest in a specific target video that is at least one of the reference video and the self-video;
further comprising a blurring processing unit that performs processing to blur an area other than the specified point of interest in the specific target video,
The image synthesis apparatus according to claim 1 or 2, wherein the synthesis section synthesizes the image subjected to blurring processing by the blurring processing section for the specific target image. 6. The video synthesizing device according to claim 5, wherein the specifying unit specifies a point of interest that is a moving point. Further comprising a reception unit that receives attention point information indicating attention points from the user,
6. The video synthesis apparatus according to claim 5, wherein the specifying section specifies the point of interest according to point of interest information received by the receiving section. A video synthesis method in which processing is performed using a storage unit storing a reference video that is a video of an action to be imitated whose action is to be imitated by a user, a video acquisition unit, a compositing unit, and a video output unit. There it is,
a step in which the image acquisition unit acquires a self-image that is an image of an operation target that is an object to be operated by the user;
the combining unit combining the reference video and the self-video to generate a composite video;
the video output unit outputting the composite video,
In the step of generating the composite video, the video compositing method comprises composing the reference video and the self-video so that the ratio of composition of the reference video and the self-video continuously and repeatedly changes over time. A computer capable of accessing a storage unit in which a reference image, which is an image of the movement of the imitation target whose movement is to be imitated by the user, is stored;
a video acquisition unit that acquires a self-image that is a video of the action of an action target that is to be operated by the user;
a synthesizing unit that synthesizes the reference image and the self-image to generate a composite image;
Function as a video output unit that outputs the composite video,
The combining unit is a program that combines the reference video and the self-video so that a combination ratio between the reference video and the self-video continuously and repeatedly changes over time.

Images