JP2015154429A - System for giving video substance - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make a part of an object in video substantial.SOLUTION: A video acquisition unit 10 captures video image including at least one subject and at least one target which is to be made to be substantial, the target being visible in a state of being connected to the subject. An information acquisition unit 12 acquires motion information including a variation in a position of a contact between the subject and the target which is to be made to be substantial. A video reproduction unit 20 deletes the target which is to be made to be substantial from the video and reproduces it on a screen. A substance reproduction unit 22 controls, on the basis of the motion information, a position of an object 26 obtained by making substantial the target which is to be made to be substantial so that the object 26 becomes visible in the state of being connected to the subject in the video.

Description

  The present invention relates to a video materialization technique for materializing a part of an object on a video.

  Improvements in communication technology have made it possible for people in remote locations to communicate with each other by video and audio using video chat such as video conferencing and videophone. As a video chat technique for sharing a state on a work table between remote locations, for example, a technique described in Non-Patent Document 1 can be cited.

Naomi Yamashita, Katsuhiko Tsuji, Hideaki Kuzuoka, Junji Hirata, Shigetoshi Aoyagi, “Proposal and Evaluation of a Method for Improving Gesture Visibility of Remote Users”, IPSJ Transactions, Vol. 52, No. 1, pp. 97 -108, January 2011

  In the conventional video chat technique as described in Non-Patent Document 1, it is possible to transmit the motion of a remote person or object using video images. However, with the conventional video chat technology, it is impossible to view or touch the movement of a person or an object in the image in three dimensions. If a stereoscopic technique such as a 3D display is used, a remote object can be viewed three-dimensionally to some extent, but it cannot be touched yet. If the movements of humans and objects in the video are materialized, it is possible to grasp their states more accurately and to expect smooth communication.

  An object of the present invention is to facilitate communication in a video chat communication by materializing a person or object in a video.

  In order to solve the above problems, a video materialization system of the present invention includes a transmission device and a reception device. The transmission device includes a video acquisition unit that captures an image including at least one subject and at least one materialization target that is continuously visible to the subject, and at least a change in a position of a contact point between the materialization target and the subject in the video. And an entity information acquisition unit for acquiring the operation information. The receiving device continuously recognizes the object and the subject in the video based on the operation information, the object in which the materialization target is materialized, the video playback unit that erases the materialization target from the video and reproduces it on the screen. An entity reproduction unit that controls the position of the object so as to be possible.

  According to the video materialization technology of the present invention, it is possible to materialize a part of a person or an object on the video, so that it is easy to grasp the movement of a remote person or object. Therefore, communication can be facilitated.

FIG. 1 is a diagram illustrating a functional configuration of a video materialization system. FIG. 2 is a diagram illustrating a processing flow of the video materialization method. FIG. 3 is a diagram for explaining a specific configuration example of the entity reproduction unit. FIG. 4 is a diagram for explaining a specific configuration example of the entity reproduction unit. FIG. 5 is a diagram for explaining a specific configuration example of the entity reproduction unit.

  The video materialization technology of the present invention uses robot technology, motion tracking technology, and video composition technology to replace a part of a person or object on the video with a physical entity and jump out of the screen. It makes it possible to move on top.

  The video materialization system according to the embodiment of the present invention includes a robot arm, a positioning mechanism, a motion tracking device and software, video composition software, and a video conference terminal. In the video materialization system, a part of a person or object in a video is instantiated by a robot or the like and popped out from the video, and is instantiated according to the movement of a human or object in the video by motion tracking technology and video composition technology. Move the object.

  Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the component which has the same function in drawing, and duplication description is abbreviate | omitted.

  A configuration example of a video materialization system according to an embodiment will be described with reference to FIG. The video materialization system includes a transmission device 1 and a reception device 2. The transmission device 1 and the reception device 2 are connected to the communication network 9. The communication network 9 only needs to be configured so that the transmission device 1 and the reception device 2 can communicate with each other. For example, a circuit-switching communication network such as an analog or digital public telephone network or a private telephone network, Alternatively, it can be configured by a packet switching communication network such as the Internet, a LAN (Local Area Network), and a WAN (Wide Area Network).

  The transmission device 1 includes a video acquisition unit 10, an entity information acquisition unit 12, a camera 14, and a sensor 16. The transmission device 1 is a special configuration in which a special program is read into a known or dedicated computer having a central processing unit (CPU), a main storage device (Random Access Memory, RAM), and the like. Device. For example, the transmission device 1 executes each process under the control of the central processing unit. The data input to the transmission device 1 and the data obtained in each process are stored in, for example, a main storage device, and the data stored in the main storage device is read out as needed and used for other processing. The

  The camera 14 is, for example, a video camera that captures images of a person or an object as a subject. For example, there are no restrictions on the functions that the camera 14 should have, such as the resolution that can be taken, the recording medium of the video, the presence / absence of a microphone, and whether it is a digital analog or the like. Can be used.

  The sensor 16 is, for example, a motion capture sensor that sequentially collects three-dimensional coordinates of an object. By collecting the three-dimensional coordinates of the subject of the image by the sensor 16, the position of the subject and the change in the position can be acquired. Motion capture sensors that can be used as the sensor 16 are, for example, “Japan Microsoft Corporation,“ Xbox 360-Kinect ”, [online], [Search January 27, 2014], Internet <URL: http: // www. xbox.com/ja-JP/kinect> (Reference 1) ”.

  As shown in FIG. 1, the camera 14 and the sensor 16 may be external peripheral devices connected to the transmission device 1 or may be incorporated in the housing of the transmission device 1. Further, as shown in FIG. 1, the camera 14 and the sensor 16 may be configured as separate devices, or may be configured as one device in which the camera 14 and the sensor 16 are incorporated in one casing.

  The receiving device 2 includes a video reproduction unit 20, an entity reproduction unit 22, a display 24 and an object 26. The receiving device 2 is a special device configured by reading a special program into a known or dedicated computer having a central processing unit (CPU), a main storage device (Random Access Memory, RAM), and the like. Device. For example, the receiving device 2 executes each process under the control of the central processing unit. The data input to the receiving device 2 and the data obtained in each process are stored in, for example, the main storage device, and the data stored in the main storage device is read out as necessary and used for other processing. The

  The display 24 is, for example, a large liquid crystal display having a screen for playing back video captured by the camera 14 of the transmission apparatus 1. As shown in FIG. 1, the display 24 may be an external peripheral device connected to the receiving device 2, or may be incorporated in one plane of the receiving device 2. For example, there are no restrictions on the functions that the display 24 should have, such as displayable resolution, presence / absence of speakers, and whether it is digital / analog, and any display can be used as long as the video captured by the camera 14 can be reproduced. be able to.

  The object 26 is an object that materializes the materialization target in the video. The object 26 is configured to be movable on the screen surface of the display 24. The materialization target may be any object as long as there is a possibility of being captured in a video. For example, if the subject of the image is a single person and the materialization target is configured as the part from the elbow of the human arm, the object 26 may be a device that imitates the human arm, such as a robot arm. . Further, for example, if the subject of the video is a human holding an object such as a pointing stick in his hand and the materialization target is configured as a pointing stick, the object 26 may be an object having the same shape as the pointing stick. .

  With reference to FIG. 2, an example of the processing flow of the video materialization method executed by the video materialization system according to the embodiment will be described in the order of procedures.

  In step S <b> 10, the video acquisition unit 10 of the transmission device 1 uses the camera 14 of the transmission device 1 to capture a video including at least one subject and at least one materialization target that can be viewed continuously with the subject. . The captured video is transmitted to the receiving device 2 via the communication network 9. The captured video may be encoded and compressed in an appropriate data format in order to increase the transfer efficiency during transmission.

  In step S <b> 12, the entity information acquisition unit 12 of the transmission device 1 uses the sensor 16 of the transmission device 1 to acquire a change in the position of the contact point between the materialization target and the subject in the video. As the information to be acquired, not only the change in the position of the contact point between the materialization target and the subject but also detailed operation information such as the direction and angle of the subject and the materialization target may be acquired. The acquired information is transmitted to the receiving device 2 via the communication network 9.

  In step S <b> 20, the video reproduction unit 10 of the reception device 2 deletes the materialization target from the video received from the transmission device 1 and reproduces it on the display 24 of the reception device 2. As a method for deleting the materialization object from the video, an existing video synthesis technique such as chroma key synthesis may be used.

  In step S <b> 22, the entity reproduction unit 22 of the reception device 2 controls the position of the object 26 based on the motion information received from the transmission device 1 so that the object 26 and the subject in the video can be visually recognized continuously. To do. More specifically, the position of the object 26 is controlled so that the position of the contact point in the object 26 and the position of the contact point in the subject in the video reproduced on the screen of the display 24 match. Further, when the orientation and angle of the subject and the materialization target are acquired, the direction and angle of the object 26 are adjusted so that the contour of the subject and the materialization target match.

  The method for controlling the position of the object 26 will be described in more detail with reference to FIG. FIG. 3 shows an example in which a part from the elbow of a human being in a remote place is materialized with a robot arm. An object 26 (a robot arm in the example of FIG. 3) that is a materialized portion that has jumped out of the display 24 is fixed via positioning mechanisms 222 to 226 installed under the table 221 on which the display 24 is installed. In the example of FIG. 3, the object 26 is fixed to the front end of a transparent plate 222 such as an acrylic plate, and the rotation axis 224 that moves the acrylic plate 222 along the rail 223 fixed in the horizontal direction, and the vertical direction The robot arm 26 can move on the screen surface of the display 24 by positioning with a biaxial linear motion mechanism comprising a rotary shaft 226 that moves along a rail 225 fixed to the rail 225.

  The object 26, which is a materialized portion that protrudes from the display 24, may not only move in parallel on the screen surface of the display 24, but may also be adjusted in angle so as to be visually connected to the image visually by a rotation mechanism or the like. . In the example of FIG. 4A, the horizontal angle of the robot arm 26 is adjusted so that the portion from the human shoulder to the elbow in the image and the robot arm 26 are in a straight line. In the example, the vertical angle of the robot arm 26 is adjusted.

  The size of the object 26 can be changed according to the movement of the materialization target. With reference to FIG. 5, an example of changing the length of the materialized robot arm when the materialized object is the subject's arm will be described. The robot arm 26 includes a robot hand 265, a cylindrical first arm portion 264 with the robot hand 265 attached to the tip thereof, and a cylindrical second arm portion 263 into which the first arm portion 264 can be inserted, The first arm portion 264 and a winding mechanism 261 and 262 connected by wires. As shown in FIG. 3, the robot arm 26 is fixed to the upper end of the acrylic plate 222. The wires are made not to stand out along the ridges of the acrylic plate 222, and the winding mechanisms 261 and 262 are installed at the lower end of the acrylic plate 222. By operating the winding mechanisms 261 and 262, the first arm portion 264 is configured to slide in the second arm portion 263. FIG. 5A shows an example in which the robot arm 26 is extended. By loosening the winding mechanism 261 and winding up the winding mechanism 262, the first arm 264 protrudes from the second arm 263, and the entire length of the robot arm 26 can be extended. FIG. 5B shows an example in which the robot arm 26 is contracted. By loosening the winding mechanism 262 and winding up the winding mechanism 261, the first arm 264 is drawn into the second arm 263, and the overall length of the robot arm 26 can be reduced. For example, if the subject in the video protrudes the arm forward, the robot arm's overall length is extended, and if the arm is retracted backward, the robot arm's overall length is shortened to further improve the reproducibility of the materialized object. be able to.

  According to the video materialization technology of the present invention, a part of an object in a video is materialized and popped out, thereby making it easy to grasp a three-dimensional movement that is difficult to understand in a planar video. In the example shown in FIG. 3 and FIG. 4, it is easy to grasp what a remote person is pointing to by materializing a remote person's arm. In addition, according to the video materialization technology of the present invention, it is possible for persons between remote locations to come into contact with each other via the materialization part. For example, it is possible to perform a handshake by grasping hands (for example, robot hands) that are materialized from each other.

  The present invention is not limited to the above-described embodiment, and it goes without saying that modifications can be made as appropriate without departing from the spirit of the present invention. The various processes described in the above embodiment may be executed not only in time series according to the order of description, but also in parallel or individually as required by the processing capability of the apparatus that executes the processes or as necessary.

[Program, recording medium]
When various processing functions in each device described in the above embodiment are realized by a computer, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, the present apparatus is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

DESCRIPTION OF SYMBOLS 1 Transmission apparatus 2 Reception apparatus 9 Communication network 10 Image | video acquisition apparatus 12 Entity information acquisition part 14 Camera 16 Sensor 20 Image | video reproduction | regeneration part 22 Entity reproduction part 24 Display 26 Object

Claims (6)

A video materialization system including a transmission device and a reception device,
The transmitter is
A video acquisition unit that captures a video including at least one subject and at least one materialization object that is continuously visible with the subject;
An entity information acquisition unit that acquires at least operation information including a change in position of a contact point between the object to be materialized and the subject in the video;
Including
The receiving device is
An object that materializes the above materialization target;
A video playback unit for deleting the materialization object from the video and playing it on the screen;
An entity reproduction unit that controls the position of the object so that the object and the subject in the video are continuously visible based on the motion information;
Video materialization system including The video materialization system according to claim 1,
The video materialization system, wherein the entity reproduction unit controls the position of the object so that the position of the contact point on the object matches the position of the contact point on the subject in the video reproduced on the screen. The video materialization system according to claim 1 or 2,
The entity reproducing unit controls the position of the object so that the object can be seen as a part of the subject following the movement of the subject in the image. The video materialization system according to any one of claims 1 to 3,
The above-mentioned substance reproduction part changes the size of the above-mentioned object according to the operation of the subject in the above-mentioned picture. The video materialization system according to any one of claims 1 to 4,
The subject is a portion including at least one arm of one person,
The materialization object is a portion of the one arm from the elbow,
The contact is the elbow;
The entity reproduction unit controls the position of the object so as to reproduce the movement from the elbow of the one arm. The video materialization system according to any one of claims 1 to 4,
The subject is a portion including at least the hand that grasps the object of one person who grasps the object in the hand,
The materialization object is the object,
The contact is a hand holding the object,
The above-mentioned substance reproduction part controls the position of the above-mentioned object so that the above-mentioned object may be grasped and moved by the above-mentioned hand.

Images