JP6156934B2 - Spatial synthesis system - Google Patents

Description

  The present invention relates to a space synthesis technique for reproducing the presence of a conversation partner in different spaces in the same space.

  In a video chat in which video images are mutually presented between interlocutors in remote locations, the presence of the conversation partner can be felt closer than in the case of a conversation using only voice (for example, telephone or voice chat). In video chatting, many researches have been conducted on how to further enhance the presence of the conversation partner. For example, there is a method of presenting a video as if the conversation partner is in front of you. Specifically, there are a method of presenting the other person in a life-size manner on a large display, a method of determining the positional relationship between the camera and the display so that eye contact is established, and the like.

  On the other hand, a method has also been proposed in which only the body of the conversation partner is cut out by chroma key composition and synthesized next to the video of the user (see Non-Patent Document 1). The method described in Non-Patent Document 1 looks like a mirror in the display in front of the user, as if there is a conversation partner next to the user. Hereinafter, a system that implements the method described in Non-Patent Document 1 is referred to as a mirror system.

Osamu Morikawa, Takanori Maesako, “HyperMirror: Toward Pleasant-to-use Video Mediated Communication System”, CSCW '98, 149-158, 1998.

  Since the conventional method is a method for visually sensing the presence of a conversation partner, the user cannot feel the presence of the partner with other senses.

  An object of the present invention is to provide a space synthesis technique capable of reproducing the presence of a conversation partner in a different space in the same space in video chat communication.

  In order to solve the above-described problems, a space composition system of the present invention includes a space composition device in which at least one person and a shared object exist in each of a plurality of spaces, and are installed in each of the plurality of spaces. The space synthesis device includes a video acquisition unit that acquires a video including a person and a shared object, a motion information acquisition unit that acquires motion information that tracks the motion of the person, and a person in another space as a conversation partner. Based on the video playback unit that extracts the video of the conversation partner from the video acquired by the space synthesizer in another space, combines the video of the conversation partner with the video acquired by the video acquisition unit, and the operation information of the conversation partner And a motion reproduction unit that reproduces the behavior of the conversation partner by operating the shared object.

  According to the present invention, since the presence of the conversation partner in different spaces can be reproduced in the same space, it becomes easy to grasp the operation of the conversation partner. Therefore, communication can be facilitated.

FIG. 1 is a diagram for explaining an example of how to use the space synthesis system. FIG. 2 is a diagram illustrating a functional configuration of the space synthesis system. FIG. 3 is a diagram illustrating a functional configuration of the space synthesis device. FIG. 4 is a diagram illustrating a functional configuration of the shared object. FIG. 5 is a diagram for explaining an example in which a shared object is a chaise longue. FIG. 6 is a diagram illustrating a processing flow of the space synthesis system.

  In the spatial synthesis technology of the present invention, in the mirror system described in Non-Patent Document 1 that synthesizes the body of the conversation partner next to the video of the user and presents the video like a mirror, Add a system to present the presence of the conversation partner through the senses. An object having a function of presenting the presence of the conversation partner (hereinafter referred to as a shared object) is placed between the synthesized video of the user and the conversation partner, and operates in accordance with the movement of the synthesized video of the conversation partner. . That is, the user can feel the body movement of the conversation partner through the movement of the shared object. Compared to the conventional mirror system, it is an innovative point of the present invention that it is possible to reproduce the feeling that the conversation partner is adjacent not only through the image of the conversation partner but also through other senses.

With reference to FIG. 1, the example of the usage method of the space synthesis system of this invention is demonstrated. The large display 5 and the camera 3 is installed in front of the user _61, and displays the image captured by the camera 3 on the display 5. The position and size of the image are adjusted to be the same as the image reflected on the mirror installed at the place of the display 5. Next to the user 6 ′ ₁ in the video, the video 6 ′ _{2 of} only the body of the remote conversation partner 6 ₂ is superimposed using video synthesis technology such as chroma key synthesis. Between the position of the user ₆₁ corresponding to the combined image of the dialogue partner 6 ₂ installing the shielding portion 7. Two spaces separated by the shielding part 7, the space ₈₁ of each user ₆₁ side, shows the space ₈₂ of the dialogue partner 6 ₂ side. User 6 ₁ being in the space ₈₁ can not see the space ₈₂ of the dialogue partner side directly by the shielding part 7.

The shared object 2 is installed so as to connect the two spaces 8 ₁ and 8 ₂ . Shared object 2 is intended to operate in accordance with the dialogue partner 6 ₂ of body movements by a drive mechanism such as an actuator. User 6 ₁ can feel the body movements dialogue partner 6 ₂ haptically through shared objects 2. Body movements dialogue partner 6 _2, spatial compounding apparatus dialogue partner 6 ₂ side which is remotely located (not shown) is acquired by the motion tracking technology and sensor technology, the general communication technologies, dialogue partner 6 with ₂ video 6 _'2 is transmitted to the spatial compounding device of the user ₆₁ side (not shown). In spatial synthesis device of the user ₆₁ side performs control of the shared object 2 Synthesis of video.

In the example of FIG. 1, a chaise lounge shared objects 2 loveseat, by installing a shielding portion 7 in the center of the chaise longue, from the user ₆₁ sitting on one end so can not see the other end ing. Another example of the shared object 2 is a handrail. Established the shielding portion 7 in the center of the handrail, leave as a user ₆₁ according the one end and the other end is not visible, it is to provide a vibration to the handrail in accordance with the body movements dialogue partner 6 ₂ You can also.

  The shared object 2 can be configured to present the presence of the conversation partner not only through tactile sense but also through visual and auditory senses. For example, if the shared object 2 is a chaise longue and the user is standing by the chaise longue, or if the shared object 2 is a handrail and the user is standing in front of the handrail, the chaise longue or handrail By vibrating, it is possible to feel the body movement of the conversation partner through vision or hearing even when the user does not touch the shared object.

In the example of FIG. 1, the shielding part 7 is a partition made of an opaque material. Screen, the operation of the shared object 2 in the space ₈₂ of the dialogue partner side is an example for invisible from the user _61. The shielding unit 7 does not have to completely hide the space 8 ₂ on the conversation partner side from the space 8 ₁ on the user side. For example, the screen 7 is made of a translucent material such as frosted glass to create a space 8 ₂ on the conversation partner side. It is only necessary that the detailed state of is not understood. At this time, aside and mannequin instead of dialogue partner 6 _2, by such empty appear to frosted glass over, it is possible to promote the illusion of the user. In addition, by placing an Android that moves in synchronization with the body movement of the conversation partner instead of the mannequin doll, the illusion of the user can be further effectively promoted.

In the example of FIG. 1, the user ₆₁ and the dialogue partner 6 ₂ are each alone, but shared object 2 was a chaise lounge two-seater may even user ₆₁ and the dialogue partner 6 ₂ are not alone, respectively. For example, a shared object 2 Cage user 6 ₁ two as settee three seat, the user ₆₁ sits across, sit between the user ₆₁ by giving vibration to the center of the chaise dialogue partner 6 ₂ Operation May be reproduced.

[Embodiment]
Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the component which has the same function in drawing, and duplication description is abbreviate | omitted.

With reference to FIG. 2, the structural example of the space synthesis system which concerns on embodiment is demonstrated. The space synthesis system includes n space synthesis devices 1 ₁ ,..., 1 _n installed in n (≧ 2) spaces. In the i (i = 1,..., n) -th space 8 _i , at least one person 6 _i and a shared object 2 _i exist. In the following description, it referred to as a user person 6 _i existing in i-th space _{8 i, j (j = 1} , ..., n, i ≠ j) th interaction person 6 _j present in the space 8 _j opponent Call it. It is assumed that the conversation partner is not a specific person but each of n-1 persons in other spaces. The person 6 _i is in contact with the shared object 2 _i at least at one point.

The space synthesis devices 1 ₁ ,..., 1 _n are connected to the communication network 9. The communication network 9 only needs to be configured so that the spatial compositing devices 1 ₁ ,..., 1 _n can communicate with each other. For example, a circuit switching system such as an analog or digital public telephone network or a private telephone network is used. The communication network or a packet switching communication network such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network) can be used.

  A configuration example of the space synthesis device 1 will be described with reference to FIG. The space synthesis device 1 includes a video acquisition unit 10, a motion information acquisition unit 12, a video playback unit 14, a motion playback unit 16, a shared object 2, a camera 3, a sensor 4, and a display 5. The space synthesizing device 1 is a special configuration in which a special program is read into a known or dedicated computer having a central processing unit (CPU), a main storage device (Random Access Memory, RAM), and the like. Device. For example, the space synthesis device 1 executes each process under the control of the central processing unit. The data input to the space synthesizer 1 and the data obtained in each process are stored in, for example, the main storage device, and the data stored in the main storage device is read out as necessary and used for other processing. Is done.

  The camera 3 is, for example, a video camera that captures images of a person or an object as a subject. For example, there are no restrictions on the functions that the camera 3 should have, such as the resolution that can be taken, the recording medium of the video, the presence / absence of a microphone, and whether it is a digital analog or the like. Can be used.

  The sensor 4 is, for example, a motion capture sensor that sequentially collects three-dimensional coordinates of an object. By collecting the three-dimensional coordinates of the subject of the image by the sensor 4, the position of the subject and the change in the position can be acquired. Motion capture sensors that can be used as the sensor 4 are, for example, “Japan Microsoft Corporation,“ Xbox 360-Kinect ”, [online], [searched on January 27, 2014], Internet <URL: http: // www. xbox.com/ja-JP/kinect> (Reference 1) ”.

  As shown in FIG. 3, the camera 3 and the sensor 4 may be external peripheral devices connected to the space synthesis device 1, or may be incorporated in the housing of the space synthesis device 1. Further, as shown in FIG. 3, the camera 3 and the sensor 4 may be configured as separate devices, or may be configured as one device in which the camera 3 and the sensor 4 are incorporated in one casing.

  The display 5 is, for example, a large liquid crystal display having a screen for playing back images taken by the camera 3 of the space synthesis device 1. As shown in FIG. 3, the display 5 may be an external peripheral device connected to the space synthesis device 1, or may be incorporated in one plane of the space synthesis device 1. For example, there are no restrictions on the functions that the display 5 should have, such as displayable resolution, presence / absence of speakers, digital analog, and the like, and any display can be used as long as it can play back images captured by the camera 3. be able to.

  A configuration example of the shared object 2 will be described with reference to FIG. The shared object 2 includes, for example, an actuator 20, a motor driver 22, an IO board 24, and a power source 26. The actuator 20 is connected to the IO board 24 via the motor driver 22. The IO board 24 is connected to the motion reproduction unit 16 of the space synthesis device 1. The power source 26 is connected to the motor driver 22 and supplies power. The IO board 24 operates the motor driver 22 based on a control signal input from the operation reproduction unit 16 by serial communication. The motor driver 22 controls the voltage and current input to the actuator 20 based on the control signal from the IO board 24. The actuator 20 controls the operation of the shared object 2 under the control of the motor driver 22.

  With reference to FIG. 5, a specific configuration example will be described with an example in which the shared object 2 is implemented as a chaise longue. In a chaise lounge, when a person sits next to you, when the neighbor changes his legs, or when he shakes poorly, the chaise lounge can vibrate and feel the body movement of the neighbor. This chaise longue vibration is reproduced with a shared object.

The shared object 2 implemented as a chaise longue includes a seat surface 21 and four or more legs 23, and a magnetic body 25, an elastic body 27, and an electromagnet 29 are incorporated in at least one leg 23. When a current is passed through the electromagnet 29, a magnetic force is generated, and the magnetic body 25 is pulled in the direction of the electromagnet 29. As a result, the seat surface 21 of the chaise lounge tilts toward the legs 23. For example, when the conversation partner 6 _j in the composite video is seated, the user 6 _i feels as if the conversation partner 6 _j is sitting next to the electromagnet 29 by turning on the power to the electromagnet 29. Further, when the dialogue partner 6 _j has a poor extortion, as a result of the seat surface 21 by repeating the supply to the electromagnet 29 the ON and OFF periodically is to vibrate small increments, the user 6 _i is as if next It feels like the person sitting on the bed is poverty.

  With reference to FIG. 6, an example of the processing flow of the spatial synthesis method executed by the spatial synthesis system according to the embodiment will be described in the order of procedures.

In the following description, the user person 6 _i existing in i-th space 8 _i, j-th space 8 and the dialogue partner person 6 _j present in _j, spatial compounding device 1 _i installed in the space 8 _i The operation will be mainly described. The space 8 _{j where the} conversation partner exists is not limited to one space, and may be a plurality of spaces. Even when there are dialogue partners in a plurality of spaces, the operations of the space synthesis device 8 _j installed in the other space 8 _j are the same.

In step S10 _i, the video acquisition unit 10 _i of the spatial synthesis device 1 _i may video included with the camera 3 _i spatial synthesizer 1 _i, and the user 6 _i present in the space 8 _i and the shared object 2 _i is To get. The acquired video is input to the video playback unit 14.

In step S10 _j, the video acquisition unit 10 _j of the spatial synthesis unit 1 _j, using the camera 3 _j of the spatial synthesis unit 1 _j, includes a dialogue partner 6 _j present in the space 8 _j and the shared object 2 _j Get video. The acquired video is transmitted to the space synthesis device 1 _i via the communication network 9.

In step S12 _j, the operation information acquiring unit 12 _j of the spatial synthesis unit 1 _j, using the sensor 4 _j spaces synthesizer 1 _j, acquires operation information to track the operation of the dialogue partner 6 _j. The acquired motion information is transmitted to the space synthesis device 1 _i via the communication network 9. The captured video may be encoded and compressed in an appropriate data format in order to increase the transfer efficiency during transmission.

In step S14 _i, the video reproducing section 14 _i of the spatial synthesis unit 1 _i extracts the image of the dialogue partner 6 _j from video spatial synthesis unit 1 _j is acquired. Subsequently, the video of the conversation partner 6 _j is combined with the video acquired by the video acquisition unit 10 _i , and the combined video is reproduced on the display 5 _i .

In step S16 _i , the motion reproduction unit 16 of the space synthesis device 1 _i reproduces the operation of the conversation partner 6 _j by operating the shared object 2 _i based on the operation information of the conversation partner 6 _j .

  The conventional mirror system visually reproduces the feeling that the conversation partner is next to it. However, when I looked at the neighbor, it was clear that there was no dialogue partner there, and I could not physically feel the existence of the other party. On the other hand, the spatial composition system of the present invention reproduces the feeling of having a conversation partner in the adjacent space blocked by the screen, and the conversation partner is not only visually but also tactile by a shared object that connects the two spaces. You can feel the existence of

  The space synthesis technique of the present invention combines the presence of a remote conversation partner in the user's space and makes the presence of the partner more familiar. By substituting communication in a face-to-face environment by remote communication using this invention, the cost and energy required for movement can be reduced.

  The present invention is not limited to the above-described embodiment, and it goes without saying that modifications can be made as appropriate without departing from the spirit of the present invention. The various processes described in the above embodiment may be executed not only in time series according to the order of description, but also in parallel or individually as required by the processing capability of the apparatus that executes the processes or as necessary.

[Program, recording medium]
When various processing functions in each device described in the above embodiment are realized by a computer, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, the present apparatus is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

1 Spatial composition device 2 Shared object (chair)
3 Camera 4 Sensor 5 Display 6 Person 7 Shield (Screen)
8 Space 9 Communication network 10 Video acquisition unit 12 Motion information acquisition unit 14 Video playback unit 16 Motion reproduction unit

Claims (3)

A space composition system including at least one person in each of a plurality of spaces and at least one shared object installed so as to connect the plurality of spaces, and including a space composition device installed in each of the plurality of spaces. There,
The space synthesizer is
A video acquisition unit for acquiring a video including the person and the shared object;
A motion information acquisition unit that acquires motion information tracking the physical motion of the person;
Using a person in another space as a conversation partner, extract the conversation partner's video from the video acquired by the space synthesis device in the other space, and synthesize the conversation partner's video with the video acquired by the video acquisition unit. A video playback unit for playback,
A motion reproduction unit that reproduces the physical motion of the conversation partner by operating the shared object based on the motion information of the conversation partner;
Spatial synthesis system including The spatial synthesis system according to claim 1,
The space synthesizer is
A spatial synthesis system further comprising: a shielding unit that divides the shared object into a plurality of parts and prevents other parts from being visually recognized from the part where the person is present. The space synthesis system according to claim 2,
The shared object is a chaise lounge that can be seated by two people, and the space is visually divided by a partition installed in the center of the chaise lounge, and the person is one of two spaces divided by the partition. Sitting in the chaise longue in the space,
The video playback unit synthesizes the video so that the conversation partner is seated in the other space of the chaise longue,
The space reproduction system, wherein the motion reproduction unit vibrates at least one leg of the chaise longue based on motion information of the conversation partner.

Images