RU2461143C2 - Method of creating virtual space effect with composite video - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: first image of a room in which said videoconferencing participant is located is obtained; an image of the room in which remote videoconferencing participants are located is obtained with the remote videoconferencing participants; second images are selected therefrom, which contain an image of remote videoconferencing participants; a third image is obtained by superposing the first and second images, wherein the position of second images relative the first image is selected based on conditions for creating the most realistic virtual space effect, and the third image is displayed for said videoconferencing participant.

EFFECT: creating a realistic virtual space effect without the need for using videoconferencing studios with the same interior.

4 dwg

Description

Technical field

The invention relates to methods for creating the effect of "virtual presence". The effect of virtual presence can be achieved in video telephony or video conferencing systems, which in turn are related to communication and telecommunication technology.

State of the art

Commercial samples of video conferencing systems (VKS) appeared in the mid-80s, in particular thanks to the development of PictureTel.

Known video conferencing systems provide the simultaneous transmission of video, sound and documentary information between two or more subscribers (participants) over existing digital telephone networks (ISDN) or more modern networks based on the IP protocol.

In the case when there are more than 2 participants, 2 main display modes are used: the “active participant” mode, when only the currently speaking (active) participant in the video conferencing session is visible on the screen; “permanent presence” mode, when all remote participants of a videoconferencing communication session are simultaneously visible.

The mode of "constant presence" is more organic for perception, as it is close in format to a regular meeting, when participants are in the same room, for example, a meeting room, and continuously see each other. In the simplest case, the “permanent presence” mode is implemented by splitting the screen (or several screens) of the video conferencing system into rectangular, often equal to each other, areas and display images of remote participants separately in each screen area. Obviously, such a widely used method is far from perfect, since the image on the screen does not look much like the usual prototype - the kind of participants in a meeting in a meeting room. It should be noted that with a large number of participants a combination of the “permanent presence” and “active participant” modes is possible.

To increase realism, i.e. To create the illusion that all remote participants are in the same room, the concept of “virtual presence” or “telepresence” has been put forward.

Participants who are in the video conferencing room (studio) under consideration are called local, and those who are in remote studios relative to the considered one are called remote.

To ensure the effect of “virtual presence” (VP), the following conditions are fulfilled, in particular (simultaneously or separately): coincidence of the direction of view (remote and local participants); correspondence of the position and size of images of remote participants in relation to local participants; Identity of the interior visible on the screen (remote and local).

The coincidence of the direction of the gaze of the remote and local participants (conditionally against each other) increases emotional contact and interactivity between the interlocutors during the conversation due to the well-known psychological effect of “eye contact” - “eye contact”.

The coincidence of position and size is also important, since for the sake of completing the effect of VP, it is desirable that the participants are displayed in full size and in the interior corresponding to the real meeting room, for example, at the negotiating table.

In addition to the remote participants, the interior of a remote or remote video conferencing studio is usually visible on the display device. Obviously, to create the effect of the VP, the identity of the interior of the remote studio and the interior of the local studio visible on the screen becomes important. It is advisable to achieve the complete identity of the interior elements (wall decoration, furniture, lighting, etc.) of the parts of the local and remote studios that fall into the field of view of the cameras. However, this task to date has not had an effective technical solution.

The analogue - the Cisco Telepresence system (Networks, 2006, No. 17, the article “Distances do not matter”) is an improved VKS system, which, unlike the known ones, allows creating the illusion of the presence of remote subscribers in one room. This effect of "virtual" presence significantly increases the comfort of communication and, consequently, the effectiveness of negotiations.

The room for the operation of the system is equipped with the following components: a conference table, a videoconferencing terminal with one or more monitors and a video camera installed in a certain way, as well as a high-quality speech reception and playback system.

The effect of the VP is created in the following way: on the one side of the negotiation table the places of local subscribers are placed, on the opposite side of the table, in the intended location of the remote subscribers, 1 or more monitors and video cameras are installed. When establishing a communication session, remote subscribers are displayed on one or more monitors in full size.

As you can see, the Cisco Telepresence system successfully solves the problem of matching the position and size of images.

The task of matching the interior visible on the screen in this system is solved by creating physically identical VP studios, which significantly increases the cost of the system itself, requires specialized premises, and limits subscribers in choosing the decoration and decor.

The problem of matching the direction of view in this system has not been solved, since the system in each studio has a single video camera and, accordingly, a single point of "look" for all remote participants.

An analogue selected as a prototype is described in US patent No. 4400724 "Virtual space teleconference system".

Due to the use, in addition to the places of real participants, located on the conference table of a system of N display devices and N video cameras forming N places of “surrogate” participants, this invention solves two tasks of the VP: matching the direction of view (remote and local participants); correspondence of the position and size of images (deleted participants in relation to local participants);

However, the third task - the identity of the interior visible on the screen (remote and local) is not considered.

The aim of the proposed technical solution is to create a virtual presence effect with a combined video image, a realistic virtual presence effect without the need for video conferencing studios with an identical interior.

This goal is achieved due to the fact that in the method of creating the effect of virtual presence in one room of mutually remote videoconferencing participants, the first image containing the image of the room in which the given video conferencing participant is located is obtained, the image of the premises where the remote videoconferencing participants are located with the remote participants is received video conferencing, allocate second images from them, which contain the image of the remote participants in the video conferencing, receive the third image by combining the first image and the second image, optimize the third image according to the criterion of realism, demonstrate the third image to this participant of video conferencing.

Disclosure of invention

To create a realistic effect, VPs use the transmission of video, sound and documentary information.

The methods of transmitting sound and documentary information when creating the VP effect operate in a manner known in the videoconferencing systems (as shown in Section 2.1), therefore, to disclose the method of creating the “virtual presence” effect with a combined video image, it is sufficient to consider only the transmission and formation of the video image in the process videoconferencing session.

Let us agree to call the room where the image of the videoconferencing system with the effect of VP is viewed as the local studio, and the room (or premises) where the videoconferencing of the participants of the videoconferencing with the effect of VP is called the remote studio (or remote studios).

In this case, the room includes the following elements: building shell (floor, walls, ceiling); subject content (equipment, furniture, decoration elements); functional appliances (lighting, air conditioning, communications).

To organize a two-way videoconferencing session with the VP effect, they perform the actions described below, taking each studio both local and remote with respect to all other studios.

The method of creating the effect of virtual presence with a combined video image allows you to effectively solve the problem of the location of remote participants in the interior of the local studio and the identity of the visible part of the interior of the remote (remote) studios and the interior of the local studio.

The method of creating the effect of virtual presence with a combined video image is different in that the real (physically existing) interiors and the environment of the local studio and remote studios may not coincide.

The essence of the method of creating a virtual presence effect with a combined video image is that on the display devices located in the local studio, they show the image of the remote or remote participants, combined with the image of the local studio. For this, from the full image of that part of the remote studio, which is in the field of view of the image capturing means, a second image is isolated. The second image in this application means an image of a remote participant or remote participants, isolated from the full image of that part of the remote studio, which is in the field of view of the image capture means. The first image in this application means the image of the room in which the participant in the video conferencing is located, i.e. image of the corresponding part of the local studio. By combining the first image and the second image, a third image is obtained. Moreover, to obtain a third image, a second image (a selected image of the remote (deleted) participants) is superimposed on the first image. The third image is optimized by the criterion of realism. In this case, the second image (images of the remote (remote) participants) is positioned relative to the first image (the corresponding fragment of the local studio image) based on the conditions for creating the most realistic effect of the VP, for example, the condition for the direction of view to coincide.

The third image (the image of the remote participants combined with the image of the local studio) may consist of several mutually overlapping plans. In particular, from three plans: the background is part of the first image, i.e. the part of the local studio that serves as the background, the middle plan is the second image, i.e. image of the remote participant, the foreground is another part of the first image, i.e. local studio content items that may partially overlap the remote participant’s image and background.

Remote participants can be divided into groups, and the third image, i.e. the combined image of the remote participants and the image of the local studio can be obtained separately for each member of each group and shown on a separate display tool in the local studio.

As equivalent to the image of the local studio, i.e. the first image, can be used: captured and digitally recorded image of a real local studio; image obtained by artistic graphics; digitally-indicated image of a real local studio with the removal or addition of individual elements; image obtained by computer simulation methods, similar visually with the image of a real studio; a digitally captured and digitally captured image using art graphics that is visually similar to a real studio image.

Brief Description of the Drawings

The implementation of the method is illustrated by drawings, where Fig.1 and Fig.1a is a diagram according to option 1 with the allocation of the second image of the remote subscriber and the overlay on the first image of the local studio by means of a remote studio; figure 2 - diagram according to option 2 with the allocation of the second image of the remote subscriber and overlay on the first image of the local studio by means of the local studio; figure 3 is a diagram according to option 3 with the allocation of the second image of the remote subscriber by means of a remote studio and overlay on the first image of a local studio by means of a local studio.

The implementation of the invention

Consider the process of transmitting and forming a video image of a participant to create the effect of VP in one of the directions - from a remote VKS studio to a local VKS studio - represented by the following chain of transmitting and generating a video image:

Remote videoconferencing studio with remote subscriber (s) -> transmission medium -> local videoconferencing studio with local subscriber (s).

In general, the equipment of a local and remote VKS studio that implements a virtual presence effect with a combined video image may include the following elements: interior of a VKS remote studio (furniture, decoration) for comfortable accommodation of a participant (participants) of VKS 1 session; video cameras for shooting a remote participant in the interior of a remote studio 2; means for generating an image of a local studio, including a video camera or electronic camera 8 and / or a computer with image 10 prepared or received in the course of a videoconferencing session; display means, including: electron beam monitors, plasma panels, liquid crystal screens, front and rear projection screens 3; means for extracting the image of the remote participant (subscribers) from the image of the interior of the remote studio 4; overlay the selected image of the remote participant on the interior of the local studio 5; means for encoding / decoding an image and interfacing with information transmission equipment 6; the interior of the local studio (furniture, decoration) for comfortable accommodation of the participant (participants) of VKS 9; image storage means 10.

The transmission medium 7 includes well-known network and channel-forming equipment necessary for transmitting encoded video conferencing images between studios. In particular, an IP network, ISDN or any other network suitable for transmitting real-time video images and providing the necessary bandwidth can be used as an information transmission medium.

The direction of the formation and transmission of the full video image from the remote VP studio to the local studio is shown by a solid line.

The direction of transmission of the first image (part of the interior of the local VP studio), necessary for the formation of the third (full) image, is shown by the dotted line.

There are three possible ways to implement the virtual presence effect with a combined video image:

1. The selection of the image of the remote participant (second image) and overlay on the image of the local studio (first image) by means of a remote studio (figure 1, figa).

Isolation of the image of the remote participant (second image) is carried out by means of 4 remote studios. In this case, well-known image extraction methods are used, in particular, the Chromo key method. The selected second image of the remote participant is superimposed on the elements of the first image of the interior of the local studio 9, necessary to replace the image in the field of view of the remote video camera 2 means 5 of the remote studio. The first image of the local studio is created by means of 8 and is continuously transmitted to the remote studio during the session (Fig. 1).

A variant is possible when the image of the local studio (the first image) is transferred to a remote studio before conducting a VK session or is permanently stored by means of a remote studio 10 (Fig. 1a).

2. The selection of the image of the remote participant (second image) and overlay on the image of the local studio (first image) by means of the local studio (figure 2)

Isolation of the image of the remote participant (second image) is performed by means of 4 local studios. In this case, well-known image extraction methods are used, in particular, the Chromo key method. The selected image of the remote participant (second image) is superimposed on the elements of the first image of the interior of the local studio 9, necessary to replace the image in the field of view of the remote video camera 2 means 5 of the local studio. The specified image elements of the local studio (first image) can be prepared in advance or obtained in the process of the session by means of 8.

3. The selection of the image of the remote participant (second image) by means of a remote studio and overlay on the image of the local studio (first image) by means of the local studio (figure 3).

Isolation of the image of the remote participant (second image) is carried out by means of 4 remote studios. In this case, well-known image extraction methods are used, in particular, the Chromo key method. The selected image of the remote participant (second image) is superimposed on the elements of the first image of the interior of the local studio 9, necessary to replace the image in the field of view of the remote video camera 2 means 5 of the local studio. The specified image elements of the local studio (first image) can be prepared in advance or obtained in the process of the session by means of 8.

Thus, the introduction of the characteristics indicated in the formula of the invention ensures the achievement of the claimed technical result.

Claims (1)

A method of creating the effect of the virtual presence of mutually remote videoconferencing participants in one room, characterized in that the first image of the room in which the given video conferencing participant is located, the image of the rooms where the remote video conferencing participants are located, with the remote videoconferencing participants, is extracted from them second images that contain the image of the remote participants of the video conferencing, get the third image by combining the first of the image and the second image, the positions of the second image relative to the first image is selected from the conditions to create the most realistic effect of virtual presence, and the third image demonstrate this party videoconference.

Images