JP2023134045A - Communication apparatus, communication system, display method, and program - Google Patents

Abstract

To solve the following problem: conventionally, kinematic parallax is proposed in which a video image of a partner located in another place is changed in accordance with a point of view (eye positions) in a person on a real space located in one place, to display the partner with a person size in accordance with a distance on a virtual space, on a display in the one place, thereby allowing users to feel as if they are present in the same room with their partners, however, not all the persons on the virtual space hold a conference, or the like, and some virtual users on the virtual space may feel someone's eyes watching and feel discomfort.SOLUTION: A communication apparatus selectively displays an image of a subject located in another place or a character image related to the subject, in accordance with a distance between one place and the other place on a virtual space.SELECTED DRAWING: Figure 34

Description

The present disclosure relates to a communication device, a communication system, a display method, and a program.

In recent years, telework, which uses ICT (information and communication technology) to enable flexible working styles that allow effective use of time and location, has become widespread. This makes it possible to achieve work-life balance, improve sales efficiency, and improve customer satisfaction. Furthermore, with the spread of telework, the performance of systems such as video (teleconferencing) via the Internet is improving (see FIG. 34).

Furthermore, in communication, the sense of distance between people is very important because it affects the ease of speaking and the sense of tension (Edward Hall, Hidden Dimension, 2000, Misuzu Shobo). However, video conferencing has a problem in that, unlike communication in real space (or physical space), it is difficult to get the feeling of being in the same space as the person with whom you are communicating (hereinafter referred to as a "same-room feeling").

Therefore, in Patent Document 1, motion parallax is realized in which the image of the conversation partner at another base is changed according to the viewpoint (eye position) of a person in real space at one base, and the distance in virtual space is A technology has been disclosed that can give each user a feeling of being in the same room as the conversation partner by displaying the conversation partner on a display at one base in a size corresponding to the size of the person.

Patent No. 5833525

However, not all people in the virtual space necessarily hold a meeting, etc., and some virtual users in the virtual space may not know who is watching them, causing a problem of not feeling secure.

The invention according to claim 1 is a communication device that makes a call with another communication device at another base, the other communication device transmitting image data of the other base obtained by photographing a subject at the other base. A receiving means for receiving the image data transmitted by the other communication device, and a receiver for receiving the image data at the other base based on the received image data according to the distance between the own base and the other base in the virtual space. A communication device characterized by having a display control means that switches and displays an image and a character image related to the subject.

As explained above, according to the present invention, the communication device switches and displays an image of a subject at another base and a character image related to the subject, depending on the distance between the own base and the other base in the virtual space. , it is possible to eliminate as much as possible the problem that virtual users in a virtual space do not feel safe because they do not know who is watching them.

1 is a schematic diagram of a communication system according to an embodiment of the present invention. FIG. 2 is an electrical hardware configuration diagram of a communication management server. FIG. 2 is an electrical hardware configuration diagram of a communication device. FIG. 2 is a functional block diagram of a communication system. FIG. 2 is a functional block diagram of a communication system. This is a conceptual diagram of a communication management table. FIG. 3 is a conceptual diagram of a session management table. FIG. 3 is a conceptual diagram of a display management table. FIG. 2 is a conceptual diagram visually showing data managed by a display management table using points on a display. FIG. 3 is a conceptual diagram of a selective placement information management table. FIG. 3 is a conceptual diagram of a fixed placement information management table. FIG. 2 is a conceptual diagram of a location management table showing the latest location of a virtual user. FIG. 2 is a conceptual diagram showing the placement positions of virtual participants on world coordinates. FIG. 2 is a conceptual diagram showing the placement positions of virtual participants on world coordinates. FIG. 3 is a sequence diagram showing a process for participating in a specific communication session. It is a display example of a login screen in a communication device. 7 is a flowchart showing a position specifying process. FIG. 3 is a diagram illustrating a selection screen for a type of arrangement position in a communication device. FIG. 3 is a diagram illustrating a display example of a selection screen for selecting a virtual company. FIG. 7 is a diagram illustrating a display example of a selection screen for selecting a virtual department. FIG. 6 is a diagram illustrating a display example of a selection screen for selecting a virtual room (floor). FIG. 3 is a sequence diagram showing display information transmission processing. FIG. 2 is a sequence diagram showing processing until luminance image data, depth image data, and sound data transmitted from a predetermined communication device (for example, communication device 1a) reach another communication device. FIG. 3 is a conceptual diagram of brightness image data and depth image data that are transmitted. FIG. 2 is a sequence diagram showing a process of receiving image data and sound data by the communication device 1d. 7 is a flowchart showing image display processing. It is a diagram showing an example of a distance from a virtual user D1 and an image to be displayed. 3 is a flowchart illustrating processing for calculating the position of an object in the world coordinate system. It is a conceptual diagram of transformation from a camera coordinate system to a modeling coordinate system, and transformation from a modeling coordinate system to a world coordinate system. It is a conceptual diagram in which an image of a user D1 and a virtual display are arranged in a modeling coordinate system. FIG. 3 is an image diagram showing conversion from a modeling coordinate system to a world coordinate system. It is a figure which shows the image displayed on the display 118 of the user D1's communication device. It is a figure which shows the image displayed on the display 118 of the user D1's communication device. It is an example of a display on a display in the case of a conventional video conference.

Embodiments of the present invention will be described below with reference to the drawings.

[Overview of communication system]
The communication system of this embodiment is a system for performing video calls in a virtual space (also referred to as a "virtual area").

First, the outline of the configuration of the communication system of this embodiment will be explained using FIG. 1. FIG. 1 is a schematic diagram of the configuration of a communication system according to an embodiment of the present invention.

As shown in FIG. 1, the communication system of this embodiment is constructed by a plurality of communication devices 1a, 1b, 1c, 1d and a communication management server 6, and communicates via a communication network 100 such as the Internet. be able to. The connection form of the communication network 100 may be either wireless or wired.

Communication devices 1a, 1b, 1c, and 1d are located at bases A, B, C, and D, respectively. Note that, hereinafter, any communication device among the communication devices 1a, 1b, 1c, and 1d will be referred to as "communication device 1." The communication device 1 can photograph and collect sound at its own base, and can output images obtained by photographing at another base and sounds obtained by collecting sounds at other bases. The communication device 1 will be explained in detail later.

Further, FIG. 1 shows a case where communication devices 1a, 1b, 1c, and 1d are used by users A1, B1, C1, and D1, respectively. For example, the communication device 1a photographs and records the state of its own base A, and transmits the image to the communication devices 1b, 1c, and 1d of other bases B, C, and D via the communication network 100 and the communication management server 6. Send data and sound data. Similarly, the communication devices b1, 1c, and 1d photograph and record the situation at their own base, and transmit image data and sound data to communication devices at other bases via the communication network 100 and the communication management server 6. do.

The communication management server 6 is configured by a computer, and manages and controls communication of the communication device 1 and manages the types of image data (general images and special images) to be transmitted and received. Therefore, the communication management server is also a communication control server. Note that the communication management server 6 is configured by a single computer or a plurality of computers. When configured by multiple computers, the communication management server is also a communication management system (or communication control system).

Note that the communication device 1 also includes a digital television, a smartphone, a smart watch, a car navigation device, a medical device, and the like.

[Hardware configuration]
Next, the hardware configurations of the communication device 1 and the communication management server 6 of this embodiment will be described in detail using FIGS. 2 and 3.

<Hardware configuration of communication management server>
Next, an electrical hardware configuration diagram of the communication management server will be described using FIG. 2. FIG. 2 is an electrical hardware configuration diagram of the communication management server.

As shown in FIG. 2, the communication management server 6 includes, as a computer, a CPU 601, ROM 602, RAM 603, HD 604, HDD (Hard Disk Drive) controller 605, display 606, external device connection I/F (Interface) 608, It includes a network I/F 609, a bus line 610, a pointing device 612, and a media I/F 614.

Among these, the CPU 601 controls the entire operation of the communication management server 6. The ROM 602 stores programs used to drive the CPU 601 such as IPL. RAM 603 is used as a work area for CPU 601. The HD 604 stores various data such as programs. The HDD controller 605 controls reading and writing of various data to the HD 604 under the control of the CPU 601. The display 606 displays various information such as a cursor, menu, window, characters, or images. External device connection I/F 608 is an interface for connecting various external devices. The external device in this case is, for example, a USB memory or a printer. The network I/F 609 is an interface for data communication using the communication network 100. The bus line 610 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 601 shown in FIG. 2.

Further, the pointing device 612 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The media I/F 614 controls reading or writing (storage) of data to a recording medium 613 such as a flash memory. The recording media 613 also includes DVDs, Blu-ray Discs, and the like.

<Hardware configuration of communication device>
Next, an electrical hardware configuration diagram of the communication device will be described using FIG. 3. FIG. 3 is an electrical hardware configuration diagram of the communication device.

The communication device 1 is, for example, a tablet terminal. As shown in FIG. 3, the communication device 1 includes a CPU 101, a ROM 102, a RAM 103, an EEPROM 104, a photographing unit 105, an image sensor I/F 106, an acceleration/direction sensor 107, a media I/F 109, and a GPS receiving section 111. ing.

Among these, the CPU 101 controls the operation of the communication device 1 as a whole. The ROM 102 stores the CPU 101 and programs used to drive the CPU 101 such as IPL. RAM 103 is used as a work area for CPU 101. The EEPROM 104 reads or writes various data such as programs for the communication device 1 under the control of the CPU 101. The imaging unit 105 is a type of built-in imaging means that images a subject (mainly a self-portrait) and obtains image data under the control of the CPU 101. The imaging device I/F 106 is a circuit that controls driving of the imaging unit 105. The acceleration/direction sensor 107 is a variety of sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor that detect geomagnetism. The media I/F 109 controls reading or writing (storage) of data to a recording medium 108 such as a flash memory. GPS receiving section 111 receives GPS signals from GPS satellites.

The communication device 1 also includes a long-distance communication circuit 112, an imaging unit 113, an imaging device I/F 114, a microphone 115, a speaker 116, a sound input/output I/F 117, a display 118, an external device connection I/F (Interface) 119, It includes a short-range communication circuit 120, an antenna 120a of the short-range communication circuit 120, and a touch panel 121.

Among these, the long-distance communication circuit 112 is a circuit that communicates with other devices via the communication network 100. The imaging unit 113 is a type of built-in imaging means that images a subject and obtains image data under the control of the CPU 101, and will be described in detail later. The imaging device I/F 114 is a circuit that controls driving of the imaging unit 113. Microphone 115 is a built-in circuit that converts sound into electrical signals. The speaker 116 is a built-in circuit that converts electrical signals into physical vibrations to produce sounds such as music and voice. The sound input/output I/F 117 is a circuit that processes input/output of sound signals between the microphone 115 and the speaker 116 under the control of the CPU 101. The display 118 is a type of display means such as a liquid crystal display or an organic EL (Electro Luminescence) display that displays an image of a subject, various icons, and the like. The external device connection I/F 119 is an interface for connecting various external devices. The near field communication circuit 120 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 121 is a type of input means by which the user operates the communication device 1 by pressing the display 118.

The communication device 1 also includes a bus line 110. The bus line 110 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 101 shown in FIG. 4.

Here, the imaging unit 113 will be explained. The imaging unit 105 includes a brightness camera 105a, a depth camera 105b, and a bus line 105e that electrically connects these to the imaging element I/F. Among these, the brightness camera 105a is a device that converts received light into an electrical signal and obtains image data using an image sensor such as a CCD or CMOS. For example, an RGB color camera is included.

The depth camera 105b acquires depth image data by capturing an image of a subject, such as a person or object, and measuring the distance from the subject to the depth camera 105b using a laser method using LiDAR (Light Detection And Ranging). The depth camera 105b is a camera that can capture the shape of a subject as a three-dimensional object, detect various movements such as a human skeleton, and reflect the detected movements in subsequent processing in real time. Therefore, the depth camera 105b includes a laser transmitter 105b1 and a laser receiver 105b2. The laser transmitter 113b1 emits laser light. The laser beam emitted by the laser transmitter 105b1 is reflected by the object, and the laser receiver 113b2 receives the reflected laser beam, converts it into an electrical signal, and extracts necessary information. Note that although the above describes a method of measuring depth using LiDAR, the method is not limited to this. For example, a method of measuring depth using a TOF (Time of Flight) method may be used. Further, the depth may be measured using a stereo measurement method using an RGB color camera that is a normal camera, and the method of measuring the depth is not limited to the above description.

Note that the imaging unit 113 includes a brightness camera 113a, a depth camera 113b, and a bus line 113e that electrically connects these to the imaging element I/F. Further, the depth camera 113b further includes a laser transmitter 113b1 and a laser receiver 113b2. The brightness camera 113a, the depth camera 113b, the bus line 113e, the laser transmitter 113b1, and the laser receiver 113b2 are the brightness camera 105a, the depth camera 105b, the bus line 105e, the laser transmitter 105b1, and the laser in the photographing unit 105, respectively. Since the configuration is the same as that of the receiver 105b2, a description thereof will be omitted.

Note that recording media such as CD-ROMs in which the above programs are stored and HDs in which these programs are stored can both be provided domestically or internationally as program products.

[Functional configuration of embodiment]
Next, the functional configuration of this embodiment will be explained using FIGS. 4 to 14. 4 and 5 are functional block diagrams of the communication system.

<Functional configuration of communication device>
Here, the functional configuration of the communication device 1a of the communication device 1 will be explained.

As shown in FIG. 4, the communication device 1a includes a detection section 30a, a transmission/reception section 31a, a reception section 32a, an image/sound processing section 33a, a display control section 34a, a judgment section 35a, a sound collection section 38a, a storage It has a read processing section 39a and a detection section 40. Each of these units is a function or means that is realized when any of the components shown in FIG. It is. Furthermore, the communication device 1a has a storage unit 3000a constructed from a ROM 102, a RAM 103, and an EEPROM 104 shown in FIG.

Furthermore, the communication device 1a has a brightness image acquisition section 36a, a depth image acquisition section 37a, and a sound collection section 38a as functions realized by operating the photographing unit 105 in accordance with instructions from the CPU 101. Functions realized by the photographing unit 105 are surrounded by broken lines.
(Position management table)
A location management DB 3007a composed of a location management table is constructed in the storage unit 3000a. Note that since the location management DB 3007a has the same data structure as the location management DB 6007 of the communication management server 6, the location management DB 6007 will be explained later.

(Functional configuration of communication device)
Next, each functional configuration of the communication device 1a will be explained in more detail using FIGS. 3 and 4.

The detection unit 30a of the communication device 1a is realized by a command from the CPU 101 to the acceleration/azimuth sensor 107, and detects the moving direction and rotation direction of the communication device 1a.

The transmitting/receiving unit 31a is realized by the processing of the CPU 101 on the long distance communication circuit 112, and transmits and receives various data (or information) to and from other devices (for example, the communication management server 6) via the communication network 100. .

The reception unit 32a is mainly realized by the processing of the CPU 101 on the touch panel 121, and receives various selections or inputs from the user.

The image/sound processing section 33a is mainly realized by the processing of the CPU 101, and performs various image processing and sound processing. For example, the image/sound processing unit 33a plays a role as a generation unit and generates a three-dimensional object of a subject in virtual space based on data of a luminance image and a depth image.

The display control unit 34a is mainly realized by the processing of the CPU 101, and causes each display 118 to display various images.

The determining unit 35a is mainly realized by the processing of the CPU 101, and performs various determinations.

The brightness image acquisition unit 36a is mainly realized by the processing of the CPU 101 for the brightness camera 105a, and acquires brightness image data by photographing a subject. Here, although it is described as a brightness image, this is a term used for comparison with a depth image, and a brightness image is not an image containing only brightness information, but also has color information.

The depth image acquisition unit 37a is mainly realized by the processing of the CPU 101 for the depth camera 105b, and acquires depth image data by photographing a subject.

The sound collection unit 38a is mainly realized by the processing of the CPU 101 for the microphone 115, and collects sound to obtain sound data.

The storage/read processing unit 39a is mainly realized by the processing of the CPU 101, and stores various data (or information) in the storage unit 3000a and reads various data (or information) from the storage unit 3000a.

Note that the communication devices 1b, 1c, and 1d have the same functions, storage units, and databases as the communication device 1a. Further, the functions of the communication devices 1b, 1c, and 1d are substantially realized by the PCs 3b, 3c, and 3d, respectively. In FIGS. 4 and 5, the end of the code is only changed from a to b, c, and d so that the transmitting/receiving section 31a in the communication device 1a corresponds to the transmitting/receiving section 31b in the communication device 1b. The explanation of is omitted.

<Functional configuration of communication management server>
Next, the functional configuration of the communication management server 6 will be explained using FIGS. 2 and 4.

As shown in FIG. 4, the communication management server 6 includes a transmitting/receiving section 61, a determining section 65, a creating section 66, and a storage/reading processing section 69. Each of these units is realized by one of the components shown in FIG. 2 operating in accordance with an instruction from the CPU 601 according to a communication management (control) program expanded from the HD 604 onto the RAM 603. It is a function or means. Further, the communication management server 6 has a storage unit 6000 constructed by a ROM 602, a RAM 603, and an HD 604 shown in FIG.

(Communication management table)
FIG. 6 is a conceptual diagram showing the communication management table. In the storage unit 6000, a communication management DB 6001 is constructed, which includes a communication management table as shown in FIG. In this communication management table, each connection ID of all the communication devices 1 managed by the communication management server 6 is managed in association with a password, an IP address, and a user name. For example, in the communication management table shown in FIG. 6, the connection ID of the communication device 1a is "01aa", the password is "aaaa", the IP address is "1.2.1.3" as IPv4, and the name is shown to be "Hanako R". Note that the IP address is shown simply. Further, the IP address may be IPv6 instead of IPv4. Furthermore, instead of an IP address, it may be a domain name (a character string indicating a name used as an e-mail or home page address).

(Session management table)
FIG. 7 is a conceptual diagram showing the session management table. A session management DB 6002 configured by a session management table as shown in FIG. 7 is constructed in the storage unit 6000. In this session management table, session IDs and connection IDs of communication devices that have participated in video calls in the same communication session are stored and managed in association with each other. Among these, the session ID is an example of session identification information for identifying a communication session for realizing a video call, and is generated for each virtual area such as a virtual conference room. It may be possible to substitute the session ID as the meeting ID. The meeting ID indicates an identifier for managing a meeting (a discussion between two or more people) on the communication management server.

(Display management table)
FIG. 8 is a conceptual diagram showing the display management table. A display management DB 6004 configured by a display management table as shown in FIG. 8 is constructed in the storage unit 6000. This display management table includes the vendor ID and product ID of the GUID (Globally Unique Identifier) of the display, the vertical and horizontal sizes (positions of each corner) and resolution of the display in real space (real coordinate system), and the display in virtual space. The positions of each corner of the display (modeling coordinate system) are stored and managed in association with each other. Note that the real coordinate system refers to a coordinate system based on an arbitrary position in a three-dimensional world that is the real world.

Here, using FIG. 9, positions indicated by data managed in the display management table will be visually explained using points on the display. FIG. 9 is a conceptual diagram visually showing the positions indicated by the data managed in the display management table as points on the display.

For example, when the GUID of the display is "vid_10ca&pid_0001", points indicating the positions of each of the four corners of the display 118 in physical space (real coordinate system) are shown. Similarly, when the GUID of the communication device is "vid_10ca&pid_0001", points indicating the positions of the four corners of the display in the virtual space (modeling coordinate system) are shown.

Further, when the GUID of the display is "vid_11ca&pid_0010", points indicating the positions of each of the four corners of one display of the communication device 1 in physical space (real coordinate system) are shown. Similarly, when the GUID of the communication device 1 is “vid_11ca&pid_0010”, points indicating the positions of the four corners of one display of the communication device 1 in the virtual space (modeling coordinate system) are shown.

Further, when the GUID of the display is "vid_12ca&pid_0100", points indicating the positions of each of the four corners of one display of a general notebook PC in physical space (real coordinate system) are shown. Similarly, when the GUID of the display is "vid_12ca&pid_0100", points indicating the positions of the four corners of one display of the notebook PC in the virtual space (modeling coordinate system) are shown.

(Selection placement information management table)
FIG. 10 is a conceptual diagram showing the selected placement information management table. In the storage unit 6000, a selective placement information management DB 6005 is constructed, which is constituted by a selective placement information management table as shown in FIG. In this selective placement information management table, selective placement information is configured for each record, and the reference number (NO), company name, department name of this company, and room (floor) of this department are managed in association.
Furthermore, each room is managed in association with a session ID (communication ID) and conversion parameters from Model coordinates to World coordinates. The conversion parameter in this case indicates a specific position when a virtual specific user enters the virtual room (floor). For example, it is the position where the virtual desk (chair) of a specific virtual user is placed. Note that the session ID (communication ID) of the selected placement information management DB 6005 has the same concept as the item (attribute) of the "session ID (communication ID)" of the session management DB 6002 described above, so the explanation will be omitted.

(Fixed placement information management table)
FIG. 11 is a conceptual diagram showing the fixed placement information management table. In the storage unit 6000, a fixed placement information management DB 6006 is constructed, which is composed of a fixed placement information management table as shown in FIG. In this fixed location information management table, fixed location information is configured for each record, and a connection ID of a communication device, a session ID (communication ID), and a conversion parameter from Model coordinates to World coordinates are managed in association with each other. The conversion parameter in this case indicates a specific position when a virtual predetermined (arbitrary) user enters the virtual room (floor). For example, it is the entrance of a virtual room. Note that the session ID (communication ID) of the fixed placement information management DB 6006 has the same concept as the item (attribute) of the "session ID (communication ID)" of the above-mentioned session management DB 6002, so a description thereof will be omitted.

(Position management table)
FIG. 12 is a conceptual diagram of a location management table showing the latest location of a virtual user. In the storage unit 6000, a location management DB 6007 is constructed, which is composed of a location management table as shown in FIG. While the above-mentioned selective placement information management DB 6005 and fixed placement information management DB 6006 are static databases, the location management DB 6007 is a dynamic database. Further, the location management DB 6007 copies data managed by the selective placement information management DB 6005 and the fixed placement information management DB 6006.

In the position management table, a connection ID of a communication device, a session ID (communication ID), a conversion parameter from a Model coordinate system to a World coordinate system, and a movement vector are managed in association with each other. In this case, the conversion parameters are managed by copying specific conversion parameters from among the conversion parameters managed in the selected placement information management DB 6005, or each conversion parameter managed in the fixed placement information management DB 6006. Managed by copying certain conversion parameters. In FIG. 19, which will be described later, when the "selected placement" button b1 is selected, specific conversion parameters managed in the selected placement information management DB 6005 are copied to the location management DB 6007. Further, in FIG. 18, which will be described later, when the "fixed placement" button b2 is selected, specific conversion parameters managed in the fixed placement information management DB 6006 are copied to the location management DB 6007. Furthermore, in these cases, the connection ID and session ID of the logged-in communication device are also copied, along with the specific conversion parameters that are copied and managed.

Furthermore, in the position management table, a movement vector is also managed for each connection ID, that is, for each virtual user, to indicate the position to which the virtual user has moved from the copied and managed position to the position to which the virtual user has moved within the virtual space. ing.

The conversion parameters from the Model coordinate (modeling coordinate) system to the world coordinate system (World coordinate) are used to transform an object from the modeling coordinate system in the first virtual space to the second virtual space, as shown in FIG. Indicates the conversion parameters used when converting to the world coordinate system in space.

Note that a virtual world drawn using computer graphics (CG) has one coordinate system that serves as a reference for the world. This is called the world coordinate system. If the shape of an object is given directly in the world coordinate system, the shape of the object must be redefined each time the object moves within the world coordinate system. Therefore, a coordinate system (called modeling) has been established to determine the shape of an object using three-dimensional computer graphics images. This coordinate system is called a modeling coordinate system.

Here, world coordinates and modeling coordinates will be explained using different expressions. The world coordinate system is a coordinate system in which objects displayed in computer graphics represent the entire space. A coordinate system that indicates the position of an object in space, and is used to handle the placement and movement of objects. It is possible to describe the position and orientation of each solid. Everything is in one big space called the world coordinate system, and if all objects are defined on this coordinate system from the beginning, it will be difficult to handle the display process.Therefore, the shape and deformation of each solid placed in the space will be difficult. Modeling coordinates are used to handle the The origin is set at one of the vertices or the vicinity of the solid, and the positions and orientations of the points, lines, surfaces, etc. that make up the solid are described. Display objects freely defined in individual modeling coordinate systems can all be placed on a unified world coordinate system to create a graphics space or a virtual three-dimensional world.

Here, the movement of a virtual user within the virtual space will be explained using FIGS. 13 and 14. 13 and 14 are conceptual diagrams showing the placement positions of virtual participants on world coordinates. FIG. 13 shows the positions of each virtual user at the time when the virtual user D1 enters a specific room (floor). FIG. 14 shows a state in which the virtual user D1 enters a specific room (floor) and then moves in the direction of the arrow.

In FIG. 13, virtual users A1, B1, C1, and D1 are located at participant positions a11, a12, a13, and a14, respectively. Similarly, the virtual display positions d11 to d15 are arranged in accordance with the organizer position a11, participant position a12, participant position a13, participant position a14, and bystander position a15, respectively. In this state, nothing is managed in the movement vector column of the position management DB 6007 (or it is managed as "0").
In this state, as shown in FIG. 14, when the virtual user D1 moves to the front of the virtual user A1 at a short distance, the movement vector column of the position management DB 6007 shows the direction of movement and the movement. A movement vector indicating the size (distance) of is managed.

Note that, as described above, the location management DBs 3007a to 3007d have the same data structure as the location management DB 6007, so a description thereof will be omitted.

(Functional configuration of communication management server)
Next, each functional configuration of the communication management server 6 will be described in more detail using FIGS. 2 and 4.

The transmission/reception unit 61 of the communication management server 6 is realized by the processing of the CPU 601 on the network I/F 609, and exchanges various data (or information) with other devices (for example, the communication device 1) via the communication network 100. Send and receive.

The determining unit 65 is mainly realized by the processing of the CPU 601, and performs various determinations.

The creation unit 66 is mainly realized by the processing of the CPU 601, and creates an image data ID.

The storage/read processing unit 69 is mainly realized by the processing of the CPU 601 on the HD 604 and HD 605 shown in FIG. (or information).

[Processing or operation of embodiment]
Next, the processing or operation of this embodiment will be described using FIGS. 15 to 34.

<Participation in video call>
First, a process for participating in a specific communication session (virtual space) will be described using FIGS. 15 to 23. FIG. 15 is a sequence diagram showing a process for participating in a specific communication session. FIG. 16 is a display example of a login screen in a communication device.

First, the user of base A (for example, user A1) enters his or her own connection ID and password on the login screen displayed on the display 118 by the display control unit 34a, and presses the "OK" button. The unit 32a accepts the connection ID and password, and the transmitting/receiving unit 31a transmits a login request to the communication management server 6 (S21). This login request includes a connection ID, a password, and the IP address of the sender (communication device 1a). Thereby, the transmitting/receiving unit 61 of the communication management server 6 receives the login request.

Next, the storage/read processing unit 69 of the communication management server 6 searches the communication management DB 6001 (see FIG. 6) using the connection ID and password included in the login request as a search key, and searches for the same connection ID and the same password. Authentication is performed by determining whether the set of passwords is managed (S22). Here, the explanation will be continued assuming that the same connection ID and the same password are managed by the storage/read processing unit 69. In this case, the storage/read processing unit 69 adds the sender's IP address received in step S21 to the record of the connection ID and password pair used for authentication.

Next, the storage/read processing unit 69 reads out all selected placement information from the selected placement information management DB 6005, and reads specific fixed placement information including the connection ID of the communication device 1a from the fixed placement information management DB 6006 (S23). . Then, the transmitting/receiving unit 61 transmits a response to the communication device 1a indicating that the authentication was successful (S24). This response includes all of the selected placement information read above and the fixed placement information regarding the communication device 1a. Thereby, the transmitting/receiving unit 11a of the communication device 1a receives the response.

Next, the communication device 1a executes a process of specifying the position of the virtual user A1 in the virtual space based on the received selective placement information and fixed placement information (only the communication device 1a). Here, the process of step S25 will be explained in detail using FIGS. 17 to 21. FIG. 17 is a flowchart showing the position specifying process. FIG. 18 is a diagram showing a selection screen for the type of placement position in the communication device. FIG. 19 is a diagram showing a display example of a selection screen for selecting a virtual company. FIG. 20 is a diagram showing a display example of a selection screen for selecting a virtual department. FIG. 21 is a diagram showing a display example of a selection screen for selecting a virtual room (floor).

First, as shown in FIG. 18, the display control unit 34a displays a selection screen for selecting the type of arrangement on the display 118 of the communication device 1a based on the selected arrangement information (S180). This selection screen displays a "select placement" button b1 and a "fixed placement" button b2. The "select placement" button b1 is a button for user A1 etc. to press when selecting and entering a desired room in multiple virtual spaces.The "fixed placement" button b2 is a button for user A1 etc. to press in advance This is a button to be pressed when entering the vicinity of an individual's seat in a predetermined virtual space.

Next, the user A1 presses either button b1 or b2, and the reception unit 32a accepts the selection (S181).

Next, when the determining unit 35a determines that the “selected layout” button b1 has been pressed (S182; YES), the display control unit 34a, as shown in FIG. 19, based on the selected layout information. A company selection screen for selecting a company to visit is displayed on the display 118 (S183). This company selection screen displays "company" buttons b11, b12, and b13 indicating each of a plurality of companies. Note that any number of "company" buttons may be displayed.

Here, when the user A1 presses a desired button (here, the "AA company" button b11), the reception unit 32a accepts the selection of the company (S184). Thereby, the display control unit 34a displays a department selection screen for selecting a department to visit on the display 118, as shown in FIG. 20, based on the selection arrangement information (S185). On this department selection screen, "Department" buttons b111, b112, and b113 indicating each of a plurality of departments are displayed. Note that any number of "department" buttons may be displayed.

Here, when the user A1 presses a desired button (here, the "AA1 Headquarters" button b111), the reception unit 32a accepts the selection of the department (S186). Thereby, the display control unit 34a displays a room selection screen for selecting a room to visit on the display 118, as shown in FIG. 21, based on the selected arrangement information (S187). This department selection screen displays "room" buttons b1111, b1112, and b1113 indicating each of a plurality of rooms (floors). Note that any number of "room" buttons may be displayed.

Here, when the user A1 presses a desired button (here, the "AA1 Headquarters" button b111), the reception unit 32a accepts the selection of the department (S186). Thereby, the display control unit 34a displays a room selection screen for selecting a room to visit on the display 118, as shown in FIG. 21 (S187). This department selection screen displays "room" buttons b1111, b1112, and b1113 indicating each of a plurality of rooms (floors). Note that any number of "room" buttons may be displayed.

Here, when the user A1 presses a desired button (here, the "AA11 room" button b1111), the reception unit 32a accepts the selection of the room (S188). As a result, position information indicating the selected room is transmitted in step S26, which will be described later.

On the other hand, if the determining unit 35a determines in step S182 that the "select placement" button b1 has not been pressed (that is, the "fixed placement" button b2 has been pressed) (S182; NO), the determination unit 35a performs the steps described below. In S26, fixed position information indicating the position around the user's seat in a specific virtual room is transmitted.

Next, returning to FIG. 15, the transmitting/receiving unit 31a transmits a request to participate in the virtual space to the communication management server 6 (step S26). This participation request includes position information related to the button whose selection was accepted in step S25. This location information includes a session ID indicating a communication session for entering the selected room, a connection ID of the transmission source (here, the communication device 1a), and the selected placement information or fixed placement information received in step S24. Among the conversion parameters included in , a conversion parameter for indicating the specific position of the room selected in step S25 is included. Thereby, the transmitting/receiving unit 51 of the communication management server 6 receives the participation request.

Next, the storage/read processing unit 69 performs virtual space participation processing on the session management DB 6002 (see FIG. 7) (S27). Specifically, the storage/read processing unit 69 associates the connection ID of the communication device 1a received in step S26 with the record having the same session ID as the session ID received in step S26 in the session management DB 6002. management.

Next, the storage/read processing unit 69 reads out other connection IDs associated with the same session ID as the session ID associated with the connection ID managed in step S27 (S28). Then, the storage/read processing unit 69 sends the location information including the connection ID of the communication device 1a received in step S26 and the location information including other connection IDs read out in step S28 to the location management DB 6007. Each new record is registered (S29).

Next, the transmitting/receiving unit 61 transmits a participation request response to the participation request in step S26 above to the communication device 1a (S30). This participation request response includes location information including all connection IDs associated with the same session ID. Thereby, the transmitting/receiving unit 31a of the communication device 1a receives the participation request response. Then, in the communication device 1a, the storage/read processing unit 39a updates the contents of the location management DB 3007a by managing all the location information received in step S30 in the location management DB 3007a (S31).

Similarly, the transmitting/receiving unit 61 transmits participation information to the communication device 1b that is participating in the same communication session as the communication device 1a (S32). This participation information includes location information including all connection IDs associated with the same session ID. Thereby, the transmitter/receiver 31b of the communication device 1b receives the participation information. Then, in the communication device 1b, the storage/read processing unit 39b updates the contents of the location management DB 3007b by managing all the location information received in step S32 in the location management DB 3007b (S33). Note that the same process as step S32 is performed for communication devices other than the communication device 1b participating in the same communication session as the communication device 1a, and the same process as step S33 is performed for communication devices other than the communication device 1b. will be held.

In addition, in the above, after the selection buttons b1, b2, etc. shown in FIG. 18 are selected, the buttons shown in FIGS. 19 to 21 are selected a total of three times, but the selection is not limited to this. . For example, in FIG. 18, the number of selections may be reduced by buttons that include at least two of the company, department, and room.

<Display information transmission process>
Next, display information transmission processing will be described using FIG. 22. FIG. 22 is a sequence diagram showing display information transmission processing.

First, the transmitting/receiving unit 31a of the communication device 1a transmits an information addition request to the communication management server 6 (S41). This information addition request includes the connection ID of the sending device (communication device 1a) and display information. Thereby, the transmitting/receiving unit 61 of the communication management server 6 receives the information addition request.

Next, the storage/read processing unit 69 of the communication management server 6 reads the corresponding session ID by searching the session management DB 6002 (see FIG. 7) using the connection ID received in step S41 as a search key. S42).

Further, the storage/read processing unit 69 stores display information, which is one record of data, in the display management DB 6004 (see FIG. 8) (S43). Thereby, the display information of each communication device 1 can be centrally managed.

Furthermore, if there is a communication device participating in the same communication session, the communication management server 6 transmits the image data ID, connection ID, and display information of the communication device 1a to this communication device, thereby sharing information. There is a need to. Therefore, the transmitting/receiving unit 61 of the communication management server 6 transmits an information addition notification to another communication device (for example, the communication device 1b) participating in the same communication session as the communication device 1a (S48). This information addition notification includes the image data ID, and the connection ID and display information of the communication device 1a that is the transmission source received in step S41. Thereby, the transmitting/receiving unit 11b of the communication device 1b receives the information addition notification.

As described above, all communication devices that have participated in the virtual space of the same communication session can share additional information of other communication devices.

<Transmission of image data and sound data>
Next, the image data and sound data transmission process will be explained using FIGS. 23 and 24. FIG. 23 is a sequence diagram illustrating processing until brightness image data, depth image data, and sound data transmitted from a predetermined communication device (for example, communication device 1a) reach another communication device. FIG. 24 is a conceptual diagram of brightness image data and depth image data to be transmitted.

First, using FIG. 23, image data (in this case, brightness image data and depth image data) and sound data are transmitted from the communication device 1a to a plurality of other communication devices 1b, 1c, and 1d via the communication management server 6. The process for sending will be explained.

First, the communication device 1a starts photographing and collecting sound at base A, so that the brightness image acquisition unit 36a acquires brightness image data, the depth image acquisition unit 37a acquires depth image data, and the sound collection unit 38a acquires sound data (S101). In this case, the communication device 1a shoots a moving image, but it may also shoot a still image.

Next, in the communication device 1a, the storage/read processing unit 39a updates the movement vector of the own device (communication device 1a) in the position management DB 3007a (S102). Then, the transmitting/receiving unit 31a of the communication device 1a transmits the connection ID of the communication device 1a as the transmission source, in addition to image data (luminance image data, depth image data) and sound data, to the communication management server 6 ( S103). In this case, the image data includes an image data ID for identifying the image transmitted from the device itself, and the movement vector Taa updated in step 102. As a result, the transmission/reception unit 61 of the communication management server 6 receives the image data (luminance image data, depth image data, image data ID, movement vector Taa), sound data, and the connection ID of the communication device 1a that is the transmission source. do.

Here, the transmitted luminance image data and depth image data will be explained using FIGS. 3 and 24. In FIG. 3, since the photographing unit 105 for taking self-portraits is provided, the image data includes data of the luminance image Lc1 and data of the depth image Dc1. The data of the brightness image Lc1 and the depth image Dc1 are data acquired by the photographing unit 105.

For example, the data resolution of the luminance image Lc1 and the depth image Dc1 is each 4K, but as shown in FIG. 24, the transmission source communication device 1 compresses a total of 2 data into one image data. By combining these images, the resolution of this single image data becomes 4K. Note that 4K is an example, and HD, FHD, 8K, etc. may also be used. Next, returning to FIG. 23, in the communication management server 6, the storage/read processing unit 69a updates the movement vector of the communication device 1a in the position management DB 6007 (S104). Then, by using the session management DB 6002 (see FIG. 7), the transmitting/receiving unit 61 of the communication management server 6 communicates with each of the other communication devices 1b to 1d participating in the video call in the same communication session as the communication device 1a. In response, image data (luminance image data, depth image data, image data ID, movement vector Taa), sound data, and the connection ID of the communication device 1a that is the transmission source are transmitted (S105, S107, S109). As a result, the transmitting/receiving units 31b, 31c, and 31d of each of the communication devices 1b, 1c, and 1d transmit image data (luminance image data, depth image data, image data ID, movement vector Taa), sound data, and transmission sources. Receive the connection ID of the communication device 1a.

Then, in each communication device 1b, 1c, 1d, the storage/read processing unit 39b, c, d updates the movement vector of the communication device 1a in the position management DB 3007b, c, d, respectively (S106, S108, S110 ).

<Receiving image data and sound data>
Next, the process of receiving image data and sound data will be described using FIGS. 25 to 33. FIG. 25 is a sequence diagram showing the process of receiving image data and sound data by the communication device 1d.

As shown in FIG. 25, in the communication device 1a, the storage/read processing unit 39a updates the movement vector of the own device (communication device 1a) in the position management DB 3007a (S121). The process in step S121 is the same as step S102 in FIG. 23.

Next, the transmitting/receiving unit 31a sends image data (luminance image data, depth image data, image data ID, movement vector Taa), sound data, and the connection ID of the communication device 1a as the transmission source to the communication management server 6. (S122). Thereby, the transmitting/receiving unit 61 of the communication management server 6 receives the image data, the sound data, and the connection ID of the communication device 1a. This step S122 processing is the same as step S103 in FIG. Then, in the communication management server 6, the storage/read processing unit 69a updates the movement vector of the communication device 1a in the position management DB 6007 (S123). The process in step S123 is the same as step S104 in FIG. 23.

Next, by using the session management DB 6002 (see FIG. 7), the transmitting/receiving unit 61 of the communication management server 6 communicates with the other communication device 1d participating in the video call in the same communication session as the communication device 1a. , image data (luminance image data, depth image data, image data ID, movement vector Taa), sound data, and the connection ID of the communication device 1a that is the transmission source (S124). Thereby, the transmitting/receiving unit 31d of the communication device 1d receives the image data, the sound data, and the connection ID of the communication device 1a. This process is the same as step S109 in FIG.

Similarly, the transmitting/receiving unit 31d of the communication device 1d receives image data (luminance image data, depth image data, image data ID, movement vector Tab), sound data, and a transmission source of the communication device 1b from the communication management server 6. The connection ID of the communication device 1b is received (S125). Further, the transmitting/receiving unit 31d of the communication device 1d receives from the communication management server 6 image data (luminance image data, depth image data, image data ID, movement vector Tac) of the communication device 1c, sound data, and communication as a transmission source. The connection ID of the device 1c is received (S126).

Next, in the communication device 1d, the image/sound processing unit 33d performs image display and sound output based on each data received in steps S124 to S125 and the information addition notification received similarly to step S44 in FIG. The following process is performed (S127). In this case, since the sound output is a general process of converting each sound data (electrical signal) into vibration, the image display process will be described in detail using FIG. 26.

<Image display processing>
FIG. 26 is a flowchart showing image display processing.

First, the storage/read processing unit 39d searches the location management DB 3007d using the connection ID received in steps S124 to S126 in FIG. ) is read out (S201). Then, the image/sound processing unit 33d stores the position of the user D1 in the virtual space of the own device (communication device 1d) read out in step S201, the position of the other communication device 1a read out in step S201, The distances between 1b and 1c and the respective positions of users A1, B1, and C1 on the virtual space are measured (S202).

Here, the relationship between the virtual user D and the distance will be explained using FIG. 27. FIG. 27 is a diagram showing an example of the distance from the virtual user D1 and the displayed image. Note that the displayed images will be explained later. As shown in FIG. 27, the distance from the virtual user D to a nearby position is the first distance, and the distance from the far position is the second distance.

Then, returning to FIG. 26, if the distance is less than the first distance (S203; less than the first distance), the process proceeds to step S204. Furthermore, if the distance is equal to or greater than the second distance (S203; equal to or greater than the second distance), the process proceeds to step S211. Further, if the distance is greater than or equal to the first distance and less than the second distance (S213; greater than or equal to the first distance and less than the second distance), the process proceeds to step S213.

(Display processing when the distance is less than the first distance)
Here, display processing when the distance is less than the first distance will be explained.

((Calculation process of object position in world coordinate system))
First, the communication device 1d calculates the position of the object in the world coordinate system (S204). Here, display processing when the image type is a brightness/depth image will be described in detail using FIGS. 28 to 31. FIG. 28 is a flowchart showing the process of calculating the position of an object in the world coordinate system.

First, the image/sound processing unit 33d converts an image of an object into three dimensions based on the brightness image data and depth image data of each base including the own base in the virtual space (modeling coordinate system) (S221).

Here, using FIG. 29, three-dimensionalization of an image of an object will be explained. FIG. 29 is a conceptual diagram of the conversion from the camera coordinate system to the modeling coordinate system and the conversion from the modeling coordinate system to the world coordinate system. Note that the camera coordinate system is a coordinate system that determines how the world looks on a projection plane, with the physical camera as the origin in the three-dimensional world that is the real world. It is necessary to convert the coordinates given in real coordinates to the coordinate system seen in the camera's coordinate system.

In order to cause images of users at multiple bases in real space to appear in one virtual space such as a virtual conference room, the image/sound processing unit 33d first individually converts objects in the real space of each base to the first It is necessary to make it three-dimensional by defining it in a modeling coordinate system (Modeling Coordinate System) as a virtual space, and then define the image of each object in one world coordinate system (World Coordinate System).

Therefore, first, the image/sound processing unit 33d processes five sets of images, such as the luminance image Lc1 and the depth image Dc1, defined in the camera coordinate system as shown in FIG. Three-dimensionalization is performed by converting onto one first virtual space (modeling coordinate system) using (Equation 1). In FIG. 29, user A1 is represented as an object. In this case, the image of user A1 in the modeling coordinate system is represented by a set of many points, as shown in the enlarged view ev of FIG. In this embodiment, since images are taken from five angles by five brightness depth photographing devices 5a to 5e, the image of user A1 is represented as a collection of many overlapping points. In this way, users A1, B1, C1, and D1 are each independently defined in separate modeling coordinate systems (3-dimensionalized).

Note that the vector P is a projection matrix, and the vector P-1 is a matrix for converting from the camera coordinate system to the real coordinate system. Vector M is a matrix for converting from the real coordinate system to the modeling coordinate system. This output value indicates the point (coordinates) of the modeling coordinate system and color information.

Next, returning to FIG. 28, the storage/read processing unit 39d reads out the display information of the own device (communication device 1d) stored in the storage unit 3000d (S222). The image/sound processing unit 33d then displays the virtual display (also referred to as a virtual screen) of its own base in the modeling coordinate system based on "information on the position of each corner of the display in the virtual space (modeling coordinate system)" in the display information. ) dd is placed (S223). FIG. 30 is a conceptual diagram in which the image of the user D1 and the virtual display are arranged in the modeling coordinate system. For example, as shown in FIG. 30, the image/sound processing unit 33d further arranges (defines) a virtual display dd on the image of the user D1 at its own location.

Returning to FIG. 28, the storage/read processing unit 39d also retrieves, from the location management DB 3007d (see FIG. 12), " ``Conversion parameters from modeling coordinate system to world coordinate system'' and ``movement vector'' are read out (S224). Then, the image/sound processing unit 33d performs modeling conversion of the image of each object and the virtual display of its own base from the modeling coordinate system to the world coordinate system in the virtual space (S225).

Here, the conversion from the modeling coordinate system to the world coordinate system will be explained using FIG. 29 again.

As shown in FIG. 29, the image/sound processing unit 33d converts the image of the user A1 in the modeling coordinate system onto a common second virtual space (world) using (Equation 2). Modeling Transformation to (coordinate system). Modeling transformation is a transformation process to place a display object defined on a modeling coordinate system on a world coordinate system, and is a combination of four transformation processes: scaling, rotation, shearing, and parallel translation of a display object. shows.

Note that the vector W is a matrix for converting from the modeling coordinate system to the world coordinate system. This output value indicates a point (coordinates) in the world coordinate system and color information.

FIG. 31 is an image diagram showing the conversion from the modeling coordinate system to the world coordinate system. As shown in FIG. 31, the image/sound processing unit 33d models the images A1, B1, C1, D1 of each modeling coordinate system and the virtual display dd4 of its own base into one common world coordinate system. Convert. Thereby, images of users A1, B1, C1, and D1 at each base can participate in a video call of a common communication session.

((Displaying images visible to the user))
Next, returning to FIG. 26, the process of displaying an image that the user D1 can view on the communication device 1 will be described.

As shown in FIG. 26, the image/sound processing section 33a uses the luminance image data acquired by the luminance image acquisition section 36d and the luminance image data acquired by the depth image acquisition section 37d to Skeletal information such as eyes, nose, shoulders, etc. is detected (S204).

Next, the image/sound processing unit 33a sets the center of both eyes of the skeleton information as a viewpoint position, and based on this viewpoint position, the user D1 in the world coordinate system as shown in the center of FIG. An image of another user A1 etc. that can be viewed using dd is created (S205). In other words, the image/sound processing unit 33a converts the display object in the world coordinate system into an image displayed on the virtual display dd based on the viewpoint position. This transformation means transformation from the world coordinate system to the viewpoint coordinate system, and is called visual field transformation. The viewpoint coordinate system is a coordinate system that has its origin at the position of the camera (viewpoint) in virtual space and extends in the direction the user wants to view (the direction in which the subject exists).

In the world coordinate system, calculate the intersection of a ray directed from the viewpoint to a pixel on the virtual display dd using the ray tracing method and a polygon (virtual triangle, polygon, etc. that makes up the 3D model) . This ray is a ray that travels from the image direction toward the viewpoint and traces it back from the viewpoint. This method is called the ray tracing method, and is also called the reverse line-of-sight search method because it traces the line of sight in the opposite direction. Then, it finds the intersection closest to the viewpoint, that is, the object that first intersects with the ray (in the field of computer graphics, it is called the above-mentioned "polygon"), and paints the pixels on the virtual display with the color of that object. If there is no object (polygon) that intersects with the ray, paint the pixel with the background color. By performing this process on the pixels of the entire virtual display dd, an image displayed on the virtual display dd is created.

In addition to the ray tracing method, there are other methods such as the Z buffer method, and the method for creating an image displayed on the virtual display dd is not limited to the above method.

Next, the image/sound processing unit 33a converts the image in the virtual space (world coordinate system) created above into an image to be displayed on the display 118 of the communication device 1d in the real space (S206). Then, the display control unit 34d displays the image converted in step S206 on the display 118 of the communication device 1d in the real space (S207). In this case, the space in which the image is displayed on the display 118 is a two-dimensional plane image, and the resolution depends on the performance of the computer, especially devices such as display memory installed on the video board (or graphics board). , called the device coordinate system.

(Display processing when the distance is greater than or equal to the second distance)
Next, display processing when the distance is greater than or equal to the second distance will be described.

The storage/reading processing unit 39d reads pre-stored icon data from the storage unit 3000d. Note that the icon is an example of a character image. Character images also include avatars.

Next, the image/sound processing unit 33d changes the size of the icon according to the distance between the virtual user D1 and another virtual user who is the object of measurement. For example, as shown in FIG. 27, at a second distance or more from the virtual user D, if it is close, the icon is changed to large C1, if it is further away, the icon is changed to medium C2, and if it is further away Change the icon to small C3. Although three stages of change are shown here, changes may be made in any number of stages. After the process in step S212, the process proceeds to step S205.

(Display processing when the first distance is greater than or equal to the second distance)
Next, display processing when the distance is greater than or equal to the first distance and less than the second distance will be described. In this case, the image/sound processing unit 33a selects a luminance image in front of the virtual user D1 and the other virtual user targeted for measurement, out of a total of 10 images as shown in FIG. A planar image is created using only Lc (S213). For example, as shown in FIG. 27, a plane image B is created at a distance greater than or equal to the first distance and less than the second distance from the virtual user D. After the process in step S213, the process proceeds to step S205.

<Image display example>
Next, various display examples in step S207 in FIG. 26 will be explained using FIGS. 32 and 33.

(Example of display shown to conference participants (user D1))
((Initial display))
FIG. 32 is a diagram showing an image displayed on the display 118 of the communication device of the user D1. Further, FIG. 32 is a display example based on the arrangement positions of the virtual participants shown in FIG. 13. As shown in FIG. 32, in the virtual space (here, the world coordinate system), beyond the line of sight of the user D1, there is a plane image of the user A1 diagonally in front of the right and a little further back; Furthermore, a large icon showing the front of the user B1 is located at the back, and a small icon showing the back of the user C1 is located at the farthest back directly in front. The name of each user is also displayed. This name is displayed using name information managed in the communication management DB 6001. This name information is sent in advance to each communication device 1 from the communication management server 6 along with the connection ID. Note that, in principle, only one user exists at one position in the virtual space (world coordinate system).

In this way, the display control unit 34d causes the display 118 of the communication device 1d to display a composite image obtained by combining various images of the users A1, B1, and C1.

((Displayed after moving))
Next, as the user D1 operates the mouse 13 or the like in the real space, the virtual user D1 moves close to the virtual user A1 in the virtual space, as shown in FIG. The display in this case will be explained using FIG. 33. FIG. 33 is a diagram showing an image displayed on the display 118 of the communication device of the user D1.

As shown in FIG. 33, in the virtual space (here, the world coordinate system), beyond the line of sight of the user D1, there is a 3D image of the user in the immediate vicinity, and a 3D image of the user is located diagonally to the left and a little further back. In the plane image of user B1, an icon indicating the back of user C1 is located slightly diagonally to the left and further back. In this way, the display control unit 34d causes the display 118 of the communication device 1d to display a composite image obtained by combining various images of the users A1, B1, and C1.

[Main effects of the embodiment]
As described above, according to the present embodiment, the display control unit 34d displays an image of the subject at the other base based on the image data received by the transmitting/receiving unit 31d, depending on the distance between the own base and the other base in the virtual space. and a character image related to this subject are displayed while being switched. This makes it possible to eliminate as much as possible the problem that the virtual user C1 and the like in the virtual space do not feel safe because they do not know who is watching them.

The display control unit 34d also displays a planar image of the subject at another base based on the brightness image data received by the transmitting/receiving unit 31, and a flat image of the subject at the other base based on the data of the brightness image received by the transmitting/receiving unit 31, depending on the distance between the own base and the other base in the virtual space. A stereoscopic image of a subject at another base based on brightness image and depth image data is switched and displayed. This allows the user D1 to know that a user whose name he has forgotten but whose face he remembers is participating in the virtual space, and to understand the facial expressions of the users. Furthermore, when the distance between user D1 and another user (for example, user A1 in FIG. 34) becomes quite close in the virtual space, the two-dimensional image changes to a three-dimensional image, giving the user a feeling of being in the same room as the user with whom they are communicating. I can do that.

Further, the display control unit 34d switches and displays character images of a plurality of sizes depending on the distance between the own base and other bases in the virtual space. Further, the display control unit 34d increases the size of the character image and displays it as the distance between the own base and other bases in the virtual space becomes shorter. As a result, the user D1 knows that he has approached the virtual user even if the user is displayed as an icon. Note that although the user D1 cannot grasp the facial expressions of the virtual users indicated by the icons, he can grasp who exists in the virtual space from the displayed names.

〔supplement〕
The communication management server 6 may be simply called a "server". The communication management server 6 may be provided in the form of cloud computing. Cloud computing refers to a usage form in which computer resources are provided in the form of services via a computer network such as the Internet, and the provided computer resources are servers. The forms of services provided include SaaS (Software as a Service), PaaS (Platform as a Service), HaaS, and IaaS (Hardware/Infrastructure as a Service), but the form of service provision is not limited.

Furthermore, in the above embodiment, as shown in FIG. 29, each communication device 1 once individually defines objects in the real space of each base in the modeling coordinate system as the first virtual space. Although the images of each object are converted into three-dimensional images using a single world coordinate system, the present invention is not limited to this. For example, the communication management server 6 converts the objects in the real space of each base into three-dimensional objects by individually defining them in the modeling coordinate system as the first virtual space, and then converts the images of each object into one world. It may also be converted into a coordinate system. In this case, the communication management server 6 transmits the image data converted into the world coordinate system to the communication device 1 at each base. That is, a communication system including a first communication device (for example, communication device 1d), a second communication device (for example, communication device 1a), and a server (for example, communication management server 6) is configured to Changing the position of the subject in the first communication device in virtual space and the position of the subject in the second communication device in virtual space using image data captured by the device and image data captured by the second communication device. The apparatus includes a providing means for providing a display image generated according to possible positional relationships, and a display control means for displaying the display image provided by the providing means.

A plurality of CPUs described in the above embodiments may be provided in one device.

Furthermore, each function of the above embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" as used herein refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or a processor designed to execute each function explained above. It includes devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array), SOC (System on a chip), GPU (Graphics Processing Unit), and conventional circuit modules.

Furthermore, in communication between each of the communication devices 1a to 1d and the communication management server 6, other servers, routers, etc. may relay data.

1a Communication device (an example of another communication device, an example of the first other communication device)
1b Communication device (an example of another communication device, an example of a second other communication device)
1c Communication device (an example of other communication device)
1d Communication device (an example of own device)
118 Display (an example of display means)
6 Communication management server 31d Transmitting/receiving unit (an example of a transmitting means, an example of a receiving means, an example of a second receiving means)
32d Reception department (an example of reception means)
34d Display control unit (an example of display control means)
61 Transmitting/receiving unit (an example of first receiving means)

Claims (8)

A communication device that makes a call with another communication device at another base,
Receiving means for receiving image data by the other communication device transmitting image data of the other base obtained by the other communication device photographing a subject at the other base;
a display control means for switching and displaying an image of a subject at the other base based on the received image data and a character image related to the subject according to a distance between the own base and the other base in a virtual space;
A communication device comprising: The communication device according to claim 1,
The image data is data of a brightness image and a depth image of the other base obtained by the other communication device photographing a subject at the other base in real space,
The display control means displays a planar image of the object at the other base based on data of the received brightness image, and the received brightness image and depth, according to the distance between the own base and the other base in the virtual space. A communication device that switches between displaying a stereoscopic image of a subject at the other location based on image data. The communication device according to claim 1 or 2, wherein the display control means switches and displays the character images of a plurality of sizes depending on the distance between the own base and another base in the virtual space. . 4. The communication device according to claim 3, wherein the size of the character image is increased and displayed as the distance between the own base and the other base in the virtual space becomes shorter. 5. The communication device according to claim 1, wherein the character image is an avatar or an icon. A communication system comprising a communication device that communicates with another communication device at another base, and a communication management server capable of communicating with the communication device, the communication system comprising:
The communication management server includes:
a first receiving unit that receives image data of the other base obtained by the other communication device photographing a subject at the other base, and the image data transmitted by the other communication device;
transmitting means for transmitting the image data to the communication device;
Equipped with
The communication device includes:
second receiving means for receiving the image data from the server;
a display control means for switching and displaying an image of a subject at the other base based on the received image data and a character image related to the subject according to a distance between the own base and the other base in a virtual space;
A communication system comprising: A display method executed by a communication device that makes a call with another communication device at another base,
a receiving step in which the other communication device transmits image data of the other base obtained by photographing a subject at the other base, and thereby receives the image data;
a display control step of switching and displaying an image of a subject at the other base based on the received image data and a character image related to the subject, depending on a distance between the own base and the other base in a virtual space;
A display method characterized by performing. A program that causes a computer to execute the method according to claim 7.