JP6730577B2 - Information processing apparatus, information processing system, control method thereof, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing system, a control method thereof, and a program.

In recent years, mixed reality (hereinafter referred to as MR) technology has become widespread. Using MR technology, a user wearing a head-mounted display (HMD) will have a simulated experience of an MR space (a mixed reality space in which a physical space and a virtual space are superimposed) in which a physical object and a CG model are arranged. Can be provided.

Patent Document 1 describes a technique of drawing an image in which the marker is hidden when the marker is detected in order to prevent the user from being aware of the presence of the marker.

Conventionally, depending on the HMD that is worn in order to experience the MR space, the field of view of the user is narrowed due to the wearing, and there is a problem that the user cannot fully grasp the surrounding situation. For example, when another user is approaching from outside the field of view, it is possible that the user suddenly bumps into it.

JP-A-2000-350860 JP, 10-154243, A

Patent Document 2 discloses a technique of displaying the position of an avatar in a virtual space on a radar map on a PC screen. That is, the user can confirm the position of another person in the same virtual space as his or her avatar on the radar map from a third-party viewpoint.

On the other hand, in the case of the MR mechanism, a plurality of people may share the same virtual space, but they may be in different places in the real space. There is no danger of hitting another user in another location in the real world. On the other hand, if you approach another user in the same place, you risk hitting it.

In addition, since the HMD has a smaller image display unit than a general PC display, it is considered that he does not want to display useless information as much as possible.

An object of the present invention is to provide a mechanism for properly displaying the position of another user in a wearable display device.

The present invention is

According to the present invention, it is possible to provide a mechanism for appropriately displaying the positions of other users in a wearable display device.

It is a figure showing an example of a system configuration in an embodiment of the present invention. It is a figure which shows an example of the hardware constitutions of various apparatuses in embodiment of this invention. It is a figure showing an example of functional composition of various devices in an embodiment of the present invention. 3 is a flowchart showing a flow of transmission/reception of position/orientation information of an HMD in the first embodiment of the present invention. 6 is a flowchart showing a flow of user position identification display processing according to the first embodiment of the present invention. It is a figure which shows an example of a structure of various data in embodiment of this invention. FIG. 6 is an image diagram of sharing a virtual space in the embodiment of the present invention. FIG. 5 is an explanatory diagram of directions based on a user in the embodiment of the present invention. FIG. 5 is an explanatory diagram of directions based on a user in the embodiment of the present invention. It is a figure which shows an example of the screen structure of the identification display of a user position in embodiment of this invention. It is a figure which shows an example of the screen structure of the identification display of a user position in embodiment of this invention. It is a figure which shows an example of the screen structure of the identification display of a user position in embodiment of this invention. It is a flow chart which shows an outline of distance judgment processing in a 2nd embodiment of the present invention. It is a flow chart which shows an outline of identification display of an avatar in a 3rd embodiment of the present invention. FIG. 6 is an explanatory diagram of an icon display area and an arrangement position of icons in the embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing an example of a system configuration according to an embodiment of the present invention.

The system of the present invention includes a server 200, a client PC 100, and a head mounted display 101 (hereinafter, HMD 101). In FIG. 1, various devices and the HMD 101 are assumed to be connected by networks 150, 151, and 152, but wired or wireless communication is possible as long as the server 200, the client PC 100, and the HMD 101 can communicate with each other.

As shown in FIG. 1, various devices according to the present invention are communicatively connected via a network. For example, the server 200 and the client PC 100 (collective name of the client PCs 100A to 101F) are connected. Further, the client PC 100 is communicably connected to an HMD 101 (a generic name of HMD 101A to HMD 101F) managed by each client PC 100.

The server 200 stores a three-dimensional model to be superimposed on a real image captured by the HMD 101. The three-dimensional model is distributed to each client PC 100 connected to the server 200 and stored in the external memory of the client PC 100.

The client PC 100 acquires a real image from the HMD 101 managed by the client PC 100, generates an MR image in which a three-dimensional model is superimposed on the real image (a mixed reality image in which a real image and a virtual space image are superimposed), and then the HMD 101. The MR image is transmitted to the HMD 101 so as to be displayed on the display. The HMD 101 displays the received MR image on the display.

In the illustrated environment, there are two bases, a base 1 and a base 2, and users (experiencers) wearing the HMD 101 share a virtual space at each base, and virtual objects (3D model) arranged in the virtual space are arranged. /CG) is shared. The base here means a real space in another place in the real world. For example, the room of the Tokyo branch and the room of the Osaka branch are designated as base 1 and base 2, respectively.

In the present invention, for example, in an image displayed on the HMD 101, control is performed to identify and notify the presence of the HMD 101, the HMD 101 of another base, and the HMD 101 of the own base. The above is the description of FIG. 1.

It is assumed that a plurality of markers are attached inside the room of each base. The position and orientation of the HMD 101 can be specified by using the marker, and the position and orientation of arranging the virtual object can be determined.

The shape of the marker does not matter, but in the embodiment, it is assumed that the marker has a square shape and has the same size. A unique marker number is embedded in each marker. Then, it is assumed that each marker can be identified when the image is picked up by a camera provided in the HMD 101, and the marker number can be obtained when the marker is decoded. The type of marker (the type of role played by the marker) is a position detection marker for determining the position and orientation of the HMD 101, and a virtual object arrangement used to draw a virtual object at a position defined by the marker. There are two types of markers. The position detecting markers are attached in advance to known positions such that at least three markers are included in the visual field of the built-in camera regardless of the position and orientation of the HMD 101. And

The principle of detecting the position and orientation of the HMD 101 in the above configuration is to find the distance from the HMD 101 to each position detection marker from the size of each of the three position detection markers (the positions are known) in the captured image. .. Then, conversely, the position where the three distances obtained from the three position detection markers overlap is determined as the position of the HMD 101. Further, the attitude of the HMD 101 may be obtained from the arrangement of the three position detecting markers in the captured image.

When the viewpoint is in the normal direction of the marker, the marker image looks like a square. When the viewpoint deviates from the normal direction, the square appears distorted according to the degree of the deviation. That is, from this distortion, the orientation of the plane defined by the marker with respect to the axis of the viewpoint is known, the distance between the viewpoint and the marker can be detected from the size of the marker, and the plane to which the marker is attached can be defined. Further, in the embodiment, the marker is provided with two marks that can be distinguished from each other, and two vectors from the origin to the mark on the plane defined by the marker are defined as two points. It is assumed that one axis in the normal direction from the center position of the axis or the marker defines three axes that define local three-dimensional coordinates.

The virtual object placement marker is a marker used to specify the position where the virtual object is displayed. When the virtual object placement marker is detected, the server 200 and the CPU 201 of the client PC 100 position the position where the virtual object placement marker exists (to be exact, offset from the barycentric position of the virtual object placement marker as described later). Processing for displaying the virtual object at the position (with). For example, the process of arranging the three-dimensional model at the position in the virtual space indicated by the offset is performed.

A virtual object is a three-dimensional model (also called three-dimensional CAD data, 3D model, CG, etc.) arranged in a virtual space. It should be noted that the data of the three-dimensional model is associated with the position/orientation information indicating, in the external memory of the server 200 and the client PC 100, the position and orientation of the three-dimensional model at which position in the virtual space. Is remembered.

The coordinates of the central position of the virtual object placement marker are {Xv, Yv, Zv}, and the reference coordinates in the data of the three-dimensional model are {Xi, Yi, Zi} (where i=0, 1, 2,. ..), the distance L is obtained by the following equation.
L={(Xv-Xi)2+(Yv-Yi)2+(Zv-Zi)2}1/2

Here, the minimum distance L when i is changed is the distance between the virtual object placement marker and the three-dimensional model placed according to the position of the virtual object placement marker. In the above, when the distance is calculated, the square root is finally obtained, but if it is possible to determine the magnitude, it is not necessary to obtain the parallel root, and the square sum of the difference between the coordinates may be calculated. ..

In the above equation, the center position of the virtual object placement marker=the center position of the virtual object. As described above, the position where the virtual object is arranged is given an offset indicated by “position” with respect to the center position of the virtual object arrangement marker.

It should be noted that although depending on the number of bits that can be embedded in the marker, a marker that also serves as a position detection marker and a virtual object placement marker may be defined.

It should be appreciated that the above embodiment is an example according to the present invention. For example, in order to detect the position and orientation of the HMD 101, in the above embodiment, it is assumed that there are three position detecting markers in the visual field imaged by the position detecting markers, and many position detecting markers are pasted in the room. However, the present invention is not limited thereto. The position detection marker has a mark that defines the direction. If not only the coordinates of the position detection marker but also the shape and dimensions are known, from the distortion of the position detection of the captured image, the size, and the position of the mark, 1 The position and orientation of the HMD 101 can be specified only from one position detection marker. Further, instead of using the position detection marker, a publicly known technique of detecting the position and orientation of the HMD itself by using an optical sensor, a magnetic sensor, an ultrasonic sensor, or the like may be used. That is, any means may be used for detecting the position and orientation of the HMD.

Next, with reference to FIG. 2, hardware configurations of various devices in the embodiment of the present invention will be described.

In the present invention, the hardware configurations of the client PC 100 and the server 200 are assumed to be the same. It is assumed that the client PC 100 is directly connected to the HMD 101, and the server 200 is connected to the client PC 100.
The CPU 201 centrally controls each device and controller connected to the system bus 204.

In addition, the ROM 202 stores various programs necessary for realizing the functions executed by various devices such as a BIOS (Basic Input/Output System) which is a control program of the CPU 201, an operating system (OS), and the like.

The RAM 203 functions as a main memory, a work area, etc. of the CPU 201. The CPU 201 implements various operations by loading a program or the like required for executing the processing into the RAM 203 and executing the program.

Various programs used by the client PC 100 and the server 200 of the present invention to execute various processes described later are recorded in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as necessary. Is. Furthermore, definition files and various information tables used by the program according to the present invention are stored in the external memory 211.
The input controller (input C) 205 controls input from a pointing device (input device 210) such as a keyboard and a mouse.

The video controller (VC) 206 controls display on a display device such as the right eye/left eye display 222 provided in the HMD 101. For example, an external output terminal (for example, Digital Visual Interface) is used to output to the right/left eye display 222. The right/left-eye display 222 is composed of a display for the right eye and a display for the left eye.

The memory controller (MC) 207 is an adapter for a hard disk (HD) or a flexible disk (FD) or a PCMCIA card slot that stores a boot program, browser software, various applications, font data, user files, edit files, various data, and the like. It controls access to the external memory 211 such as a card type memory connected via the.

A communication I/F controller (communication I/FC) 208 connects and communicates with an external device via a network, and executes communication control processing on the network. For example, internet communication using TCP/IP is possible. The communication I/F controller 208 also controls communication with the optical sensor 104 via Gigabit Ethernet (registered trademark) or the like.

The general-purpose bus 209 is used to capture an image from the right-eye/left-eye video camera 221 of the HMD 101. Input from the right-eye/left-eye video camera 221 is performed using an external input terminal (for example, an IEEE 1394 terminal). The right/left-eye video camera 221 includes a right-eye video camera and a left-eye video camera.

The CPU 201 enables display on the display by executing an outline font rasterization process in a display information area in the RAM 203, for example. Further, the CPU 201 enables a user instruction with a mouse cursor or the like (not shown) on the display. The above is the description of FIG.

Next, with reference to FIG. 3, an example of a functional configuration of various devices in the embodiment of the present invention will be described.

The HMD position specifying unit 301 specifies the position and orientation of the HMD 101 connected to the client PC 100. Specifically, the position and orientation of the HMD 101 are specified by recognizing the marker on the real image captured by the HMD 101 and using the method described above.

The position information storage unit 302 is a storage unit that stores the information of the position and orientation specified by the HMD position specifying unit 301 in an external memory. The position information transmitting unit 303 is a processing unit that transmits the position information (X, Y, Z coordinates) of the HMD 101 specified by the HMD position specifying unit 301 to the server 200. The position information transmission unit transmits, together with the position information, the identification information of the HMD 101 located at the position indicated by the position information to the server 200.

The server 200 receives the position information transmitted by the position information transmitting unit 303 of the client PC 100 by the position information receiving unit 313 and stores the position information in the position information storage unit 311 in association with the received identification information of the HMD 101. The position information transmission unit 312 uses the position information of the HMD 101, the identification information of the HMD 101, and the base information to which the HMD 101 belongs stored in the position information storage unit 311 as a client PC 100 other than the client PC 100 corresponding to the HMD 101 (that is, another HMD 101). To the client PC 100) corresponding to.

The position information receiving unit 304 of the client PC 100 is a receiving unit that receives the position information/identification information/base information of the HMD 101 of the other client PC 100.

The icon determination unit 305 determines whether the HMD 101 at the position received by the position information reception unit 304 is an HMD at the same base as the HMD 101 of the own device, an HMD at another base, or another HMD at the same base. The determination unit determines the type of icon (object) indicating the position. The icon here means the X, Y, Z coordinates of the other HMD for notifying the user of the position of the other HMD when the other HMD (experienced person) is outside the field of view of the HMD 101. It is an icon displayed at the corresponding position on the display screen.

The icon display control unit 306 displays the type of icon determined by the icon determination unit 305 according to the position received by the position information reception unit 304. The display is controlled in a predetermined area (icon display area that is an icon display area) of the display screen.

The virtual object reception unit 307 is a reception unit that receives the virtual object stored in the virtual object storage unit 314 of the server 200 and transmitted from the virtual object transmission unit 315.

The virtual object storage unit 308 stores the virtual object received by the virtual object reception unit 307, and the virtual object display control unit 309 displays on the real image obtained from the HMD 101 according to the position and orientation information of the virtual object corresponding to the virtual object. Then, the MR image on which the virtual object is superimposed is controlled to be displayed on the HMD 101 (for example, the MR image is generated and transmitted to the HMD 101). The above is the description of FIG.

<First Embodiment>

Next, referring to FIG. 4, it is a flowchart showing a flow of transmission and reception of position and orientation information of the HMD in the embodiment of the present invention.

In step S401, the CPU 201 of the client PC 100 receives the image acquired from the right-eye/left-eye video camera 221 of the HMD 101 via the general-purpose bus 209 of the client PC 100 and holds it in the RAM 203.

In step S402, the two-dimensional marker image is extracted (detected) from the physical space image acquired in the RAM 203 of the client PC 100, the position/orientation of the HMD 101 is specified, and the HMD 101 is stored in the RAM 203 (corresponding to position/orientation acquisition means). Here, the identification information of the HMD 101 (HMD ID described later in FIG. 6) and the position/orientation information of the HMD 101 (X, Y, Z values of 603 and 604 described later in FIG. 6) are associated and stored in the RAM. It should be remembered.

In step S403, the CPU 201 of the client PC 100 transmits the position/orientation information identified in step S402, the identification information of the HMD 101 (HMD corresponding to the client PC 100) in the position/orientation, and the base information to the server 200 by communication. To do. The base information is identification information indicating the base information of the client PC 100, and is the identification information of the physical space in which the HMD 101 corresponding to the client PC 100 exists (described later in 602 of FIG. 6).

In step S404, the CPU 201 of the server 200 receives the position/orientation information, the identification information of the HMD 101, and the site information from the client PC 100 by communication. In step S405, the CPU 201 of the server 200 saves the HMD identification information, base information, and position/orientation information received in step S404 in an external memory. The format to be stored is shown in the HMD information 600 of FIG. 6, for example. The above is the description of FIG.

Here, an example of the configuration of various data according to the embodiment of the present invention will be described with reference to FIG. 6.

The HMD information 600 is stored in the external memory of the client PC 100 and the server 200. The HMD information 600 includes an HMD ID 601, which is identification information uniquely indicating the HMD 101, a base ID 602 (affiliation information) which is identification information uniquely indicating a base to which the HMD 101 belongs, a position 603 which is position information of the HMD, and attitude information of the HMD. It is information including a certain posture 604 (corresponding to belonging information storage means). It is assumed that the client PC 100 stores the HMD ID 601 and the position 603/posture 604 of the HMD 101 corresponding to its own device, and the base ID 602 of the base to which the HMD belongs in the external memory in association with each other. The server 200 provides the HMD ID 601 of each HMD 101 corresponding to each client PC 100 (example: client PCs 100A to 100F in FIG. 1) connected to the server 200, its position 603/posture 604, and the base ID 602 of the base to which each HMD belongs. It is assumed that it is received from each client PC 100 and is stored in the external memory of the server 200 in association with each HMD ID 601.

It is assumed that the client PC 100 stores the HMD information corresponding to the client PC 100 other than its own device in the external memory by receiving it from the server 200.

The model information 610 is stored in the external memory of the client PC 100 and the server 200. The model information 610 includes a model ID 611 that is identification information of a three-dimensional model (virtual object) that is placed in the MR space (placed in a virtual space that is superimposed on the real space), a model name 612 that is a file name of the actual data of the model, This is information in which a position 613 indicating the position and orientation in which the three-dimensional model is arranged, a posture 614, and a storage location 615 indicating the storage location of the three-dimensional model are stored in association with each other.

The icon information 620 is stored in the external memory of the client PC 100 and the server 200. The icon information 620 is information in which the icon ID 621 that is the identification information of the icon and the icon 622 that is the file name of the actual data of the icon are associated with each other. In the present embodiment, for example, when indicating the position of another HMD 101 belonging to the same base as the HMD 101 displaying the icon, the icon ID=Ic111 is used, and the HMD 101 displaying the icon belongs to a different base. When indicating the position of another HMD 101 to be operated, the user is notified of the positional relationship between the position of each HMD 101 and the HMD 101 displaying the icon by using the icon ID=Ic222. The above is the description of FIG. 6.

Next, with reference to FIG. 5, a flow of the user position identification display process according to the first embodiment of the present invention will be described.

In step S501, when the CPU 201 of the server 200 establishes the connection with the client PC 100, the model information 610 of FIG. 6 and the three-dimensional model data indicated by the model name 612 (file name) are transmitted to the client PC 100. In step S502, the CPU 201 of the client PC 100 receives the three-dimensional model data and the model information 610 from the server 200 and saves the model information 610 in a predetermined storage location in the external memory. Then, the location (address) where the three-dimensional model data is stored is inserted and stored in the storage location 615 of the model information 610.

In step S503, the CPU 201 of the client PC 100, when the 3D model data stored in the external memory is within the field of view for the real image acquired from the right-eye/left-eye video camera 221 of the HMD 101 corresponding to the own device, An MR image on which model data is superimposed is generated.

In step S504, the CPU 201 of the server 200 provides the client PC 100 with the site IDs of all the client PCs 100 (the IDs of the sites to which the HMD 101 corresponding to the client PC 100 belongs), the HMD ID, and the position/orientation of the HMD. Send to. In step S505, the CPU 201 of the client PC 100 receives the HMD ID of each HMD 101 corresponding to all the client PCs 100, the position/posture information of the position/posture information of the HMD, and the base information from the server 200, and stores them in the external memory ( Corresponds to position and orientation acquisition means).

Then, the processes of steps S506 to S512 or S506, S507, S514 to S517 are executed for all the saved HMDs 101 (other than the HMD corresponding to the own device).

In step S506, the CPU 201 of the client PC 100 acquires one piece of HMD information 600 of another HMD 101 stored in the external memory. Then, in step S507, it is determined whether the acquired position 603 of the other HMD 101 is within the field of view (=field of view, angle of view) of the HMD 101 (here, for example, HMD 101A) corresponding to the own device. If it is within the field of view, the process proceeds to step S514, and if not, the process proceeds to step S508. The information on the angle of view is as indicated by 630 in FIG. In the present embodiment, it is assumed that the angle of view of the left-eye video camera and the angle of view of the right-eye video camera of the HMD 101 are the same as the angle of view at which both cameras can capture images. The relationship between the angle of view of the left-eye video camera, the angle of view of the right-eye video camera, and the angle of view at which both cameras can be imaged is shown at 800 in FIG. The reference point at 800 in FIG. 8 is the position of the HMD 101.

In step S508, the base ID 602 of the other HMD 101 acquired in step S506 and the base ID 602 of the HMD 101A corresponding to the own device are acquired from the external memory. Then, in step S509, it is determined whether the acquired base ID of the other HMD 101 and the base ID of the own base are the same (are the same base). If it is a different base (if the base ID is different), the process proceeds to step S510, and if the same base, the process proceeds to step S511.

In step S510, the CPU 201 of the client PC 100 acquires the icon image for another site (Ic222.jpg in FIG. 6) from the external memory 211.

In step S511, the CPU 201 of the client PC 100 acquires the icon image for the same site (Ic111.jpg in FIG. 6) from the external memory 211.

In step S512, the CPU 201 of the client PC 100 determines the actual image acquired from the right-eye/left-eye video camera 221 of the HMD 101, based on the position/orientation information of the other HMD 101 acquired in step S506, according to the base ID of the HMD 101. An image in which the icon acquired in step S510 or step S511 is superimposed is generated. Then, when the processes of steps S506 to S512 or S506, S507, S514 to S517 have been completed for all other HMDs 101, the superimposed image generated in step S512 and/or step S518 is used for the right eye of the HMD 101. -Output to the HMD 101 for display on the left-eye display 222 (step S519). When the processes of steps S506 to S512 or S506, S507, S514 to S517 have not been completed for all the other HMDs 101, the process is returned to step S508, and the unacquired HMD 101 of the other HMDs 101 is acquired.

Here, with reference to FIGS. 7 to 10, the arrangement of the icons and the directions such as front and rear of the HMD will be described.

The positional relationship between the user A (wearer of the HMD 101A) and another user (other HMD 101) is shown in 810 of FIGS. 7 and 8. The field of view is forward (forward). For user A, user E is a user who can say in the field of view (forward (forward)). User B is a user on the left side (left direction) outside the field of view. User C is a user behind (backward) outside the field of view. User D is a user on the right side (outward direction) outside the field of view. It is assumed that the multi-function peripheral is a three-dimensional model (virtual object), which is placed in a virtual space and superposed on a real image using the technique described in 610 of FIG. 6 and the cited document.

In the description of the present embodiment, as shown by 700 and 710 in FIG. 7, the user B and the user E (the user B wearing the HMD 101B and the HMD 101E) are in the same base 1 (the actual room at the Tokyo branch) as the user A. There is a user E) who wears a user C, and a user C and a user D (a user C who wears the HMD 101C and a user D who wears the HMD 101D) exist in a base 2 (an actual room in the Osaka branch) different from the user A. It is assumed that Then, by sharing the virtual space and displaying an avatar (a three-dimensional model showing the position and orientation of each user) at each user's position, it is as if a user at another site were in the same space as in 720 of FIG. Show them to do.

The CPU 201 of the client PC 100 arranges icons corresponding to the positions of other HMDs 101 and the bases of other HMDs 101, as in 1010 of FIG. 10, with respect to the real image 1000 (FIG. 10) captured by the camera of the HMD 101, for example. Generate (superimposed) screen (image). The placement position of the icon is determined by using, for example, a technique described in a known technique (eg, Japanese Patent Laid-Open No. 8-229238). The placement and display of the icon is performed within a predetermined area 1011 on the real image (on the MR image).

That is, the position of the other HMD 101 viewed from the HMD 101A is specified, the position on the predetermined area 1011 corresponding to the position is determined, and the icon corresponding to the base of the other HMD 101 is added to the determined position on the image. Deploy.

Returning to the explanation of FIG. When it is determined in step S507 that the position 603 of the other HMD 101 is within the field of view (=field of view, angle of view) of the HMD 101 corresponding to the own device, the other HMD 101 stored in the external memory is determined in step S514. The base ID 602 of is acquired.

Then, in step S515, it is determined whether the base ID of another HMD 101 and the own base are the same base. If it is a different site, the process proceeds to step S516, and if it is the same site, the process proceeds to step S518.

In step S516, the CPU 201 of the client PC 100 acquires the avatar image (three-dimensional model) from the external memory 211. Then, in step S517, the avatar image of the position/orientation of the other HMD 101 is added to the real image acquired from the right-eye/left-eye video camera 221 of the HMD 101 based on the position/orientation information of the other HMD 101 acquired in step S506. A superposed MR image is generated. Then, when the processes of steps S506 to S512 or S506, S507, S514 to S517 for all other HMDs 101 are completed, the superimposed image generated in step S512 and/or step S518 is used for the right eye of the HMD 101. It is output to the HMD 101 to be displayed on the left-eye display 222 (step S519). When the processes of steps S506 to S512 or S506, S507, S514 to S517 have not been completed for all the other HMDs 101, the process is returned to step S508, and the unacquired HMD 101 of the other HMDs 101 is acquired.

The avatar image (three-dimensional model/virtual object) is stored in a predetermined storage location of the server 200, is distributed to the client PC 100 in step S501, and the client PC 100 stores it in its own external memory. Further, the position/orientation of the avatar image in the virtual space is changed by the client PC 100 according to the position/orientation of the HMD 101 in the real space. Specifically, the client PC 100 generates the avatar information 640 of FIG. 6 on the external memory. The avatar information 640 is associated with the ID 641 of the avatar, the avatar name 642 that is the file name of the avatar image, and the storage location 645 of the avatar image, and writes the information of the position 643 and the attitude 644 of the HMD 101 corresponding to the avatar at any time, This is the table to be updated. The CPU 201 of the client PC 100 continuously receives the position and orientation information of the other HMDs 101 from the server 200, and sets the received position and orientation information of each HMD 101 in the position 643 and the posture 644 of the avatar information 640 corresponding to each HMD 101. Overwrite. The avatar image is arranged in the virtual space based on the position 643 and the posture 644. The above is the description of FIG.

As described above, according to the present invention, it is possible to provide a mechanism for properly displaying the positions of other users in a wearable display device.

For example, a user at the same base and a user at another base can be displayed in a distinguishable manner.

<Second Embodiment>

The second embodiment will be described. In the second embodiment of the present invention, icons are displayed for other HMDs within a predetermined distance from the HMD 101. This is because there is a high possibility that it will collide with other HMD users who are within a predetermined distance from their own HMD, so whether or not they actually collide with other HMD users is the user wearing the HMD. This is because there is a low possibility that it will collide with users of other HMDs that are a predetermined distance away from their own HMD.

Hereinafter, with reference to FIG. 13, details of the distance determination processing in the second embodiment of the present invention will be described. Regarding the second embodiment, processing different from/added to the first embodiment will be described, and description of processing common to the first embodiment will be omitted.

The CPU 201 of each device executes the processing of FIG. 4 and the processing up to step S507 of FIG. 5 as described in the first embodiment.

If the CPU 201 of the client PC 100 determines in step S507 of FIG. 5 that the position of another HMD 101 is not within the field of view, the process proceeds to step S1301. In step S1301, the CPU 201 of the client PC 100 refers to the position information of the other HMD 101 that is determined to be out of the field of view and the position information of the HMD 101A corresponding to the own device, and each of the two position information indicates. The distance between the positions is measured to determine whether the other HMD 101 is within a predetermined distance from the HMD 101A. It is assumed that the value of the predetermined distance is stored in the client PC 100 and the server 200 in advance.

If it is determined that the distance is within the predetermined distance, the process proceeds to step S508, and if it is determined that the distance is not within the predetermined distance, the process proceeds to step S518.

That is, only the icons of other HMDs 101 within a predetermined distance from the HMD 101A are placed on the screen as indicated by 1010 in FIG. The above is the description of FIG. 13.

According to the second embodiment, it is possible to provide a mechanism for appropriately displaying the position of another user who is likely to collide with the wearable display device.

<Third Embodiment>

A third embodiment will be described. In the third embodiment of the present invention, an avatar image corresponding to the location of the HMD 101 is displayed.

For example, a situation is conceivable in which an image of an avatar is superimposed around the user and the user becomes an avatar. When arranging and displaying the avatar so as to wrap the user, there is no risk of collision if the user does not actually exist in the avatar, but if the user actually exists in the avatar, the bodies may collide. There is a nature.

Therefore, in the third embodiment, the avatar images are switched and displayed according to the location of the HMD 101.

Hereinafter, with reference to FIG. 14, details of the avatar identification display processing according to the base in the third embodiment of the present invention will be described. Regarding the third embodiment, processing different from/added to the first and second embodiments will be described, and description of processing common to the first and second embodiments will be omitted.

The CPU 201 of each device executes the processing of FIG. 4 and the processing up to step S515 of FIG. 5 as described in the first embodiment.

When the CPU 201 of the client PC 100 determines in step S515 of FIG. 5 that the base ID of the other HMD 101 is the same base (the user in the field of view is the user of the same base), the own base (Step S1401), the avatar image for its own site is placed at a position in the virtual space corresponding to the position/orientation of the other HMD 101, and an MR image superimposed on the real image is generated ( Step S1403).

On the other hand, when it is determined that the site ID of the other HMD 101 and its own site are different sites (the user in the field of view is a user of another site), an avatar image for the other site is acquired (step S1402). , An avatar image for the other base is placed at a position in the virtual space according to the position/orientation of the other HMD 101, and an MR image superimposed on the real image is generated (step S1403). Then, the process proceeds to step S518. The above is the description of FIG. 14.

According to the third embodiment, the wearable display device can appropriately display the position of the user in the field of view and the location of the user.

As described above, according to the present invention, it is possible to provide a mechanism for properly displaying the positions of other users in a wearable display device.

The first embodiment, the second embodiment, and the third embodiment described above can be freely combined.

Further, in the above-described embodiment, the arrangement position of the icon is determined by using the technique described in Japanese Patent Laid-Open No. 8-229238, but a method as shown in FIG. 15 may be used.

Specifically, in the client PC 100, the CPU 201 arranges the cube 1501 including all the positions of the HMDs 101 around the HMD 101 (for example, the HMD 101A) corresponding to the own device, as shown by 1500 in FIG. Then, a straight line from the HMD 101A to the position of another HMD 101 is specified, and the coordinates of the position where the surface of the cube intersects the straight line are specified (example: Y coordinate and Z coordinate of 1511 in FIG. 15). Then, as shown by 1520 in FIG. 15, the surface is assigned to the display area of the corresponding icon, and the surface is deformed according to the size and shape of the display area. After that, the Z coordinate of the point 1511 is set to the intermediate point in the Z axis direction of the display area (shown at 1520 in FIG. 15), and the Y coordinate and the set Z coordinate are determined as the display position of the icon.

It is assumed that the direction of the surface to be assigned to which icon display area is predetermined. The relationship between the HMD and the surface in each direction is shown in 900 to 906 in FIG. Reference numeral 900 indicates an angle of view (field of view (front)). 901 indicates the front, 902 indicates the rear, 903 indicates the left, 904 indicates the right, 905 indicates the upper side, and 906 indicates the lower side.

Further, in the above-described embodiment, an image such as 1010 in FIG. 10 is generated in step S512, but, for example, in the same step, an image as shown in FIGS. 11 and 12 may be generated. Good.

According to 1100 in FIG. 11, an icon display area in which the front, rear, upper, lower, left, and right directions are further subdivided is provided outside the image display area, and icons are arranged and displayed in the icon display area. (In step S512, an image in which the icon display area is provided outside the image display area is generated)

According to 1110 of FIG. 11, for example, an image 1110 that includes the icon display area in the direction in which it is determined that there is another user and does not include the icon display area in the direction in which no other user exists is generated and displayed on the HMD 101 ( In step S512, an image that does not show the icon display area in which no icon is displayed is generated).

According to 1200 of FIG. 12, an icon display area in which the front, rear, upper, lower, left, and right directions are subdivided is provided inside the image display area, and icons are arranged and displayed in the icon display area. (In step S512, an image in which the icon display area is provided inside the image display area is generated)

According to 1210 of FIG. 12, the client PC 100 additionally arranges the identification information of the base of the HMD 101 indicated by each icon to each icon. Further, the name of the user who is wearing the HMD 101 indicated by the icon is embedded in the icon itself. It is assumed that which HMD is associated with which user (is attached) is stored in advance in the client PC 100 and the server 200.

In order to display 1100 and 1110 in FIG. 11 and 1200 in FIG. 12, for example, the CPU 201 of the client PC 100 generates two cubes having different sizes centering on the HMD (eg, HMD 101A) of its own device. (For example, the area A and the area B of 1120 in FIG. 11) are arranged at the position (reference point) of the HMD 101A. The surface of the cube A showing a narrower area is subdivided as indicated by 1120. Then, as shown in 1130 of FIG. 11, when another HMD (example: HMD101D) exists in the region B indicated by the larger cube, the straight line from the HMD101A to the position of the HMD101D is specified, and the straight line is determined. And the coordinates (Y, Z coordinates) of the point 1131 that is the intersection of the surface of the cube indicating the area A and the area (eg, front upper right) on the surface to which the intersection belongs. Then, the surface is deformed according to the shape of the icon display area corresponding to the surface, and the point 1131 on the deformed surface (on the icon display area) is determined as the display position of the icon corresponding to the HMD 101D. The correspondence relationship between the icon display areas corresponding to the surfaces is, for example, as described in 1100 and 1120 of FIG. 11, and the information about the correspondence relationship is stored in advance in the external memory of the client PC 100 and the server 200. ..

When another HMD (eg, HMD101C) exists in the area A indicated by the smaller cube, whether the value of the coordinate of the HMD101C in the Z-axis direction is larger or smaller than HMD101A (that is, HMD101C is higher than HMD101A). Position or lower position). Hereinafter, an example in which the HMD 101C is located lower than the HMD 101A will be described with reference to 1130 in FIG. When the HMD 101C is lower than the HMD 101A, the CPU 201 of the client PC 100 specifies the X and Y coordinates of the point 1132, which is the intersection of the straight line extending vertically downward from the position of the HMD 101C and the surface of the cube B, and The area on the surface to which the point 1132 belongs (for example, lower left rear) is specified. Then, the surface is deformed according to the shape of the icon display area corresponding to the surface, and the point 1132 on the deformed surface (on the icon display area) is determined as the display position of the icon corresponding to the HMD 101D. With the above processing, it is possible to determine the display position of the icon shown in 1100 and 1110 in FIG. 11 and 1200 in FIG. 12 and display the icon.

Further, in the embodiment of the present invention, the base ID 602 shown in FIG. 6 is stored in the client PC 100. However, for example, information on the correspondence relationship between the base ID 602 and the identification information of each client PC 100 or the HMD ID 601 is displayed. By storing and managing in the server 200, the client PC 100 only transmits the identification information of the own device or the HMD ID 601 corresponding to the own device together with the information of the position and orientation of the HMD, and the server 200 side belongs to the base to which the HMD belongs. May be specified. Further, based on the information on the specified base of the HMD 101, for example, to the client PC 100 corresponding to the HMD 101A, together with the information on the position and orientation of each HMD, the icon of the HMD 101B is displayed using the icon for its own base. The HMD 101C/HMD 101D icon is instructed to be displayed using the icon for the other site, and the client PC 100 accepts the command and performs the icon display processing according to the site in accordance with the picture. Good. Similar processing may be performed on the avatar image.

Specifically, the server 200 executes the processing of steps S501 to S509 of FIG. 5, instructs the client PC 100 to acquire/superimpose an icon according to the determination result of S509, and the client PC 100 follows the instruction to execute the icon. Is acquired and superimposed (steps S510 to S512). In addition, the server 200 executes the processing of steps S501 to S507, S514, and S515 of FIG. 5, and when it is determined in S515 that the site is another site, the server 200 instructs the client PC 100 to acquire and display the avatar image, The client PC 100 acquires and superimposes the avatar image according to the instruction (steps S516, S517, S1401 to S1403).

Further, the determination process shown in FIG. 13 may be executed by the CPU 201 of the server 200.

Further, in the embodiment of the present invention, the image for displaying the icon is generated together with the real image. However, for example, the display for displaying the icon display area 1100 or 1110 in FIG. If the image display area is provided separately, in step S512, an image including an icon to be displayed in the icon display area is generated separately from the image to be displayed in the display area of the real image or the MR image, and the image is displayed in the HMD. You may make it output.

Further, in the embodiment of the present invention, the base 1 and the base 2 are described by taking different rooms of different branches as an example, but different spaces may be used in reality. For example, separate rooms existing next to each other in the same branch may be set as base 1 and base 2.

In addition, since it is only necessary that the space is different in reality, the space is not necessarily a room. For example, a space surrounded by a plurality of optical sensors that specify the position and orientation of the HMD by capturing an optical marker attached to the HMD may be used as one base.

As described above, according to the present invention, it is possible to provide a mechanism for properly displaying the positions of other users in a wearable display device.

A recording medium recording a program that realizes the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus reads and executes the program stored in the recording medium. Needless to say, the object of the present invention is achieved.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

As a recording medium for supplying the program, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a DVD-ROM, a magnetic tape, a non-volatile memory card, a ROM, an EEPROM. , A silicon disk, a solid state drive, etc. can be used.

Further, not only the functions of the above-described embodiments are realized by executing the program read by the computer, but also the OS (operating system) running on the computer is actually executed based on the instructions of the program. It goes without saying that a case where some or all of the processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program read from the recording medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the function expansion board is instructed based on the instruction of the program code. Needless to say, this also includes the case where the CPU or the like included in the function expansion unit performs some or all of the actual processing and the processing realizes the functions of the above-described embodiments.

Furthermore, by downloading and reading a program for achieving the present invention from a server, a database, or the like on a network by a communication program, the system or apparatus can enjoy the effects of the present invention. It should be noted that the present invention also includes all configurations that combine the above-described embodiments and modifications thereof.

100A client PC
100B client PC
100C client PC
100D client PC
100E Client PC
100F Client PC
101A HMD
101B HMD
101C HMD
101D HMD
101E HMD
101F HMD
200 servers 150 networks 151 networks 152 networks

Claims (9)

An information processing device capable of communicating with a device that displays an image of an object as if the object exists in a physical space ,
A first control means for the image of the physical object, Utsushidasu the first device in a different second physical space and physical space first device is located within the pre-Symbol device,
Of the physical objects located in the first physical space and the second physical space, a physical object located in the first physical space that cannot be visually recognized by the user who views the image displayed by the first device. Second control means for controlling the information to be displayed on the first device;
An information processing apparatus comprising: The information processing apparatus according to claim 1, wherein the apparatus is one of physical objects. The second control unit has information on a physical object located in the first physical space that is not visually recognized by the user, and information on a physical object located in the second physical space that is not displayed by the second control unit. The information processing apparatus according to claim 1 or 2, which is controlled so as to be distinguished from each other. The second control means controls to project information of the physical object when the distance between the first device and the physical object reaches a predetermined distance. The information processing apparatus according to any one of items. The information processing apparatus according to claim 1, wherein the information on the physical object is information on a position of the physical object. Storage means for storing information capable of specifying which device is located in which physical space among a plurality of physical spaces
The information processing apparatus according to any one of claims 1 to 5, further comprising: A method of controlling an information processing device capable of communicating with a device that displays an image of an object as if the object exists in a physical space ,
A first control step of displaying an image of a real object in a different second physical space, the first device and the real space in which the first device is located within the pre-Symbol device,
Of the physical objects located in the first physical space and the second physical space, a physical object located in the first physical space that cannot be visually recognized by the user who views the image displayed by the first device. A second control step for controlling the step of displaying information on the first device;
Control method including. A program that can be executed by an information processing device that can communicate with a device that projects an image of an object so that the object exists in the physical space,
Said information processing apparatus
A first control means for the image of the physical object, Utsushidasu the first device in a different second physical space and physical space first device is located within the pre-Symbol device,
Of the physical objects located in the first physical space and the second physical space, a physical object located in the first physical space that cannot be visually recognized by the user who views the image displayed by the first device. A program for functioning as second control means for controlling information to be displayed on the first device. An information processing system including an information processing apparatus and an apparatus for displaying an image of an object so that the object exists in a physical space ,
A first control means for the image of the physical object, Utsushidasu the first device in a different second physical space and physical space first device is located within the pre-Symbol device,
Of the physical objects located in the first physical space and the second physical space, a physical object located in the first physical space that cannot be visually recognized by the user who views the image projected by the first device. Second control means for controlling the information to be displayed on the first device;
An information processing system comprising:

Images