JP6820469B2 - Information processing equipment, information processing system, its control method and program - Google Patents

Description

The present invention relates to an information processing device, an information processing system, a control method and a program thereof.

In recent years, mixed reality (hereinafter referred to as MR) technology has become widespread. MR technology can be used to provide a user wearing a head-mounted display (HMD) with a simulated experience of a space in which a real object and a virtual object are arranged. In generating MR space (mixed reality space that combines real space and virtual space), a sensor or a two-dimensional marker may be used as a position index used for aligning the virtual space with the real space.

Patent Document 1 describes a technique of identifying the position and orientation of the HMD from a marker image in the image captured by the HMD, and generating and displaying an image of a virtual object viewed from the position and orientation of the HMD.

Cited Document 2 describes a technique of specifying the position and orientation of the HMD by using a sensor and changing the display form of the virtual object when the user's hand approaches a predetermined virtual object.

In addition, some HMDs are equipped with a line-of-sight tracking function. This is a function of recognizing the movement of the wearer's eyes by the mechanism mounted on the HMD (for example, Patent Document 3). By detecting the line of sight, it is possible to select or identify and display a virtual object in front of the line of sight.

Japanese Unexamined Patent Publication No. 2005-293141 JP-A-2009-169622 Japanese Unexamined Patent Publication No. 2011-224213

For example, you may want to simulate work related to a product using mixed reality. For example, when examining the arrangement and shape of a car engine and body, display a virtual object of the car engine part, hold a maintenance tool such as a spanner in your hand, and insert the tool between the engine and the car body. The design may be done while confirming that there is a sufficient gap and that there is a sufficient gap for the hands and arms.

Real tools can be used, but in some cases, tools can be replaced by virtual objects. In this case, in order to convey the feeling that the user is holding the tool, the user may be given a real object to which a marker is attached, and the virtual object of the tool may be superimposed on the marker to experience mixed reality. It is done.

And this is not limited to the engine, but depending on the product, it is necessary to use multiple tools properly. Therefore, in order to perform the simulation, it is necessary to change the tool at hand to the optimum tool each time. If it is troublesome to switch the real object, it is possible to switch the virtual object corresponding to the marker by operating the PC that manages the display state of the virtual object, but move it in front of the PC one by one. It is time consuming and time consuming to operate.

An object of the present invention is to provide a mechanism capable of easily switching the display of a virtual object according to the line of sight of the user.

The present invention is an information processing device including a head-mounted display device, and identifies an object of interest that the user is paying attention to from the line of sight of the user wearing the head-mounted display device. A determination means for determining a virtual object corresponding to the object of interest as a virtual object to be displayed on the head-mounted display device, separately from the specific means and the object of interest specified by the specific means. The determination means includes display control means for controlling the virtual object determined to be displayed by the determination means so as to be displayed on the head-mounted display device, and the determination means is the focus of the user specified by the specific means. When the target object becomes another object, the display control means determines that the virtual object already displayed on the head-mounted display device is a virtual object corresponding to the other object. The display control means is characterized in that a virtual object corresponding to the other object determined by the determination means is controlled to be displayed on the head-mounted display device.

According to the present invention, it is possible to provide a mechanism capable of easily switching the display of a virtual object according to the line of sight of the user.

It is a figure which shows an example of the structure of the information processing system in 1st Embodiment of this invention. It is a figure which shows an example of the hardware composition of various devices in 1st Embodiment of this invention. It is a figure which shows an example of the functional structure of various devices in 1st Embodiment of this invention. It is a flowchart which shows the flow of the display process of a virtual object in 1st Embodiment of this invention. It is a flowchart which shows the flow of the identification process of the object of interest by the line of sight in the 1st Embodiment of this invention. It is a flowchart which shows the flow of the display change processing of the virtual object according to the object of interest in 1st Embodiment of this invention. It is a figure which shows an example of various data structures in 1st Embodiment of this invention. It is a figure which shows the state of the display change of the virtual object according to the object of interest in the 1st Embodiment of this invention. It is a flowchart which shows the flow of process in 2nd Embodiment of this invention.

Hereinafter, an example of the embodiment of the present invention will be described with reference to the drawings.

First, with reference to FIG. 1, an example of the configuration of the information processing system according to the first embodiment of the present invention will be described.

As shown in FIG. 1, the various devices of the present invention are communicably connected via a network. For example, the PC 100 is communicatively connected to the HMD 101, which is a head-mounted display (head-mounted display device), by a cable.

That is, the PC 100 includes the HMD 101 as one of the display devices, and the image generated by the PC 100 can be output to the HMD 101 and displayed.

The PC 100 stores a three-dimensional model superimposed on a real image captured by the HMD 101. The three-dimensional model that is CG with respect to a real object is called a virtual object.

The PC 100 acquires a real image from the HMD 101 managed by its own machine, generates a mixed reality image (MR image) in which an image of a virtual object is superimposed on the real image, and displays the MR image on the display of the HMD 101. Send to HMD101. The HMD 101 displays the received MR image on the display.

The HMD 101 is provided with a mechanism for detecting the line of sight of the user wearing the HMD 101, and transmits the detected line of sight information to the PC 100.

The optical sensor 104 is a sensor that measures the position and orientation of the three markers 103 installed on the HMD 101, and is a device for measuring the position and orientation of the HMD 101 provided with a predetermined offset from the marker 103.

The target 105 is, for example, a portable real object to which a marker is attached. By setting the position of the virtual object at the position of the marker attached to the target and generating / displaying the MR image, the user can easily move the virtual object simply by moving the target. The above is the description of FIG.

Next, an example of the hardware configuration of various devices according to the first embodiment of the present invention will be described with reference to FIG.

The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

Further, the ROM 202 stores various programs necessary for realizing the functions executed by the BIOS (Basic Input / Output System) which is the control program of the CPU 201, the operating system (OS), and other various devices.

The RAM 203 functions as a main memory, a work area, and the like of the CPU 201. The CPU 201 realizes various operations by loading a program or the like necessary for executing a process into the RAM 203 and executing the program.

Various programs and the like used by the PC 100 of the present invention to execute various processes described later are recorded in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as needed. Further, the definition file and various information tables used by the program according to the present invention are stored in the external memory 211.
The input controller (input C) 205 controls the input from a pointing device (input device 210) such as a keyboard and a mouse.

The video controller (VC) 206 controls the display on the display such as the right-eye / left-eye display 222 included in the HMD 101. For the right-eye / left-eye display 222, for example, an external output terminal (for example, Digital Visual Interface) is used for output. The right-eye / left-eye display 222 is composed of a display for the right eye and a display for the left eye.

The memory controller (MC) 207 is an adapter to a hard disk (HD), a flexible disk (FD), or a PCMCIA card slot for storing boot programs, browser software, various applications, font data, user files, editing files, various data, and the like. Controls access to an external memory 211 such as a card-type memory connected via.

The communication I / F controller (communication I / FC) 208 connects and communicates with an external device via a network, and executes communication control processing on the network. For example, Internet communication using TCP / IP is possible. The communication I / F controller 208 also controls communication with the optical sensor 104 via Gigabit Ethernet (registered trademark) or the like.

The general-purpose bus 209 is used to capture images from the right-eye / left-eye video camera 221 of the HMD 101. Input is performed from the right-eye / left-eye video camera 221 using an external input terminal (for example, an IEEE1394 terminal). The right-eye / left-eye video camera 221 is composed of a video camera for the right eye and a video camera for the left eye.

The CPU 201 can display the outline font on the display by executing the outline font expansion (rasterization) process on the display information area in the RAM 203, for example. Further, the CPU 201 enables a user instruction with a mouse cursor or the like (not shown) on the display. The above is the description of FIG.

Next, with reference to FIG. 3, an example of the functional configuration of various devices according to the first embodiment of the present invention will be described.

The imaging unit 311 is an imaging unit that captures an captured image (real image) by a camera device. The captured image transmission unit 312 is a transmission unit that transmits the captured image acquired by executing the imaging process in the imaging unit 311 to the PC 100.

The line-of-sight information transmission unit 314 is a functional unit that generates information (line-of-sight information) related to the line of sight of the user wearing the HMD 101 and transmits it to the PC 100. For example, as described in JP-A-2010-199789 and JP-A-2016-127587, the HMD is provided with an invisible light irradiating device or a half mirror that irradiates the user's eye with invisible light. Install and capture invisible light reflected by the user's eyes Install a camera to capture the reflected light and where the user's eyes are looking on the display (on the real image displayed on the display screen) It is measured and specified with a certain sense of time. Then, the information on the position (X, Y) of the specified display information is sequentially transmitted to the PC 100 as line-of-sight information.

The MR image receiving unit 315 is a receiving unit that receives the MR image transmitted from the PC 100, and the MR image display unit 316 is a display unit that displays the MR image.

The captured image receiving unit 321 is a storage unit that receives and stores the captured image from the HMD 101. The HMD position / posture storage unit 322 is a storage unit that acquires and stores the position / posture of the HMD 101. The position and orientation of the HMD is specified by the optical sensor 104 when a plurality of (for example, three or more) markers (light reflection markers) installed in the HMD 101 are imaged by the optical sensor 104 (motion capture camera). The PC 100 receives and stores information on the position and orientation of the HMD 101 (values of the X, Y, Z coordinates and orientation of the HMD 101) from the optical sensor 104.

The virtual object information storage unit 323 is a storage unit that stores a file that stores the shape of the virtual object, information on which marker position the virtual object is displayed on, and information on the position and orientation of the virtual object in the virtual space. is there. In the present embodiment, the marker associated with the virtual object is a two-dimensional marker.

The MR image generation unit 324 is based on the image of the real space captured and transmitted by the right-eye / left-eye video camera of the HMD 101 and the latest position / orientation (position and orientation) of the HMD 101 stored in the HMD position / orientation storage unit 322. It is a functional unit that generates a mixed reality image (MR image) in which an image of an captured virtual space (an image of a virtual object) is superimposed. The MR image transmission unit 325 transmits the MR image generated by the MR image generation unit 324 so as to be displayed on the HMD 101.

The line-of-sight information acquisition unit 326 receives and acquires line-of-sight information from the HMD 101. The object identification unit 327 of interest identifies the object that the user is looking at on the display of the HMD 101 by using the line-of-sight information. That is, the object (object of interest) that the user is paying attention to is specified in the line of sight of the user.

The corresponding virtual object identification unit 328 identifies the virtual object corresponding to the object of interest specified by the object of interest identification unit 327.

For example, when the user is paying attention to the object of interest, a virtual object to be superimposed and displayed on the MR image is specified separately from the object of interest.

For example, the CG model of the spanner, which is a tool corresponding to the CG model of the nut that the user is paying attention to, is specified as a corresponding virtual object by referring to a table that stores the correspondence relationship between the nut and the spanner.

The display virtual object setting unit 329 sets the virtual object specified by the corresponding virtual object identification unit 328 as a virtual object used for generating the MR image, that is, as a virtual object to be displayed on the display screen of the HMD 101. Already displayed

For example, instead of the CG model of the tool that has already been positioned at the user's hand, the CG of the tool newly specified by the corresponding virtual object identification unit 328 according to the object of interest of the user is stored in the memory as the CG to be displayed. Remember.

When generating the MR image, the MR image generation unit 324 superimposes the virtual object stored in the table on the real image to generate the MR image. The above is the description of FIG.

The processing of each step of the flowcharts of FIGS. 4 to 6 and 9 described below is executed by the CPU 201 of the PC 100 using the functions of the functional units of the own machine.

Next, with reference to FIG. 4, the flow of the display processing of the virtual object in the first embodiment of the present invention will be described.

In step S401, the PC 100 reads the list of virtual objects stored in the external memory 211 onto the RAM 203, and uses the list of virtual objects used to generate the MR image, that is, as a list of CGs to be displayed in the HMD 101. Hold.

Further, other information necessary for generating the MR image is read from the external memory 211 and held in the RAM 203.

The information of the list of virtual objects stored in the external memory is, for example, the virtual object information 730 of FIG. The model name 731 is a file name of a file that stores the shape of the virtual object, and is the name of the virtual object.

The file name is a character string including a URL indicating a storage location in the external memory 211 of the PC 100 of the virtual object.

Type 732 is a type of virtual object. Corresponding model 733 is another virtual object corresponding to the virtual object. The corresponding model 733 is, for example, a file name of a virtual object displayed in the user's hand while the user is paying attention to the model name 731.

The position 734 and the posture 735 indicate the position and posture of the virtual object of the model name 731 in the virtual space. In the present embodiment, the position and posture in the virtual space = the position and posture in the real space. That is, a mixed reality space in which the position of the virtual space and the position of the real space match is defined.

The virtual object in which the values are inserted in the position 734 and the posture 735 is an object in which the position and posture are fixed. A virtual object for which no value is inserted in the position 734 and the posture 735 shall change the position / posture in conjunction with the position / posture of the marker associated with the virtual object.

Further, the other information necessary for generating the MR image is, for example, the target information 710, the target correspondence information 720, and the real object information 740 in FIG. 7.

The target information 710 is, for example, information related to a target which is a portable real object to which a marker is attached.

The target ID 711 is the identification information of the target which is a real object to which the marker of the marker ID 712 is attached.

The marker ID 712 is the name of the two-dimensional marker, and is the name of the marker image which is an individual identification image. The marker ID 712 is a character string including the name of the marker and the address on the external memory 211 of the PC 100 in which the image of the marker is stored. The size 713 is the vertical and horizontal size of the marker image.

The position 714 and the posture 715 are the size and inclination of the marker image included in the captured image captured by the camera of the HMD 101, and the position and posture of the marker of the marker ID 712 specified from the position and posture of the HMD 101. That is, it is the position and orientation of the target indicated by the target ID 711, and is the position and orientation that fluctuates in real time.

The detecting flag 716 is a flag indicating whether or not the marker position of the target marker ID 712 is being detected / updated. The detecting flag 716 = 1 indicates that the marker of the marker ID 712 is being imaged by the camera, and the position of the marker is being detected / updated. The detecting flag 716 = 0 indicates a state in which the camera cannot image the marker of the marker ID 712 and the position of the marker cannot be detected.

The PC 100 switches and updates the detecting flag 716 according to whether or not the marker can be recognized in the real image acquired from the HMD 101.

The target correspondence information 720 is a data table that stores and manages the target (target ID 721), the marker (marker ID 722) attached to the target, and the corresponding virtual object (model name 723).

The offset position 724 and the offset posture 725 indicate how far away from the position and posture of the marker ID 722 shown in the position 714 and the posture 715, and how much the tilt direction is changed as the position and posture of the virtual object of the model name 723. , Information on the position and orientation of the virtual object.

That is, the target correspondence information 720 is information in which the target, the marker attached to the target, and the virtual object are associated with each other.

The real object information 740 is information that sets which virtual object (corresponding model 742) is to be displayed when the user is paying attention to which type (object type 741) of the real object.

In step S401, the PC 100 generates and holds the arrangement virtual object information 750 on the RAM 203. The arrangement virtual object information 750 is a table that stores the position / orientation (position 754 / attitude 755) of the virtual object (model name 753) actually used for generating the MR image.

The target ID 751 and the marker ID 752 indicate the target and the marker to which the virtual object of the model name 753 is currently associated.

For example, a virtual object whose position and orientation are stored in the virtual object information 730 does not insert values into the target ID 751 and the marker ID 752, but inserts the model name and the position and orientation into 753 to 755, respectively, to generate an MR image. It is stored in the arrangement virtual object information 750 as a virtual object used for.

The PC 100 repeats the processes of steps S402 to S407 from the start instruction of the MR experience to the end instruction by the user. For example, it is repeatedly executed at intervals of 0.5 seconds.

In step S402, the PC 100 acquires the position / orientation information of the HMD 101 from the optical sensor 104 and stores it in the external memory. Specifically, the HMD information 700 in FIG. 7 stores the HMD ID 701, which is the identification information of the HMD 101 that has acquired the position / orientation, and the position 702 and the attitude 703 in the real space of the HMD 101 acquired from the optical sensor 104. To do.

In step S403, the PC 100 acquires a real image captured by the camera of the HMD 101 and stores it in the memory.

Then, the position and orientation of the detected real object are specified and stored by detecting the marker image in the real image. Specifically, the position / orientation of the marker detected in the real image in the target information 710 of FIG. 7 is overwritten / updated to the position 714 and the attitude 715. That is, one posture of the target is specified (step S404).

In step S405, the PC 100 refers to the target correspondence information 720 and identifies a virtual object corresponding to the marker detected from the real image in step S402. Then, the position / orientation of the virtual object is determined and stored according to the position / orientation of the target corresponding to the virtual object.

Specifically, the file name of the virtual object is stored in the model name 753, and the target ID and the marker ID associated with the target ID 751 and the marker ID 752 are inserted and stored.

Then, the coordinates are shifted by the offset position 724 and the offset posture 725 associated with the virtual object in the target correspondence information 720 from the latest position / posture of the target stored in 714 and 715, and the tilted position / posture is set. It is inserted and stored in the positions 754 and 755 as the position and orientation of the virtual object of the model name 753.

In step S406, the PC 100 refers to information on the angle of view of the camera of the HMD 101 and the position / orientation of the HMD 101, which is not shown, and the latest information on the position / orientation of the virtual object stored in the arrangement virtual object information 750. Then, an MR image for the HMD 101 is generated, output (transmitted) to the HMD 101 in step S407, and displayed (corresponding to the display control means). The above is the description of FIG.

Next, with reference to FIG. 5, the flow of the process of identifying the object of interest by the line of sight in the first embodiment of the present invention will be described.

The PC 100 repeats the processes of steps S501 to S504 at predetermined time intervals from the start instruction of the MR experience to the end instruction of the user. For example, it is repeatedly executed at intervals of 0.5 seconds.

In step S501, the PC 100 acquires the line-of-sight information of the user who is the wearer wearing the HMD 101 and who has experienced MR.

The line-of-sight information in the embodiment of the present invention is, for example, as described in JP-A-2010-199789 and JP-A-2016-127587, which is installed in HMD 101 and is not directed toward the user's eye. The user's eye obtained by imaging the reflected light of the eye with the HMD 101 using an invisible light irradiator that irradiates visible light, a half mirror, and a camera that captures the invisible light reflected by the user's eye. Is information indicating the X and Y coordinates on the display, which indicates where is being viewed on the display (on the actual image displayed on the display screen).

In step S502, the PC 100 determines whether the virtual object is superimposed on the position on the display that the user is looking at.

For example, information on the position of the user's eye itself in the HMD 101, information on the distance from the eye to the display of the HMD 101 (both are fixed positions and distances preset and stored in the HMD 101 and PC 100), and the user can display the information. In the arrangement virtual object information 750, it is determined whether or not the virtual object is located in the direction in which the user's eyes are facing, which is the information indicating where the user is looking, and the position and orientation of the HMD 101 itself.

When there is a virtual object in front of the line of sight, the virtual object closest to the HMD 101 in front of the line of sight is specified as the object of interest that the user is paying attention to, and the virtual object information 730 is stored in the RAM 203 (step S503). ).

Further, when there is no virtual object in front of the line of sight, the position on the display that the user is paying attention to and the image displayed on the display at the time of attention are present at the position of the real image that the user is paying attention to. A real object is recognized by executing a known image analysis process, and the specified real object information (for example, object type information indicating what the real object is) is obtained by the user's object of interest. Is stored in the RAM 203 as the information of (step S504). The above is the description of FIG.

Next, with reference to FIG. 6, the flow of the display change processing of the virtual object according to the object of interest in the first embodiment of the present invention will be described.

In step S601, the PC 100 sets the display mode of the virtual object to the automatic switching mode that automatically switches the display of the virtual object according to the object that the user is paying attention to in response to the user operation on the mode switching screen (not shown). change.

In step S602, the PC 100 refers to the target information 710 and determines in step S603 whether there are a plurality of targets being detected.

When there is only one target being detected, the target is specified as a target to be processed and stored in the RAM 203. If there are a plurality of targets being detected, the process proceeds to step S604, and the target at the position closest to the position of the HMD 101 is specified / selected as the processing target and stored.

In step S605, the PC 100 acquires the latest information of the object of interest stored in step S503 or step S504. For example, information on the object type of the object of interest is acquired.

Then, when the object of interest is a virtual object, the virtual object information 730 is referred to, and when the object of interest is a real object, the real object information 740 is referred to to identify the virtual object corresponding to the object of interest (step S606). ). In the virtual object information 730, the virtual object of the corresponding model 733 is specified, and in the real object information 740, the virtual object of the corresponding model 742 is specified.

If there is no corresponding virtual object (when it is not stored in the virtual object information 730 and the real object information 740), the process of FIG. 6 ends. In other words, there is no virtual object to replace with the target.

Therefore, if there is already a virtual object stored in association with the target to be processed in the placement virtual object information 750, the placement virtual object information 750 is stored in association with the target as before. The display processing of the virtual object being performed will be performed according to the movement of the target.

In step S607, the PC 100 stores the file name of the virtual object specified in step S606 in the arrangement virtual object information 750 in association with the target ID and the marker ID of the target specified as the processing target in steps S603 to S604.

Specifically, the target ID of the target to be processed is stored in the target ID 751, the marker ID of the marker attached to the target to be processed is stored in the marker ID 752, and the file name of the virtual object is stored in the model name 753. That is, the position of the virtual object is determined to be the position 754 and the posture 755.

When the same target ID and marker ID data already exist in the arrangement virtual object information 750, the model name 753 of the data is rewritten to the file name of the virtual object specified in step S606, and stored and updated. The above is the description of FIG.

In the present embodiment, the process of FIG. 4 (MR image generation process using the arrangement virtual object information 750) is performed at least 0.5 seconds after the process of step 607 is completed.

That is, by executing the process of FIG. 6, the display of the virtual object can be easily switched according to the line of sight of the user, specifically, the object of interest of the user specified by the line of sight of the user.

For example, FIG. 8 is a diagram showing a state in which the display of a virtual object is changed according to the object of interest. Reference numerals 800, 810, and 820 are examples of display screens displayed on the HMD 101, and MR images are displayed.

801 indicates the position that the user is paying attention to, that is, the line-of-sight information of the user. Reference numeral 105 is a target, and 802 is a virtual object currently stored in association with the target in 750, and is arranged in the virtual space at the positions of 754 and 755.

As shown in 810, when the user pays attention to natA, the PC100 refers to the virtual object information 730 of FIG. 7, and is a virtual object corresponding to natA. vrml (spannerA) is specified, the position is determined at the position of the target 105, an MR image is generated, and the HMD 101 is transmitted / displayed.

802 to S00A. The state of switching the virtual object to vrml is shown in the arrangement virtual object information 750 of FIG.

Further, as shown in 820, when the user pays attention to natB, the PC100 refers to the virtual object information 730 of FIG. 7, and is a virtual object corresponding to natB. vrml (spannerB) is specified, the position is determined at the position of the target 105, an MR image is generated, and the HMD 101 is transmitted / displayed.

As described above, according to the first embodiment of the present invention, it is possible to provide a mechanism capable of easily switching the display of a virtual object according to the line of sight of the user.

For example, the display of a virtual object other than the object that the user is paying attention to can be easily changed according to the line of sight of the user.

Further, when there are a plurality of targets, the display of the virtual object can be changed by narrowing down the targets.

<Second embodiment>

A second embodiment of the present invention will be described with reference to FIG. The processing, hardware configuration, functional configuration, data configuration, and the like common to the first embodiment will not be described.

In step S603, the PC 100 determines that there is only one target being detected and the target to be processed is specified, or when the target to be processed is specified in step S604, step S901 of FIG. 9 Execute the process.

The PC 100 sequentially stores in the external memory 211 the history of the position and posture of each target of the target information 910 from the start of the MR experience to the reception of the end instruction.

In step S901, the PC 100 determines whether the user has the target to be processed.

Specifically, the history of the position and posture of the target is acquired for the past predetermined time, and when the target is moving, it is determined that the user has the target in his / her hand. The predetermined time is, for example, 1.5 seconds.

When the target is placed on a desk or the like, there should be no change in position or attitude in the past 1.5 seconds, or it should be within a predetermined swing width.

In the PC100, when the change in the position / orientation value of the target is equal to or greater than a predetermined value (X, Y, Z values stored in the external memory in advance), the target is moving (moving), that is, , Determines that the user has the target in his hand (corresponds to the possessed object identification means).

When the target is held in the user's hand, the process proceeds to step S605 of FIG. That is, when the target is held in the user's hand, the virtual object corresponding to the object of interest is displayed in association with the virtual object held in the user's hand.

On the other hand, if the target is not in the user's hand (if it is, etc.), the process ends. The above is the description of FIG.

As described above, according to the second embodiment of the present invention, it is possible to provide a mechanism capable of easily switching the display of the virtual object held in the user's hand according to the line of sight of the user. ..

In other words, when the user pays attention to something while holding the real object in his hand, it is presumed that he is trying to do something with the object of interest, and depending on the object of interest. A virtual object can be displayed at the position of a real object held in the hand, and the display can be switched.

By the switching process, it is possible to reduce the time and effort that the user has to switch the virtual object corresponding to the real object.

In addition, when the user pays attention to something while the user holds the virtual object in his / her hand, it is presumed that he / she is trying to do something with the object of interest, and depending on the object of interest. You can switch the display of the virtual object you have in your hand.

By the switching process, it is possible to reduce the trouble of switching the virtual object held by the user.

In the above description, it is determined that the target is held in the user's hand when the target is moving. For example, in the placement virtual object information 750, the virtual currently associated with the target is virtual. When the object is moving, it may be determined that the target (virtual object at the target position) is held in the user's hand.

Further, the determination can be made by another method. For example, a hand-shaped virtual object is placed in the user's hand, and the position of the virtual object of the hand and the position of the target are within a predetermined distance (for example, within 5 cm), or the virtual object of the hand is associated with the current target. When the virtual object is in contact with the object, it may be determined that the target is held in the user's hand.

The data of the virtual object of the hand is stored in the external memory 211 in advance. Specifically, the arrangement of the virtual object of the hand recognizes the skin color of the user's hand from the real image captured by the HMD 101, calculates the distance to the hand by triangulation, and calculates the distance to the hand from the positional relationship between the HMD 101 and the hand. It can be realized by calculating the position / posture of the hand, determining the position / posture of the virtual object of the hand as the position / posture of the hand, and storing it in the placement virtual object information 750.

When the target is moving and touching the virtual object of the hand, it may be determined that the target (the virtual object at the position of the target) is held in the user's hand.

In the above-described embodiment, the virtual object corresponding to the real object of interest is displayed in the user's hand even when the real object is focused on, but what is the user by analyzing the real image? It is considered that a considerable amount of processing resources are consumed to identify whether or not the focus is on. In addition, it may take time to complete the analysis.

Therefore, for example, when it is determined in step S503 of FIG. 5 that there is no virtual object ahead of the line of sight, the process may be returned to step S501 without performing the analysis process of the real object.

As a result, the consumption of processing resources can be suppressed, and when the user is paying attention to the virtual object, the display of a virtual object different from the virtual object can be easily switched.

Further, for example, when the target to be focused on by the user is a small part, it is difficult to keep the line of sight on the small part. In some cases, it is easier to manage data by dividing the virtual object that is the base on which the part is attached by part, because the divided part is larger than the part and it is easier to manage the data than associating the virtual object with each part. Is. In addition, the larger the target, the easier it is to maintain the state of interest.

Therefore, for example, one virtual object is divided into parts, and the corresponding virtual object is stored for each divided part (space / area defined by four or more XYZ coordinates in the virtual space), and the user May focus on the part, position and display the virtual object corresponding to the part in association with the target.

As described above, according to the present invention, it is possible to provide a mechanism capable of easily switching the display of a virtual object according to the line of sight of the user.

For example, the display of the virtual object corresponding to the marker at the user's hand can be easily switched according to the place where the user is looking.

Further, in the above-described embodiment, the HMD 101 detects the user's line of sight, but for example, the PC 100 acquires the direction itself in which the HMD 101 is facing as the user's line-of-sight direction, and the nearest direction in which the HMD 101 is facing. The object of interest may be specified as the object of interest.

In the above-described embodiment, the position and orientation of the HMD 101 are detected and acquired by using an optical sensor. For example, as described in JP-A-2002-229730, a magnetic sensor, an ultrasonic sensor, etc. It may be detected by using. That is, the means for detecting the position and orientation of the HMD does not matter.

Further, a marker capable of detecting the position and orientation by the optical sensor 104 shown in 103 may be attached to the target so that the position and orientation of the target can be detected, stored and updated.

Further, for example, the position and orientation of the HMD 101 may be specified and acquired by using an image of a two-dimensional marker. Specifically, for example, the position and orientation of the HMD 101 are specified from the information of the two-dimensional marker (hereinafter referred to as a marker) included in the real image acquired from the HMD 101. It is assumed that a plurality of markers are attached in the actual room where the HMD is imaged. The shape of the markers does not matter, but in the present embodiment, they are square and all have the same size. It is assumed that a unique marker number is embedded in each marker. Then, it is assumed that each marker can be identified and the marker number can be obtained when decoding. It is assumed that the markers are attached to known positions in advance so that at least three markers are included in the field of view of the built-in camera regardless of the position and orientation of the HMD 101.

The PC 100 obtains the distance from the HMD 101 to each of the three position detection markers (whose positions are known) in the captured real image. Then, the position where the three distances obtained from the three position detection markers overlap is determined as the position of the HMD 101. Further, the posture of the HMD 101 may be obtained from the arrangement of the three position detection markers in the captured image.

When the viewpoint is in the normal direction of the marker, the marker image looks like a square. Then, when the viewpoint deviates from the normal direction, the square appears distorted according to the degree of the deviation. That is, from this distortion, the orientation of the plane defined by the marker with respect to the axis of the viewpoint can be found, the distance between the viewpoint and the marker can be detected from the size of the marker, and the plane to which the marker is attached can be defined. Further, in the embodiment, the markers are provided with two marks that can be distinguished from each other, the center position of the markers is set as the origin, and two vectors from the origin to the marks on the above plane defined by the markers are set to 2. It is assumed that one axis in the normal direction from the center position of the axis and the marker defines three axes that define local (local) three-dimensional coordinates.

The type of marker (the type of role played by the marker) is used as a position detection marker for determining the position and orientation of the HMD 101 and for drawing a virtual object (three-dimensional model) at a location defined by the marker. There are two types of placement markers.

The placement marker is a marker used to specify a position where a virtual object is displayed. When the placement marker is detected, the CPU 201 of the PC 100 displays a virtual object at a position where the placement marker exists (to be exact, a position offset from the center of gravity position of the placement marker as described later). Perform the processing for. For example, a process of arranging a virtual object at a position on the virtual space indicated by the offset is performed.

The data of the virtual object is stored in the external memory of the PC 100 in association with the position / orientation information indicating the position and the posture of the virtual object in the virtual space.

The coordinates of the center position of the above-mentioned placement marker are {Xv, Yv, Zv}, and the coordinates in the data to identify the shape of the virtual object are {Xi, Yi, Zi} (here, i = 0, 1, 2). , ...), the distance L is obtained by the following equation.
L = {(Xv-Xi) 2+ (Yv-Yi) 2+ (Zv-Zi) 2} 1/2

Here, the minimum distance L when i is changed is defined as the distance between the placement marker and the virtual object. In the above, the square root was finally obtained when calculating the distance, but if it is sufficient to determine the magnitude, it is not necessary to obtain the parallel root, and the sum of squares of the coordinate differences may be calculated. ..

In the above equation, the center position of the placement marker = the center position of the virtual object. As described above, the position where the virtual object is placed is given an offset with respect to the center position of the placement marker.

Although it depends on the number of bits that can be embedded in the marker, a marker that also serves as a position detection marker and a placement marker may be defined and stored in the external memory.

It should be noted that the above embodiment is an example according to the present invention. For example, in order to detect the position and orientation of the HMD 101, in the above embodiment, it is assumed that there are three position detection markers in the field of view imaged, and a large number of position detection markers are attached in the room. The present invention is not limited thereto. The position detection marker has a mark that defines the orientation, and if not only the coordinates of the position detection marker but also the shape and dimensions are known, the position detection distortion, size, and mark position of the captured image can be used as 1 The position and orientation of the HMD 101 can be specified only from one position detection marker.

The present invention includes a software program that realizes the functions of the above-described embodiments, which is directly or remotely supplied to the system or device. The present invention also includes cases where the computer of the system or device can also read and execute the supplied program code.

Therefore, in order to realize the functional processing of the present invention on a computer, the program code itself installed on the computer also realizes the present invention. That is, the present invention also includes a computer program itself for realizing the functional processing of the present invention.

In that case, as long as it has a program function, it may be in the form of an object code, a program executed by an interpreter, script data supplied to the OS, or the like.

Recording media for supplying programs include, for example, flexible disks, hard disks, optical disks, optical magnetic disks, MOs, CD-ROMs, CD-Rs, CD-RWs, and the like. There are also magnetic tapes, non-volatile memory cards, ROMs, DVDs (DVD-ROM, DVD-R) and the like.

In addition, as a program supply method, a browser of a client computer is used to connect to an Internet homepage. Then, the computer program itself of the present invention or a compressed file including the automatic installation function can be supplied from the homepage by downloading it to a recording medium such as a hard disk.

It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from different homepages. That is, the present invention also includes a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer.

Further, the function of the above-described embodiment is realized by the computer executing the read program. In addition, based on the instruction of the program, the OS or the like running on the computer performs a part or all of the actual processing, and the function of the above-described embodiment can be realized by the processing.

Further, the program read from the recording medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer. After that, based on the instruction of the program, the function expansion board, the CPU provided in the function expansion unit, or the like performs a part or all of the actual processing, and the function of the above-described embodiment is also realized by the processing.

It should be noted that the above-described embodiments merely show examples of embodiment in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

100 PC
101 HMD
103 Marker 104 Optical Sensor 150 Network

Claims (10)

An information processing device equipped with a head-mounted display device.
A specific means for identifying an object of interest that the user is paying attention to from the line of sight of the user wearing the head-mounted display device, and
A determination means for determining a virtual object corresponding to the object of interest as a virtual object to be displayed on the head-mounted display device, in addition to the object of interest specified by the specific means.
A display control means for controlling a virtual object determined to be displayed by the determination means to be displayed on the head-mounted display device, and a display control means.
A possession object identification means for identifying the virtual object held in the user's hand, and
With
When the object of interest of the user specified by the specific means becomes another object, the determination means obtains the virtual object held in the user's hand specified by the possessed object identification means. , Decided to make it a virtual object corresponding to the other object,
The display control means is an information processing device that controls a virtual object corresponding to the other object determined by the determination means so as to be displayed on the head-mounted display device. The information processing device according to claim 1 , wherein the possessed object identifying means identifies a virtual object touching a virtual object in the user's hand as a virtual object held in the user's hand. .. The information processing device according to claim 1 or 2 , wherein the possessed object identifying means identifies a virtual object moving in space as a virtual object held in the user's hand. An information processing device equipped with a head-mounted display device.
A specific means for identifying an object of interest that the user is paying attention to from the line of sight of the user wearing the head-mounted display device, and
A determination means for determining a virtual object corresponding to the object of interest as a virtual object to be displayed on the head-mounted display device, in addition to the object of interest specified by the specific means.
A display control means for controlling a virtual object determined to be displayed by the determination means to be displayed on the head-mounted display device, and a display control means.
An information processing device provided with a head-mounted display device and storing the virtual object to be displayed in association with a predetermined real object.
A specific means for identifying an object of interest that the user is paying attention to from the line of sight of the user wearing the head-mounted display device, and
Said determining means, said determining separately from the identified objects of the aimed object by a specific means, a virtual object corresponding to the object of the aimed object, as a virtual object that displays in association with the predetermined physical object The means of deciding what to do
A display control means for controlling a virtual object determined to be displayed by the determination means to be displayed on the head-mounted display device, and a display control means.
With
Wherein the display control unit, wherein as determined by the determining means, wherein the virtual object corresponding to the object aimed object, the predetermined physical object information you and controls so as to display the position of the paper processing apparatus. When there are a plurality of the predetermined real objects, the positions of the virtual objects whose display is controlled by the display control means are set according to the positions of the head-mounted display devices and the positions of the plurality of the predetermined real objects. A selection means for selecting the predetermined real object and
With
The fourth aspect of the present invention is characterized in that the determination means determines a virtual object corresponding to the object of interest as the virtual object to be displayed in association with the predetermined real object selected by the selection means. The information processing device described. A control method for an information processing device equipped with a head-mounted display device.
A specific step of identifying an object of interest that the user is paying attention to from the line of sight of the user wearing the head-mounted display device.
A determination step of determining a virtual object corresponding to the object of interest as a virtual object to be displayed on the head-mounted display device, separately from the object of interest specified by the specific step.
A display control step of controlling the virtual object determined to be displayed by the determination step to be displayed on the head-mounted display device, and
A possession object identification process for identifying the virtual object held in the user's hand, and
Including
In the determination step, when the object of interest of the user specified by the specific step becomes another object, the virtual object held in the user's hand specified by the possessed object identification step is used. , Decided to make it a virtual object corresponding to the other object,
The display control step is a control method characterized in that a virtual object corresponding to the other object determined by the determination step is controlled to be displayed on the head-mounted display device. A program for controlling an information processing device equipped with a head-mounted display device.
The information processing device
A specific means for identifying an object of interest that the user is paying attention to from the line of sight of the user wearing the head-mounted display device, and
A determination means for determining a virtual object corresponding to the object of interest as a virtual object to be displayed on the head-mounted display device, in addition to the object of interest specified by the specific means.
A display control means for controlling a virtual object determined to be displayed by the determination means to be displayed on the head-mounted display device, and a display control means.
To function as a possession object specifying means for specifying the virtual object being held in the hand of the user,
When the object of interest of the user specified by the specific means becomes another object, the determination means obtains the virtual object held in the user's hand specified by the possessed object identification means. , Decided to make it a virtual object corresponding to the other object,
The display control means is a program characterized in that a virtual object corresponding to the other object determined by the determination means is controlled to be displayed on the head-mounted display device. An information processing system that includes a head-mounted display device and an information processing device.
A specific means for identifying an object of interest that the user is paying attention to from the line of sight of the user wearing the head-mounted display device, and
A determination means for determining a virtual object corresponding to the object of interest as a virtual object to be displayed on the head-mounted display device, in addition to the object of interest specified by the specific means.
A display means for displaying a virtual object determined to be displayed by the determination means,
A possession object identification means for identifying the virtual object held in the user's hand, and
With
When the object of interest of the user specified by the specific means becomes another object, the determination means obtains the virtual object held in the user's hand specified by the possessed object identification means. , Decided to make it a virtual object corresponding to the other object,
The information processing system is characterized in that the display means displays a virtual object corresponding to the other object determined by the determination means. A control method for an information processing system including a head-mounted display device and an information processing device.
A specific step of identifying an object of interest that the user is paying attention to from the line of sight of the user wearing the head-mounted display device.
A determination step of determining a virtual object corresponding to the object of interest as a virtual object to be displayed on the head-mounted display device, separately from the object of interest specified by the specific step.
A display process for displaying a virtual object determined to be displayed by the determination process, and a display process for displaying the virtual object.
A possession object identification process for identifying the virtual object held in the user's hand, and
Including
In the determination step, when the object of interest of the user specified by the specific step becomes another object, the virtual object held in the user's hand specified by the possessed object identification step is used. , Decided to make it a virtual object corresponding to the other object,
The display step is a control method characterized in that a virtual object corresponding to the other object determined by the determination step is displayed. A program for controlling an information processing system including a head-mounted display device and an information processing device.
The information processing system
A specific means for identifying an object of interest that the user is paying attention to from the line of sight of the user wearing the head-mounted display device, and
A determination means for determining a virtual object corresponding to the object of interest as a virtual object to be displayed on the head-mounted display device, in addition to the object of interest specified by the specific means.
A display means for displaying a virtual object determined to be displayed by the determination means,
To function as a possession object specifying means for specifying the virtual object being held in the hand of the user,
When the object of interest of the user specified by the specific means becomes another object, the determination means obtains the virtual object held in the user's hand specified by the possessed object identification means. , Decided to make it a virtual object corresponding to the other object,
The display means is a program characterized by displaying a virtual object corresponding to the other object determined by the determination means.

Images