JP7024010B2 - Sending terminal, receiving terminal, sending / receiving system, and its program - Google Patents

Description

The present invention relates to a transmitting terminal, a receiving terminal, a transmitting / receiving system, and a program thereof, and relates to an augmented reality or mixed reality technique for displaying a virtual object in relation to position coordinates in real space.

In recent years, when the same real space is viewed from different viewpoints by two or more information processing terminals, the virtual objects are different on the two or more information processing terminals as if they existed in the real space. A technique for displaying in orientation (angle of view) has been proposed. In order to realize this technology, so-called geotags are required.

Geotags are pre-assigned latitude, longitude (and in some cases altitude) and direction information to virtual objects. If the position and orientation of the virtual object on the display is determined based on the geotag information when the real space including the latitude and longitude is viewed through the information processing terminal, the virtual object can be displayed as if it exists in the real space. It becomes possible to display. Also, if common geotag information can be used for two or more information processing terminals, does the virtual object exist in the real space even when each information processing terminal looks at the real space from different viewpoints? It can be displayed in different orientations (angles).

By the way, since this geotag is given in advance to a virtual object in relation to the position coordinates in the real space, we want to display it in common to two or more information processing terminals "on the spot". It cannot be used for virtual objects.

On the other hand, for example, Patent Document 1 can notify another image processing device of the position data of a virtual object to be displayed on one image processing device and display the virtual object on the other image processing device without any inconsistency. Disclose the technology to do so. Specifically, it discloses a technique for notifying information on the position and direction of a virtual object from a terminal that originally displayed the virtual object to another terminal. In other words, geotag information is added to the data to which geotag information has not been attached until then at any point during virtual display on the original terminal, and when that data is shared, the geotag information is shared. Notify another terminal of.

Japanese Unexamined Patent Publication No. 2011-175439

However, in the method disclosed in Patent Document 1, it is necessary that each information processing terminal can acquire position and direction information with high accuracy. However, it is said that the position information by GPS (Global Positioning System), which is generally used at present, has an error of several meters, and there are cases where the position cannot be acquired due to radio wave interruption indoors or underground.

An object of the present invention is to provide an information processing device that creates a visual sensation when a user pinches a virtual object with a finger. It also aims to provide the method and program.

The information processing apparatus according to the present invention is
It is a display control means for controlling the display of a virtual object based on three-dimensional shape data on the display unit, and the back side portion of the first finger of the user who operates the virtual object is in front of the virtual object. A display control means for displaying the virtual object on the display unit so that the ventral portion of the user's second finger is behind the virtual object, and a display control means.
When the operation of shifting the first finger and the second finger in opposite directions while the virtual object is pinched by the first finger and the second finger is detected, the virtual object corresponds in advance. It has a reception means that accepts instructions to select the attached menu.
It is characterized by that.

The information processing method according to the present invention
In the display control process for controlling the display of a virtual object based on three-dimensional shape data on the display unit, the back side portion of the first finger of the user who operates the virtual object is in front of the virtual object. A display control step for displaying the virtual object on the display unit so that the ventral portion of the user's second finger is behind the virtual object, and a display control step.
When the operation of shifting the first finger and the second finger in opposite directions while the virtual object is pinched by the first finger and the second finger is detected, the virtual object corresponds in advance. It is characterized by having a reception process for receiving an instruction to select an attached menu .

The program according to the present invention
Computer
It is a display control means for controlling the display of a virtual object based on three-dimensional shape data on the display unit, and the back side portion of the first finger of the user who operates the virtual object is in front of the virtual object. A display control means for displaying the virtual object on the display unit so that the ventral portion of the user's second finger is behind the virtual object, and a display control means.
When the operation of shifting the first finger and the second finger in opposite directions while the virtual object is pinched by the first finger and the second finger is detected, the virtual object corresponds in advance. It is characterized in that it functions as a reception means for receiving an instruction to select an attached menu .

According to the present invention, it is possible to create a visual sensation when a user pinches a virtual object with a finger .

It is a figure which shows the typical embodiment of the information processing terminal of this invention used in the information processing system of this invention. It is a flowchart which shows the flow of judgment and processing of the 1st information processing terminal which constitutes the information processing system of this embodiment. It is a flowchart which shows the flow of judgment and processing of the 2nd information processing terminal which constitutes the information processing system which concerns on this embodiment. An example of the display transition in the display unit of the first information processing terminal is shown, and it is a transition diagram which shows four display states (401) to (404) of the first half. An example of the display transition in the display unit of the first information processing terminal is shown, and it is a transition diagram which shows four display states (405) to (408) of the latter half. It is a state diagram which shows the usage state of the information processing terminal of this embodiment schematically by a series of procedures, and shows the setting stage (501) and the start stage (502). It is a phase diagram which shows the usage state of the information processing terminal of this embodiment schematically by a series of procedures, and shows the stage of a sharing instruction (503) and the projection stage (504) which projects an index. It is a state diagram which shows typically the use state of the information processing terminal of this embodiment, and shows the stage (505) which became shareable. It is a block diagram which shows the structure of the electronic circuit built in the main body of the information processing terminal of this embodiment.

The transmission / reception terminal (information processing terminal) of the transmission / reception system according to the present embodiment has both a function as a transmission terminal and a function as a reception terminal, but for convenience of explanation, the transmission terminal and the reception terminal will be described separately.

The transmitting terminal is attached to the head or face of the first user and has a projection means, a first image pickup means, a first display means, and a transmission means.
The projection means projects the index image onto the real object based on the index image source data.
The first imaging means acquires the index image projected on the real object by the projection means to obtain the first index image imaging data.
The real object is displayed in the first display means, and an image of the virtual object viewed from a first viewpoint in the virtual space is displayed in the vicinity of the real object.
The transmission means transmits 1) three-dimensional shape data for displaying a virtual object, 2) the index image source data, and 3) the index image imaging data.
The position and orientation of the projection means, the position and orientation of the first image pickup means, and the position and orientation of the first viewpoint sequentially follow the changes in the position and orientation of the head or face of the first user.

The receiving terminal is attached to the head or face of the second user and has a receiving means, a second imaging means, a second display means, and a calculation means.
The receiving means receives 1) three-dimensional shape data for displaying a virtual object, 2) the index image source data, and 3) the first index image imaging data transmitted from the transmitting terminal.
The second imaging means acquires the index image projected on the real object to obtain the second index image imaging data.
The real object is displayed in the second display means, and an image of the virtual object viewed from a second viewpoint in the virtual space is displayed in the vicinity of the real object.
The calculation means is based on the index image source data, the first index image imaging data, and the second index image imaging data.
The relationship between the position and orientation of the index image and the position and orientation of the second image pickup means in the real space.
The position and orientation of the second viewpoint are determined so that the relationship between the position and orientation of the virtual object and the position and orientation of the second viewpoint in the virtual space is substantially the same.
Then, the position / posture of the second image pickup means and the position / posture of the second viewpoint sequentially follow the change in the position / posture of the head or face of the second user.

Hereinafter, preferred embodiments (examples) of the present invention will be described with reference to the drawings.
The information processing terminal (transmission / reception terminal) according to the present embodiment will be described with reference to FIGS. 1, 2, 3, 6, and the like.

<Appearance configuration of information processing terminal (transmission / reception terminal)>
The information processing terminal (transmission / reception terminal) is configured with a computer having an information processing function as a basic element. In the following description, an example of an HMD (Head Mounted Display) mounted on a human head as an information processing terminal will be described. In the following description, for convenience of explanation, in principle, "HMD" is used instead of "information processing terminal". Each of the two HMDs includes a mounting means to be worn on the user's head, a projection means, an imaging means, a display means, and an information processing means.

FIGS. 1, (A), (B), and (C) show an external view of the HMD 100. 1 (A) is an external perspective view seen from the front side, FIG. 1 (B) is an external perspective view seen from the rear side, and FIG. 1 (C) is a side view of a state of being attached to the user's head. be. Generally, the HMD has a so-called goggle type in which the eyepiece is in close contact with and fixed around the user's eye by a stretchable head-mounted member, or the entire head of the user is covered. There are various types such as the so-called helmet type, which is fixed at the chin. Further, in recent years, in consideration of wearability, a relatively lightweight spectacle frame type that hangs a frame on the user's ear and fixes it has also appeared.
The HMD100 shown in FIGS. 1A to 1C corresponds to the so-called goggles type described above.

In the HMD 100, the head mounting portion 101 is for mounting the HMD 100 on the user's head, and as shown in FIG. 1, is composed of a plurality of constituent members numbered 121 to 125. As shown in FIG. 1C, in order to attach the HMD 100 to the user's head H, first, the length adjusting portion 123 is loosened by the adjuster 122 and then put on the head H. Then, after the forehead mounting portion 125 is brought into close contact with the forehead of the head H, the length adjusting portion 123 is narrowed down by the adjuster 122 so that the temporal mounting portion 121 and the occipital mounting portion 124 are brought into close contact with the temporal region and the occipital region, respectively. ..

The image pickup unit 102 is a so-called digital camera, and is composed of a plurality of members numbered 112, 116, and 117 as shown in FIG. 1 (C). Images are taken in substantially the same direction, and the change in the position and posture of the user's head is sequentially followed. Specifically, the light incident from the outside of the HMD 100 through the incident window 112 is guided to the inside of the HMD 100 by the optical prism 117, and is received by the image pickup sensor 116. The members 112, 116, and 117 have elements corresponding to the left eye and the right eye, respectively, and are distinguished by adding the symbols "L" and "R" in the drawings, respectively.

The projection unit 103 is a so-called projector, and includes a projection lens, an image display element, a light emitting element, and the like.
As shown in FIGS. 1B and 1C, the display unit 104 is composed of a plurality of members numbered 110, 114, and 115. The symbols "L" and "R" indicate the left eye and the right eye as described above. It is provided at a position corresponding to a spectacle lens in spectacles so as to be opposed to the position of the user's eye. Specifically, an image displayed by a color liquid crystal display 114 is guided by an optical prism 115, and a screen 110 is provided. Display on.

Here, there are a glass see-through method and a video see-through method as a method of superimposing and displaying graphics on a scene in the real world. The glass see-through method projects graphics using a transparent or translucent spectacle lens as a screen, and optically superimposes the graphics on a real-world scene. The video see-through method electrically superimposes graphics on a real-world scene captured by an image pickup device.

When the HMD 100 is the above-mentioned video see-through type, the screen 110 has no transparency in principle and displays the guided image as it is.
On the other hand, when the HMD 100 is the above-mentioned glass see-through type, the screen 110 is composed of a so-called half mirror. That is, it has a certain transmittance and allows the user to optically see the scene in the real space. At the same time, a mirror provided embedded in the front surface, the back surface, or the inside reflects the graphic image displayed by the color liquid crystal display 114 and guided by the optical prism 115 toward the user's eye. That is, in the case of the glass see-through type, the scene in the real space and the image of the virtual object (virtual object) are optically superimposed on the screen 110.

The output light from the optical prism 115 of the display unit 104 and the input light to the optical prism 117 of the image pickup unit 102 are aligned with the optical axis of the user's pupil, and the image pickup sensor 116 is the image sensor 116 of the user's face. An image of the real space seen by the position and direction (positional posture) is taken.
In the case of the video see-through type HMD 100, the color liquid crystal display 114 displays an augmented reality image in which a real space image acquired by the image pickup sensor 116 and an image of a virtual object are electrically superimposed (image synthesis).
On the other hand, in the case of the glass see-through type HMD 100, the color liquid crystal display 114 displays only the image of the virtual object in principle.

<Structure of the inside (electronic circuit) of the information processing terminal>
As shown in FIG. 6, the HMD 100 further includes a display control unit 130, an image pickup control unit 140, a CPU 150, a memory 160, a power supply unit 170, a communication unit 180, and a projection control unit 190 as electronic circuit elements. These electronic circuit elements are embedded inside the main body of the HMD 100 (including the head mounting portion 101) and are not shown as an appearance.
In the above, the CPU 150, the memory 160, the display control unit 130, the image pickup control unit 140, and the projection control unit 190 constitute the information processing means of the HMD 100.

The display control unit 130 controls the display of the display unit 104. For example, it controls the size, position, orientation, color, transparency, movement and change of brightness of a virtual object to be superimposed and displayed (image composition) on a real space image.

The image pickup control unit 140 performs exposure control, distance measurement control, and the like based on the calculation result of a predetermined calculation process using the image pickup data. As a result, AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and the like are performed. When the image pickup unit 102 is provided with a mechanism for inserting / removing the optical filter from the optical path, an anti-vibration mechanism, or the like, the image pickup unit 102 controls the insertion / removal of the filter, the anti-vibration, and the like according to the imaging data and other conditions.

The CPU 150 performs arithmetic processing for the entire HMD 100. By executing the program recorded in the memory 160 described later, each process of the present embodiment is realized.
The memory 160 includes a work area and a non-volatile area. Programs, system control constants, and variables read from the non-volatile area are expanded in the work area. In addition, the image data of the virtual object to be superimposed and displayed in the real space is retained for display. Further, the image pickup data captured by the image pickup unit 102 and A / D converted is developed for the purpose of image analysis, image processing, or the like.

The power supply unit 170 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, and the like, and supplies power to the entire HMD 100. It is equipped with a power switch that switches between power on and power off depending on user operation and other conditions.
Based on the control of the CPU 150, the communication unit 180 communicates with another HMD 100 in accordance with a standard such as WiFi Direct, and transmits / receives three-dimensional shape data, position / orientation data, and the like.
The projection control unit 190 controls the projection operation of the projection unit 103.

<Flow of judgment / processing of information processing terminal>
FIG. 2 shows the flow of judgment and processing executed by the first HMD 100 having the above configuration, and FIG. 3 shows the flow of judgment and processing executed by the second HMD 100 (reception terminal) having the above configuration. There is. It should be noted that this determination and processing is realized by the CPU 150 expanding the program recorded in the non-volatile area of the memory 160 into the work area and executing it.

<Flow of judgment / processing of the first information processing terminal>
First, with reference to FIG. 2, the flow of determination / processing of the first HMD100 (transmission terminal) relating to the use of the user (user) instructing data sharing will be described.
In step S201, the first HMD 100 starts imaging in the real space by the imaging unit 102 when the power is turned on by the user who uses the first HMD 100.

In step S202, the image of the virtual object is displayed on the display unit 104 under a predetermined display condition (size, position, orientation, color, transparency, etc.) by superimposing the image on the image in the real space being captured. The virtual graphics displayed immediately after the power is turned on are, for example, graphics (three-dimensional icons) that symbolically represent the storage area of the non-volatile area related to the management of the user of the HMD 100, depending on the specifications of the information processing terminal. May be good. Further, the graphics displayed before the power is turned off may be restored and used.

In step S203, the first HMD 100 determines whether any of the virtual graphics displayed by the user has been instructed / operated. In this case, the virtual graphics superimposed on the image in the real space and displayed correspond to a so-called graphical user interface (hereinafter referred to as "GUI").
For example, when a three-dimensional icon (storage icon) symbolically representing a storage area related to user management is displayed and an "open" instruction is given by a user's gesture operation or the like, it is stored in that storage area. Displays graphics that represent existing files, etc. Further, for example, when an icon symbolically representing the design data of "shoes" is displayed and "open" is instructed by a user's gesture operation or the like, the three-dimensional model of the "shoes" is displayed in the real space. It is superimposed on the image of.

By the way, when the virtual graphics are superimposed and displayed on the image in the real space, the so-called augmented reality display in which this is displayed in relation to the position coordinates in the real space is not related to the position coordinates in the real space. There is a case of displaying with (when not performing augmented reality display). When to display augmented reality and when not to display augmented reality is based on predetermined conditions. The setting of the condition for performing or not performing this augmented reality display will be described later.

When the user gives an instruction / operation to the virtual graphics (YES in step S203), it is determined whether the instruction / operation is an instruction / operation to "share" the data indicated by the graphics (YES). Step S204). If the instruction / operation is other than "sharing", the content of the instruction is executed (step S205).

In step S209, when the user's instruction / operation is the instruction / operation of this "sharing", first, the projection unit 103 omits the display position displaying the virtual graphics instructed to share. The index image is projected in the same direction in the real space. However, at this time, as shown in the following steps S206 to S209, the index image is so-called so-called so that the index image is similar to the one projected in the vertical or horizontal direction based on the projection direction angle detected by the angle sensor. Keystone correction.

For example, when the HMD 100 projects the index image toward a substantially vertical surface such as a wall surface at an angle of θ with respect to the horizontal direction, the direction along the projection direction of the index image (from the HMD 100). The so-called vertical direction when viewed) and the upper side with respect to the lower side when viewed from the HMD 100 are each deformed by 1 / cos θ times. Therefore, if the original image is corrected by multiplying it by cos θ, it will be the same as the one projected horizontally regardless of the angle of the information processing terminal.

When the HMD 100 projects the index image toward a substantially horizontal surface such as a table surface at an angle of θ with respect to the horizontal direction, the direction along the projection direction of the index image (HMD 100). The so-called vertical direction when viewed from above and the upper side with respect to the lower side when viewed from HMD100 are each deformed by -1 / sin θ times. Therefore, if each of the original images is corrected by multiplying by -sin θ, it will be the same as the one projected vertically regardless of the angle of the HMD 100.

For example, when the posture (angle θ with respect to the horizontal direction) of the HMD 100 is −45 ° ≦ θ ≦ 45 °, it can be estimated that the projection is directed toward a substantially vertical plane.
Further, when the posture (angle θ with respect to the horizontal direction) of the HMD 100 is θ ≦ −45 ° or θ ≧ 45 °, it can be estimated that the projection is directed toward a substantially horizontal surface.

That is, in step S206, the projection direction angle is acquired by the angle sensor, and in step S207, it is estimated whether to project toward a substantially vertical surface or a substantially horizontal surface.
In step S208, the index image is so-called keystone-corrected according to whether the projection plane is a substantially vertical plane or a substantially horizontal plane.
In step S209, the corrected index image is projected onto the projection surface by the projection unit 103.

Next, in step S210, the projected index image is imaged by the image pickup unit 102. Since the original image of this index image has a trapezoidal correction similar to that projected vertically or horizontally by the process described above, the tilt of the captured index image is projected with the imaging direction of the HMD 100. It reflects the inclination between the surface and the surface. Further, the size of the captured index image reflects the distance between the HMD 100 and the projection surface by correcting the spread angle of the projected image and the magnification of the image pickup lens (so-called zoom ratio). Become.

Next, in step S211 the position, orientation (positional posture), and size of the virtual graphics instructed to be shared are changed based on the captured image of the index image and displayed in an superimposed manner. In this way, the virtual object can be displayed in augmented reality as if it were placed on or adhered to a real object projected by an index.

In the above, the projection direction of the index image by the projection unit 103 and the image pickup direction of the index image by the image pickup unit 102 in the first HMD 100 coincide with the viewpoint (position and posture) of the first user. In the first HMD 100, the inclination of the projection surface on which the index image is projected is specified based on the index image source data used for projection and the index image imaging data obtained by imaging the projected index image. .. Then, this inclination or the like is reflected in the position and orientation of the virtual object for which "sharing" is instructed, and is superimposed and displayed in the vicinity of the projection surface to perform augmented reality display.

The projection of the index image may be performed by a light ray visible to the human eye or by an invisible light ray (for example, infrared ray) as long as the image pickup unit 102 of the HMD 100 can image the image. Alternatively, instead of continuous projection, for example, discontinuous projection may be performed at regular time intervals.
As a result, for example, a pattern is drawn on an actual object (projection surface), and it is difficult to distinguish between the projected index image and the pattern on the projection surface by continuous projection with visible light, so that it is used as an index. Even when recognition is difficult, it is possible to facilitate recognition of the index image. By projecting and recognizing with infrared light to remove the influence of patterns, etc., or by projecting discontinuously at regular time intervals and extracting the projected image from the difference between the captured images before and after projection, the index image can be displayed. It becomes possible to facilitate recognition.

Next, in step S212, information related to the above virtual object (three-dimensional shape data, position / orientation data, etc.) and information related to the index image (index image source data, index image imaging) are transmitted from the communication unit 180 of the first HMD 100. Data) and send.
Further, in step S213, a display indicating that "sharing is being set" is displayed on the display unit 104. This display may display characters such as "sharing is being set", or the graphics of the virtual object may be displayed in a different color or transparency, or may be displayed by blinking. It may be a thing.

Then, it waits until it receives a confirmation signal transmitted from another HMD100 (second HMD100) (step S214). The confirmation signal will be described later.
However, a predetermined end condition is provided, and when the condition is satisfied (for example, when a predetermined time has elapsed), the standby is terminated (step S215). When the confirmation signal is received, the display of "sharing is being set" is erased (step S216).

<Flow of judgment / processing of the second information processing terminal>
The flow of determination / processing of the second HMD100 (reception terminal) will be described with reference to FIG.
In step S301, the second HMD 100 starts imaging in the real space by the imaging unit 102 when the power is turned on by the second user who uses the second HMD 100.
Then, in step S302, the second HMD 100 is transmitted from the other first HMD 100, the information related to the shared virtual object (three-dimensional shape data, the position / orientation data) and the information related to the index image (index). Wait until the image source data and index image imaging data) are received.
However, a predetermined end condition is provided, and when the condition is satisfied (for example, when a predetermined operation by the user is performed), the standby is terminated (step S303).

When the second HMD 100 receives the information related to the virtual object and the information related to the index image instructed to be shared in step S304, this "information" is displayed on the display unit (second display unit) of the second HMD 100. Was received "is displayed to the second user. At this time, if the position of the first HMD 100 can be acquired, a display prompting the user to face the direction may be displayed.

Then, in step S305, it is determined whether or not there is an index image projected by the projection unit of the first HMD in the image captured by the image pickup unit (second image pickup unit) of the second HMD.
In step S306, when there is an index image, first, a "confirmation signal" to that effect is transmitted to the first HMD 100. This means that the first HYMD100 (and its user) is notified that the virtual object has been shared. The image pickup direction by the image pickup unit 102 of the second HMD 100 coincides with the viewpoint and line of sight of the user who uses the image pickup unit 102.

Further, in step S307, the position, size, and orientation (direction and inclination) of the index image in the captured image are acquired.
For example, if the pattern of the index image is neither point-symmetrical nor line-symmetrical, a real object (projection surface) on which the index image is projected by comparing the received index image source data with the first index image imaging data. ) And the position and orientation of the first image pickup means with respect to the projection surface can be specified.
Further, by comparing the received index image source data with the second index image imaging data, the position and orientation of the second imaging means with respect to the projection surface can be specified.
Then, if the relationship between the position and orientation of the first imaging means, the position and orientation of the projection surface, and the position and orientation of the second imaging means is specified,
Identify the relationship between the position and orientation of the first viewpoint in the virtual space, the position and orientation of the virtual object displayed superimposed on or near the projection surface, and the position and orientation of the second viewpoint in the virtual space. be able to.

Then, in step S308, rendering is performed based on the information (shape data, position / posture data) related to the virtual object instructed to be shared.
In step S309, the image of virtual information (image of virtual graphics) obtained by rendering is superimposed on the image in the real space captured by the image pickup unit 102 and displayed on the display unit 104.
In this way, the position and orientation are specified, and a virtual object displayed superimposed on or near the projection surface (for example, a virtual object placed about 1 cm above the projection surface and the right side surface can be seen from the first viewpoint). The image viewed from the second viewpoint in the virtual space is displayed on the second display unit 104.
For example, in the case of the positional relationship as shown in FIG. 5C, a virtual object (shoe) arranged about 1 cm above the projection surface and viewed from the “right” side surface is displayed on the display unit of the user P3, and the user P1 is displayed. A virtual object is displayed on the display unit, which is arranged about 1 cm above the projection surface and is viewed from the "left" side surface.

By the above steps S307, S308, and S309, sharing of a virtual object (virtual object) is executed in the second HMD 100 via the index image projected by the first HMD 100. The steps S307, S308, and S309 constitute step S311 for the sharing process, and realize the sharing means. As a result, the predetermined virtual object (step S211) displayed on the first HMD 100 is displayed on the second HMD 100 based on the viewpoint, and sharing is realized.

After superimposing and displaying the image of the virtual information related to the virtual object instructed to be shared, the second HMD 100 displays the image of the virtual information in augmented reality as if it actually exists at a specific position in the real space. (Step S310). That is, when it is recognized that, for example, the second HMD100 has approached the specific position by recognizing the feature points of the captured image captured in the real space, the captured image is re-rendered so that the graphic of the object becomes larger. Superimpose on.
Further, if the above-mentioned specific position is out of the user's field of view (that is, the captured image), the graphic of the virtual object is once erased. However, when the specific position comes into the user's field of view again, the graphic of the virtual object is redisplayed as if it actually exists at that position based on the information of the feature points stored in the memory 160.

<Action / effect of the embodiment>
Summarizing the above, the first HMD 100 is in the direction of displaying the augmented reality when the user (user) instructs the virtual object to be "shared". The index image is projected toward. The second HMD 100 captures the projected index image, and displays the virtual object in augmented reality based on the index image, the transmitted information related to the virtual object instructed to share, and the information related to the index image. do.
The above index image in the captured image of the second HMD 100 is, of course, viewed from the viewpoint of the second HMD 100, and also reflects the inclination of the projected surface of a real object (for example, "table"). It is a thing.

That is, the second HMD 100 generates graphics of a virtual object having a position and orientation corresponding to the position, size, and orientation (direction and inclination) of the index image in the captured image, and displays the augmented reality on the display unit. As a result, the first HMD 100 and the second HMD 100 can be displayed from their respective viewpoints on the respective display units 104 as if the same object is at the same position in the real space. ..
Therefore, for example, when the first user points to the virtual object with a real object such as a finger, the virtual object and the real object are superimposed and displayed on the second display unit, and the second user receives the first. The user appears to be pointing at the virtual object. Therefore, an augmented reality display suitable for a first user and a second user to "share" a virtual object becomes possible.

At this time, the position and orientation of the first viewpoint when drawing the virtual object displayed on the first display unit and the position of the second viewpoint when drawing the virtual object displayed on the second display unit. The relationship with the viewpoint is substantially the same as the relationship between the position and orientation of the first image pickup unit and the position and orientation of the second image pickup unit.
Although it is possible to make the same appearance by using a so-called geotag, the information processing terminal of the present invention (for example, the HMD100 described in the above embodiment) imparts latitude / longitude information to the virtual object in advance. There is no need. In addition, it is not necessary to acquire the latitude / longitude information of each terminal.
In the system using the information processing terminal of the present invention, there is a real object capable of projecting an index image in the real space where the first user and the second user exist, and the object is projected from the first terminal onto the projection surface. It suffices if each of the first terminal and the second terminal can capture the index image.
By capturing a common index image from the respective positions and orientations of the first terminal and the second terminal, the position and orientation of the first image pickup unit in the real space, the position and orientation of the projection surface, and the second The relative relationship (first relative relationship) with the position and orientation of the image pickup unit can be specified.
Then, the relative relationship (second relative relationship) between the position / posture of the viewpoint of the first terminal in the virtual space, the position / posture of the virtual object, and the position / posture of the viewpoint of the second terminal is the first. The position and orientation of the viewpoint of the second terminal are specified so as to be substantially the same as the relative relationship.
As a result, the virtual object superimposed and displayed in the vicinity of the projection surface is displayed on the first display means as an image viewed from the first viewpoint, and the image of the same virtual object viewed from the second viewpoint is displayed on the second display. It has the effect of the invention in that it can be displayed by means and a shared augmented reality display is possible.

<Display transition in the display unit of the first information processing terminal>
Next, with reference to FIGS. 4A and 4B, an operation example of the first HMD 100 that projects an index image when the user is instructed to share an image of virtual information will be described. In FIGS. 4A and 4B, the display transition in the display unit 104 of the first HMD 100 is shown by a series of display states (401) to (408). The first four display states (401) to (404) are shown in FIG. 4A, and the latter four display states (405) to (408) are shown in FIG. 4B.

The display state (401) shows an example of the display immediately after the power is turned on in the first HMD 100. In the display state (401), reference numeral 4011 indicates an image captured in the real space captured by the image pickup unit 102. Further, reference numeral 4012 is virtual graphics superimposed on the captured image 4011, and here, a storage icon representing a storage area related to the management of the user of the first HMD 100 is displayed.

The display state (402) represents a state in which the user is trying to instruct / operate the storage icon 4012 by the gesture of “pinching with a finger”. The user's hand 4023 is a part of the captured image in the real space captured by the imaging unit 102. Here, with respect to the virtual graphics 4022, the back side (the side with the nail) 4024 of the user's thumb is displayed in the front, and the ventral side (the side without the nail) 4025 of the user's index finger is displayed in the back. Display control is done.
This is done to give the user a visual sensation of pinching the graphics with their fingers, and is an image that identifies and extracts the back and ventral parts of a person's fingers from the captured image. It is based on processing technology. Such an image processing technique is already known, for example, as disclosed in Japanese Patent Application Laid-Open No. 2012-53674.

The display state (403) represents a mode in which the storage icon 4031 is instructed to "display the contents of the storage (so-called" open ")" by a predetermined gesture operation of the user. Here, the predetermined gesture operation is a gesture of "pinching the icon with a finger and then stumbling with a thumb and an index finger". Since this gesture is common to the action of opening a bundle of cards in a fan shape, it is easy to recall the word "open", and it is suitable as a so-called "open" gesture.
Of course, there is no problem even if another gesture is assigned as the so-called "open" gesture.

The display state (404) is a lower level hierarchy of the storage area represented by the storage icon 4031 as a result of the user's instruction to "open" the storage icon 4031 in the display state (403). It represents the displayed mode. In this display state (404), the icons 4041 to 4048 are icons for displaying data or files stored in the storage unit of the first HMD 100, respectively. For example, the icon 4048 displays one file of application software that displays a three-dimensional model on the display unit 104.

The display state (405) shown in FIG. 4B represents a mode in which the file icon 4052 is instructed to "open the file (by starting the application software)" by a predetermined gesture operation of the user. Here, the predetermined gesture operation is the same gesture as that performed in the display state (403), "after pinching the icon with a finger, the thumb and the index finger are attached to the gesture".

In the display state (406), the application software for displaying the three-dimensional model is started, the file indicated by the icon 4052 is expanded in the memory 160, and the object 4062 of the "shoes" is displayed superimposed on the captured image 4061 in the real space. It represents the state of being.

The display state (407) represents a state in which the user makes a gesture to the three-dimensional model 4062 in the display state (406), "Pinch the object with a finger and then squeeze the object with the thumb and index finger." ing. When this gesture is made, a menu of commands that the user can instruct the object in response to this gesture is displayed. The commands include "COPY", "CUT", "EDIT", "SHARE" and the like. After the menu is displayed, the user can further move his thumb to instruct and select one command.

The display state (408) is an augmented reality display in which this object is associated with position coordinates in real space as a result of the user selecting the command "SHARE" and instructing "sharing" of the object. Represents. That is, in the figure, the captured image 4081 is a captured image in the real world, and the virtual object 4082 is a virtual object displayed in augmented reality.
For example, by displaying a shadow under the virtual object, the virtual object "shoes" are displayed in augmented reality as if they were placed on the "table" of the real object. It is difficult to distinguish between virtual and real.

When transitioning from the display state (407) to the display state (408), the display of the virtual object was changed from the so-called normal superimposed display to the augmented reality display. The display mode has been changed in this way so that objects showing shared files are displayed in augmented reality, and objects showing files that are not shared are not displayed in augmented reality, so-called normal superimposed display. This is because the display conditions were set. In this way, the display mode related to augmented reality and the display mode related to non-augmented reality differ depending on whether the file is shared or non-shared for the following reasons.

That is, when multiple users (users) gather in a specific place to hold a meeting or collaboration, if the augmented reality is displayed as if the object is actually there, the effect of the meeting or collaboration is achieved. And efficiency is improved. For example, when one user gives a presentation while pointing to a virtual object, other users can see the virtual object and the presenter pointing to it at the same time.

On the other hand, if the display is an object related to a "non-shared" file that is not shared with other users, the display of the object is not related to the position coordinates in the real space if it is a non-augmented reality display. .. Therefore, for example, if the user does not face the direction of the position coordinates, the object will not be displayed.
That is, it is rational to change the display mode so that the object related to the "non-shared" file is not displayed in augmented reality and the object related to the "shared" file is displayed in augmented reality.

<A series of procedures based on the usage status of the first and second information processing terminals>
Next, with reference to FIGS. 5A, 5B, and 5C, an example of a series of procedures for "sharing" virtual objects using the first and second HMD100 described above will be described. In this example, a plurality of users wearing the HMD 100 on their heads gather at a specific place to hold a meeting or the like. In such a case, one user is instructed to "share" the virtual object, and another user displays the virtual object in augmented reality in relation to the common position coordinates from each viewpoint. It is a schematic representation.

Of the above series of procedures, FIG. 5A shows a setting stage (501) and a starting stage (502). FIG. 5B is a display mode viewed from the first HMD 100 and shows a stage of sharing instruction (503) and a projection stage (504) of projecting an index from the first HMD 100. FIG. 5C shows a state (505) in which a virtual object can be shared with a second HMD 100 based on an index projected from the first HMD 100.

In the setting stage (501), the users P1 and P2 wearing the above-mentioned HMD100 set the user group by a predetermined method, respectively. Here, the gesture that the users P1 and P2 wearing the HMD100 shake hands is the method of group setting. Using this gesture as a group setting method has the following meanings.

First, "handshake" is natural as a pre-meeting action. Second, compared to other gestures (eg, "bowing"), it is more reliable to identify whether or not a gesture has been made, so it is possible to prevent inadvertent sharing of objects. Third, by grouping based on the fact that the handshake was actually shaken (not only the appearance of the handshake, but also the actual physical contact), the unintended person was grouped. It should be possible to set a highly secure group without fear of adding it.
In order to make the third significance feasible, it is necessary for each HMD 100 to be provided with a means for detecting physical contact between the wearers of the HMD 100, for example, by changing the electric field state. The technique for detecting body contact by changing the electric field state is a known technique as disclosed in, for example, Japanese Patent Application Laid-Open No. 2010-193129.

The start stage (502) shows that three HMD100 users P1, P2, and P3 grouped together surround the table 5021 of a real object. In this state, since there is no virtual object shared by the three people, the three HMD100s display different virtual objects.

The stage (503) is a mode of display as seen from the user (wearer) P3 of one HMD100. Here, the virtual object 5031 is displayed only for this user P3. In the stage (503), a gesture operation for instructing "sharing" of the virtual object is performed by the finger 5032 of the user P3.

The projection stage (504) shows that the index image 5041 is projected onto the table 5021 from the HMD 100 equipped on the user P3.

In the sharing stage (505), the second HMD 100s of the other users P1 and P2 each recognize the index image 5041, and based on this, each display unit displays an image from each viewpoint of the virtual object 5051 related to this sharing. Is schematically shown in the augmented reality display. That is, the virtual object 5051 of the shoe in the sharing stage (505) is not displayed (invisible) except for the users (wearers) P1 to P3 of the HMD100.
However, each display means of the users (wearers) P1 to P3 of the HMD 100 displays the virtual object 5051 as if it were actually placed on the table 5021 from the viewpoint of each user. This is schematically shown.

For users P1 to P3 belonging to the group, this virtual object 5051 is displayed in common as if it actually exists, which is convenient for meetings and collaborations within the group. For example, in order to explain the design of "shoes", one of the users picks up the object of "shoes" in his hand, changes the direction in various ways, points with his finger, and adds an explanation such as "What is the shape here ...". , The object is displayed to other users with the pointed finger.
Therefore, the content of the explanation becomes easy to understand. Alternatively, if another user gives an opinion while pointing with a finger or the like, such as "This is the way to go ...", it is clear which part it is. Further, when the user of the group can edit / modify the virtual object 5051, for example, it is possible to proceed with the meeting while changing the color and the pattern.

It should be noted that how the group user can operate the object set as "shared" can be the same as the general concept of "shared" in the HMD100. That is, it is based on the setting of "authority" of the group user when the group is set. For example, permissions such as "view only" and "modification / overwriting possible" are preset for group users. Further, regarding when to cancel the group, in addition to the case where the cancellation operation is performed, for example, when the HMDs 100 are separated from each other by a predetermined distance or more can be set as the condition for canceling the group.

Summarizing the above, the information processing terminal (first HMD100) related to the use of the user who instructs the sharing of the virtual information image projects the index image when the sharing instruction is given. As described above, since the other information processing terminal (second HMD100) displays the image of the virtual information related to the sharing instruction based on the projected index image, other information is not available until the sharing instruction is given. The processing terminal can display the image of the virtual information. That is, when there is no sharing instruction, the image of the virtual information is not displayed on other information processing terminals, so that the information can be managed with good security.

<Other embodiments>
In this embodiment, an example of the HMD 100 that transmits a predetermined confirmation signal when an index image related to an instruction to share an image of virtual information is captured will be described.
As described in step S306 of FIG. 3, the HMD 100 according to the present embodiment transmits an "confirmation signal" to that effect to the HMD 100 projecting and displaying the index image when the index image is captured.

The index image projection means for projecting the index image is a part of the HMD 100 according to the present embodiment, and is mounted on the user's head by the head mounting means. Therefore, the user of the HMD 100 who projects the index image moves the head greatly until the other HMD 100 "confirms" the index image, that is, until the index image is captured by the image pickup means of the other HMD 100. Must not be. However, after the index image is "confirmed" by another HMD 100, it is rational to be notified promptly to that effect so that the user can recognize that he / she may no longer move his / her head.

When the HMD 100 according to the present embodiment captures an index image related to an instruction to share an image of virtual information, the reason why a predetermined confirmation signal is transmitted is to perform this notification, and this confirmation signal is used as follows. can do.

For example, the HMD 100 that projects the index image may be displayed so as not to move the head to the user until the "confirmation signal" is received after the index image is projected.
Alternatively, when there are three or more HMD 100s set in a group and no "confirmation signal" is received from most of them, it can be inferred that the projection state of the index image is not appropriate. Therefore, the HMD 100 that projects the index image may be displayed to the user so as to improve the projection state.
On the contrary, when there are 3 or more HMD100s set in a group, the "confirmation signal" is received from most of them, and when the "confirmation signal" is not received from a small part, the index of the HMD100 that does not transmit the "confirmation signal". It can be inferred that the imaging state of the image is not appropriate. In this case, it may be displayed so as to improve the imaging state for the small number of users of the HMD 100.

Summarizing the above, the HMD 100 related to the use of the user on the side sharing the virtual information image (the side that captures the index image) transmits a predetermined confirmation signal when the index image is captured. Based on whether the number of confirmation signals is larger or smaller than the total number of HMD100s, it is possible to display a display urging the user who projects the index image or the user who captures the index image to use it appropriately. Become.

As described above, according to the information processing terminal (HMD100 or the like) of the present invention, the other information processing terminals are instructed to share the virtual image based on the index image projected on the real object by one information processing terminal. It is possible to display an object in augmented reality on its display. Therefore, it is possible to display augmented reality, which is optimal for "sharing" data and files.

<Other embodiments>
The present invention is also realized by executing the following processing. That is, a computer program that realizes the functions of the above-described embodiment is supplied to the system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program. It is a process.
The configuration, shape, size, and arrangement relationship described in the above embodiments are only schematically shown to the extent that the present invention can be understood and implemented. Therefore, the present invention is not limited to the described embodiment, and can be changed to various embodiments as long as it does not deviate from the scope of the technical idea shown in the claims.

100 HMD (information processing terminal)
101 Head mounting unit 102 Imaging unit 103 Projection unit 104 Display unit 130 Display control unit 140 Imaging control unit 150 CPU
160 Memory 170 Power supply unit 180 Communication unit 190 Projection control unit 4082 Virtual object 5021 Table 5031 Virtual object 5041 Index image 5051 Virtual object

Claims (6)

It is a display control means for controlling the display of a virtual object based on three-dimensional shape data on the display unit, and the back side portion of the first finger of the user who operates the virtual object is in front of the virtual object. A display control means for displaying the virtual object on the display unit so that the ventral portion of the user's second finger is behind the virtual object, and a display control means.
When the operation of shifting the first finger and the second finger in opposite directions while the virtual object is pinched by the first finger and the second finger is detected, the virtual object is detected. An information processing apparatus comprising: a receiving means for receiving an instruction for selecting a menu associated with the object in advance. The information processing apparatus according to claim 1, wherein the menu is a command via the virtual object. The information processing apparatus according to claim 1 or 2, wherein the first finger is a thumb and the second finger is an index finger. The display control means is characterized in that the display unit displays a composite image obtained by imaging a real space and combining an image of the user's finger and an image of the virtual object. The information processing apparatus according to any one of 3. In the display control process for controlling the display of a virtual object based on three-dimensional shape data on the display unit, the back side portion of the first finger of the user who operates the virtual object is in front of the virtual object. A display control step for displaying the virtual object on the display unit so that the ventral portion of the user's second finger is behind the virtual object, and a display control step.
When the operation of shifting the first finger and the second finger in opposite directions while the virtual object is pinched by the first finger and the second finger is detected, the virtual object is detected. An information processing method comprising: a reception process for receiving an instruction to select a menu associated with the object in advance. Computer
It is a display control means for controlling the display of a virtual object based on three -dimensional shape data on the display unit, and the back side portion of the first finger of the user who operates the virtual object is in front of the virtual object. A display control means for displaying the virtual object on the display unit so that the ventral portion of the user's second finger is behind the virtual object, and a display control means.
When the operation of shifting the first finger and the second finger in opposite directions while the virtual object is pinched by the first finger and the second finger is detected, the virtual object is detected. A program for functioning as a reception means for receiving an instruction to select a menu associated with the object in advance.

Images