JP2019075125A - Transmission terminal, reception terminal, transmission/reception system, and program therefor - Google Patents

Abstract

To provide a technology for displaying a virtual object related to sharing of each user in association with common position coordinates on a real space when performing so-called augmented reality display in which a plurality of users display the virtual objects in association with the position coordinates on the real space, respectively.SOLUTION: An information processing system is formed by using first and second HMDs 100 each including: a head mounting part 101 to be mounted on a head H of the user; an imaging part 102, a display part 104 for displaying an image of virtual information while superposing it on a field of view of the user; and an information processing part 150. The first HMD 100 includes an index image projection part 103 for projecting an index image (virtual object 5041). The second HMD is configured to share an image of predetermined virtual information while changing a display position/size/direction of the image of the virtual information on the basis of the index image projected on a real object by the first HMD.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a transmitting terminal, a receiving terminal, a transmission / reception system, and a program thereof, and relates to a technique of Augmented Reality or Mixed Reality which displays a virtual object in association with position coordinates in a real space.

  In recent years, when the same physical space is viewed from different viewpoints by two or more data processing terminals, the virtual objects are different on the two or more data processing terminals as if they were present on the physical space. There has been proposed a technique for displaying the orientation (field angle). In order to realize this technology, so-called geotags are required.

  A geotag is a virtual object given in advance information of latitude, longitude (possibly altitude) and direction. When the real space including the latitude and longitude is viewed through the information processing terminal, if the display position and the direction on the display of the virtual object are determined based on the geotag information, the virtual object is displayed as if it exists on the real space. It becomes possible to display. Also, if common geotag information can be used for two or more information processing terminals, if each information processing terminal sees the real space from different viewpoints, is it as if a virtual object exists on the real space? Can be displayed in different directions (angles of view).

  By the way, since this geotag is given to the virtual object in advance in association with the position coordinates in the real space, it is desirable to display "commonly" on the spot for two or more information processing terminals It can not be used for virtual objects.

  On the other hand, for example, with regard to a virtual object to be displayed on one image processing apparatus, Patent Document 1 can notify the other image processing apparatus of its position data, and can display it on the other image processing apparatus without inconsistency. Disclose the technology to do so. Specifically, a technique is disclosed in which information on the position and direction of the virtual object is notified from the terminal that originally displayed the virtual object to another terminal. That is, with respect to data to which geotag information has not been added until then, geotag information is added at one of the virtual display points in the original terminal, and when that data is shared, the geotag information is shared Notify another terminal of

JP, 2011-175439, A

  However, in the method disclosed in Patent Document 1, it is necessary for each information processing terminal to be able to accurately acquire position and direction information. However, the position information based on GPS (Global Positioning System) generally used at present is said to have an error of several meters, and there are also cases where position acquisition can not be performed due to radio wave interruption indoors or underground.

  An object of the present invention is to provide a technology that enables sharing with another information processing terminal when “sharing” is instructed for a virtual object displayed in augmented reality on one information processing terminal.

A transmitting terminal according to the present invention is a transmitting terminal which is attached to the head or face of a first user and transmits three-dimensional shape data for displaying a virtual object.
Selection means for selecting whether to share the virtual object;
A first display in which the real object is displayed, and an image of the virtual object which can be selected or selected to be shared by the selection means in the vicinity of the real object is viewed from a first viewpoint in the virtual space Means,
Group setting means for setting a group relating to sharing of the virtual object based on physical contact between the wearers;
It is characterized by having.

The receiving terminal according to the present invention is a transmitting terminal according to any one of claims 1 to 6, which is mounted on the head or face of a second user and which displays three-dimensional data for displaying a virtual object. A receiving terminal to receive from
Receiving means for receiving the three-dimensional shape data and index image source data for projecting an index image;
A second imaging unit for capturing an index image projected on the real object to obtain second index image imaging data;
Second display means for displaying the real object and displaying an image of the virtual object selected to be shared by the selection means in the vicinity of the real object as viewed from a second viewpoint in the virtual space;
Based on the index image original data and the second index image imaging data,
A relationship between the position and orientation of the index image and the position and orientation of the second imaging unit in a physical space;
Calculating means for determining the position and orientation of the second viewpoint such that the relationship between the position and orientation of the virtual object and the position and orientation of the second viewpoint in the virtual space is substantially the same;
The position and orientation of the second imaging means and the position and orientation of the second viewpoint follow the changes in the position and orientation of the head or face of the second user.

  A transmitting and receiving system according to the present invention is characterized by comprising the transmitting terminal and the receiving terminal.

  According to the present invention, an augmented reality display of a virtual object instructed to be displayed in common at a common position on the real space without the need for a highly accurate position measurement unit or another projection projection unit Can.

It is a figure which shows typical embodiment of the information processing terminal of this invention used by the information processing system of this invention. It is a flowchart which shows the flow of judgment / process of the 1st information processing terminal which comprises the information processing system of this embodiment. It is a flowchart which shows the flow of judgment / process of the 2nd information processing terminal which comprises the information processing system which concerns on this embodiment. It is a transition diagram which shows an example of the display transition in the display part of a 1st information processing terminal, and shows four display states (401)-(404) of the first half. It is a transition diagram which shows an example of the display transition in the display part of a 1st information processing terminal, and shows four display states (405)-(408) of the latter half. It is a state diagram which shows the use condition of the information processing terminal of this embodiment typically in a series of procedures, and shows a setting step (501) and a start step (502). It is a state diagram which shows the use condition of the information processing terminal of this embodiment typically in a series of procedures, and shows the step of a share instruction | indication (503) and the projection step (504) which projects an index. It is a state figure which shows the use condition of the information processing terminal of this embodiment typically, and shows the step (505) which became shareable. It is a block diagram which shows the structure of the electronic circuit incorporated in the main body of the information processing terminal of this embodiment.

  The transmission / reception terminal (information processing terminal) of the transmission / reception system according to the present embodiment has a function as a transmission terminal and a function as a reception terminal, but for convenience of explanation, the transmission terminal and the reception terminal will be separately described.

The transmission terminal is mounted on the head or face of the first user, and has projection means, first imaging means, first display means, and transmission means.
The projection means projects the index image onto the real object based on the index image source data.
First imaging means captures the index image projected onto the real object by the projection means to obtain first index image imaging data.
On the first display means, the real object is displayed, and an image of the virtual object seen from a first viewpoint in a virtual space is displayed in the vicinity of the real object.
The transmitting means transmits 1) three-dimensional shape data for displaying a virtual object, 2) the index image source data, and 3) the index image pickup data.
The position and orientation of the projection unit, the position and orientation of the first imaging unit, and the position and orientation of the first viewpoint sequentially follow changes in the position and orientation of the head or face of the first user.

The receiving terminal is mounted on the head or face of the second user, and includes receiving means, second imaging means, second display means, and calculation means.
The receiving means receives 1) three-dimensional shape data for displaying a virtual object, 2) the index image original data, and 3) the first index image imaging data transmitted from the transmission terminal.
A second imaging unit captures the index image projected on the real object to obtain second index image imaging data.
On the second display means, the real object is displayed, and an image of the virtual object seen from a second viewpoint in the virtual space is displayed in the vicinity of the real object.
The calculation means is based on the index image original data, the first index image imaging data, and the second index image imaging data.
A relationship between the position and orientation of the index image and the position and orientation of the second imaging unit in a physical space;
The position and orientation of the second viewpoint are determined such that the relationship between the position and orientation of the virtual object and the position and orientation of the second viewpoint in the virtual space is substantially the same.
Then, the position and orientation of the second imaging means and the position and orientation of the second viewpoint sequentially follow changes in the position and orientation of the head or face of the second user.

Hereinafter, preferred embodiments (examples) of the present invention will be described based on the drawings.
An information processing terminal (transmission / reception terminal) according to the present embodiment will be described with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 6, and the like.

<Configuration of Appearance of Information Processing Terminal (Transmit / Receive Terminal)>
The information processing terminal (transmission / reception terminal) is configured with a computer having an information processing function as a basic element. In the following description, an example of an HMD (Head Mounted Display) mounted on the head of a person as an information processing terminal will be described. In the following description, for convenience of description, in principle, it will be described as "HMD" instead of "information processing terminal". Each of the two HMDs includes a mounting unit to be mounted on the head of the user, a projection unit, an imaging unit, a display unit, and an information processing unit.

(A), (B), and (C) of FIG. 1 show the external view of HMD100. 1 (A) is an external perspective view from the front side, FIG. 1 (B) is an external perspective view from the rear side, and FIG. 1 (C) is a side view of a state of being attached to the head of the user. is there. Generally, in the HMD, a so-called goggle type in which the eyepiece is in close contact and fixed around the user's eye or an entire head of the user is covered by an extendable head mounting member. There are various types such as so-called helmet types that are fixed at the jaw portion. Also, in recent years, a relatively lightweight eyeglass frame type has also appeared, in which a frame is attached to and fixed to the user's ear in consideration of wearability.
The HMD 100 shown in (A) to (C) of FIG. 1 corresponds to the so-called goggle type described above.

  In the HMD 100, the head mounting unit 101 is for mounting the HMD 100 on the head of the user, and as shown in FIG. 1, is composed of a plurality of constituent members numbered 121 to 125. In order to attach the HMD 100 to the head H of the user as shown in FIG. 1C, first, the head H is covered with the length adjustment portion 123 loosened by the adjuster 122. Then, after the forehead attachment portion 125 is brought into close contact with the forehead of the head H, the length adjustment portion 123 is squeezed with the adjuster 122 so that the temporal attachment portion 121 and the occipital attachment portion 124 are brought into close contact with the temporal region and occipital region. .

  The imaging unit 102 is a so-called digital camera, and is composed of a plurality of members denoted by reference numerals 112, 116, and 117 as shown in FIG. 1C, and the direction of the user facing the HMD 100 attached to the head H Images are taken in substantially the same direction, and sequentially follow changes in the position and orientation of the user's head. Specifically, light incident from the outside of the HMD 100 through the incident window 112 is guided to the inside of the HMD 100 by the optical prism 117, and is received by the imaging sensor 116. The members 112, 116, and 117 have elements corresponding to the left eye and the right eye, respectively, and are distinguished by adding symbols "L" and "R" in the drawing.

The projection unit 103 is a so-called projector, and includes a projection lens, an image display element, a light emitting element, and the like.
The display part 104 is comprised from the several member which attached number 110,114,115, as shown to FIG. 1, (B), (C). The symbols "L" and "R" indicate for the left eye and for the right eye as described above. It is provided at a position corresponding to the spectacle lens in the glasses so as to be opposed to the position of the user's eyes, and more specifically, the image displayed by the color liquid crystal display 114 is guided by the optical prism 115, Display on

  Here, there are a glass see-through (optical see-through) method and a video see-through method as methods for superimposing graphics on a scene in the real world. The glass see-through method projects graphics using a transparent or translucent spectacle lens as a screen, and optically superimposes graphics on a real-world scene. The video see-through method electrically superimposes graphics on the view of the real world captured by an imaging device.

When the HMD 100 is of the above-described video see-through type, the screen 110 basically has no transparency, and displays the guided image as it is.
On the other hand, when the HMD 100 is of the above-mentioned glass see-through type, the screen 110 is configured by a so-called half mirror. That is, it has a certain degree of transmission, and allows the user to optically view the scene in real space. At the same time, the color liquid crystal display 114 displays and reflects the graphic image guided by the optical prism 115 in the direction of the user's eyes by a mirror embedded on the front or back or inside. That is, in the case of the glass see-through type, the view of the real space and the image of the virtual object (virtual object) are optically superimposed on the screen 110.

The output light from the optical prism 115 of the display unit 104 and the input light to the optical prism 117 of the imaging unit 102 coincide with the optical axis of the user's pupil, and the imaging sensor 116 Capture a real space image seen by position and direction (position and orientation).
In the case of the video see-through HMD 100, the color liquid crystal display 114 displays an augmented reality image in which the real space image acquired by the imaging sensor 116 and the image of the virtual object are electrically superimposed (image synthesis).
On the other hand, in the case of the glass see-through HMD 100, the color liquid crystal display 114 basically displays only the image of the virtual object.

<Configuration of Internal (Electronic Circuit) of Information Processing Terminal>
As shown in FIG. 6, the HMD 100 further includes a display control unit 130, an imaging control unit 140, a CPU 150, a memory 160, a power supply unit 170, a communication unit 180, and a projection control unit 190 as electronic circuit elements. These electronic circuit elements are embedded inside the main body of the HMD 100 (including the head mounting portion 101) and are not shown in appearance.
In the above, the CPU 150, the memory 160, the display control unit 130, the imaging control unit 140, and the projection control unit 190 constitute an information processing unit of the HMD 100.

  The display control unit 130 performs display control of the display unit 104. For example, it controls the size and position, orientation, color, transparency of the image of the virtual object to be superimposed (image synthesis) on the physical space image, and control of movement and lightness change due to the physical space image changing.

  The imaging control unit 140 performs exposure control, distance measurement control, and the like based on the calculation result by predetermined calculation processing using the imaging data. As a result, AF (Auto Focus) processing, AE (Auto Exposure) processing, AWB (Auto White Balance) processing, etc. are performed. When the imaging unit 102 includes a mechanism for inserting and removing the optical filter from the optical path, a vibration reduction mechanism, and the like, control of the insertion and removal of the filter and the vibration reduction is performed according to the imaging data and other conditions.

The CPU 150 performs arithmetic processing of the entire HMD 100. Each process of the present embodiment is realized by executing a program recorded in a memory 160 described later.
The memory 160 comprises a work area and a non-volatile area. In the work area, a program read from the non-volatile area and constants and variables for system control are expanded. In addition, data of an image of a virtual object to be displayed superimposed on a physical space is held for display. In addition, the imaging data imaged and A / D converted by the imaging unit 102 is expanded for the purpose of image analysis, image processing, and the like.

The power supply unit 170 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, a Li battery or the like, an AC adapter or the like, and supplies power to the entire HMD 100. It has a power switch that switches power on / off according to user operation and other conditions.
The communication unit 180 communicates with another HMD 100 according to a standard such as WiFiDirect based on the control of the CPU 150 to transmit and receive three-dimensional shape data, data related to position and orientation, and the like.
The projection control unit 190 controls the projection operation of the projection unit 103.

<Flow of judgment / process of information processing terminal>
FIG. 2 shows the flow of determination and processing performed by the first HMD 100 having the above configuration, and FIG. 3 shows the flow of determination and processing performed by the second HMD 100 (receiving terminal) having the above configuration. There is. Note that this determination and processing are realized by the CPU 150 developing and executing a program recorded in a non-volatile area of the memory 160 in a work area.

<Flow of determination / process of first information processing terminal>
First, with reference to FIG. 2, the flow of determination and processing of the first HMD 100 (transmission terminal) according to use of a user (user) who instructs sharing of data will be described.
In step S201, when the power is turned on by the user using the first HMD 100, the imaging unit 102 starts imaging of the physical space.

  In step S202, the image of the virtual object is displayed on the display unit 104 under predetermined display conditions (size, position, orientation, color, transparency, etc.) superimposed on the image of the physical space being captured. The virtual graphics displayed immediately after the power is turned on are, for example, graphics (three-dimensional icons) that symbolically represent the storage area of the nonvolatile area related to the management of the user of the HMD 100 according to the specifications as the information processing terminal It is also good. In addition, graphics displayed before power-off may be recovered and used.

In step S203, the first HMD 100 determines which of the virtual graphics being displayed by the user has been instructed / operated. In this case, virtual graphics displayed superimposed on the image in the real space correspond to so-called graphical user interfaces (hereinafter referred to as "GUI").
For example, when a three-dimensional icon (storage icon) symbolically representing a storage area related to user management is displayed, when “open” is instructed by the user's gesture operation or the like, it is stored in the storage area Display graphics that represent files Also, for example, when an icon representing the design data of “shoes” is displayed symbolically, when “open” is instructed by the user's gesture operation or the like, the three-dimensional model of “shoes” is displayed in the real space Superimposed on the image of.

  By the way, when displaying virtual graphics superimposed on an image in real space, this is displayed in association with position coordinates in real space, so-called augmented reality display is not related to position coordinates in real space There are cases where it is displayed by (when not displaying augmented reality). When to display augmented reality and when to not display augmented reality are based on predetermined conditions. The setting of the condition for performing or not performing the augmented reality display will be described later.

  When the user instructs or operates on virtual graphics (YES in step S203), it is determined whether the instruction or operation is an instruction or operation of “sharing” data indicated by the graphic ( Step S204). If the instruction / operation is other than "sharing", the content of the instruction is executed (step S205).

  If it is determined in step S209 that the instruction / operation of the user is the instruction / operation of “share”, first, the display unit 103 displays the virtual graphics for which the sharing is instructed and the display position at which the virtual graphics are displayed. The index image is projected in the same direction on the real space. However, at this time, as shown in the following steps S206 to S209, based on the projection direction angle detected by the angle sensor, the index image is so-called so that it becomes similar to that projected in the vertical or horizontal direction. Correct the keystone.

  For example, when the HMD 100 projects the index image toward a substantially vertical surface such as a wall surface in a direction forming an angle of θ with the horizontal direction, the direction along the projection direction of the index image (from the HMD 100 The so-called upper and lower directions) and the upper side with respect to the lower side as viewed from the HMD 100 are each deformed by 1 / cos θ. Therefore, if correction is performed to multiply the original image by cos θ, respectively, it becomes similar to one projected horizontally regardless of the angle of the information processing terminal.

  When the HMD 100 projects the index image toward a substantially horizontal surface such as a table surface in a direction forming an angle of θ with the horizontal direction, the direction along the projection direction of the index image (HMD 100 The so-called upper and lower direction) when viewed from the top and the upper side relative to the lower side when viewed from the HMD 100 are each deformed by −1 / sin θ times. Therefore, if the original image is corrected to be multiplied by −sin θ, it becomes similar to that projected vertically regardless of the angle of the HMD 100.

For example, when the attitude (angle θ with respect to the horizontal direction) of the HMD 100 is −45 ° ≦ θ ≦ 45 °, it can be estimated that projection is performed toward a substantially vertical plane.
Further, when the attitude (angle θ with respect to the horizontal direction) of the HMD 100 is θ ≦ −45 ° or θθ45 °, it can be estimated that projection is performed toward a substantially horizontal surface.

That is, in step S206, the projection direction angle is acquired by the angle sensor, and in step S207, it is estimated whether to project toward a substantially vertical surface or a substantially horizontal surface.
In step S208, the index image is subjected to so-called keystone correction in accordance with either a substantially vertical surface or a substantially horizontal surface.
In step S209, the projection unit 103 projects the corrected index image on the projection surface.

  Next, in step S210, the projected index image is captured by the imaging unit 102. Since the original image of this index image is subjected to trapezoidal correction similar to that projected in the vertical or horizontal direction by the process described above, the inclination of the imaged index image is determined by the imaging direction of the HMD 100 and the projection. It reflects the inclination between the surface. In addition, the size of the captured index image reflects the distance between the HMD 100 and the projection plane by performing correction in consideration of the spread angle of the projection image and the magnification (so-called zoom ratio) of the imaging lens. Become.

  Next, in step S211, the position, orientation (position and orientation), and size of the virtual graphics instructed to be shared are changed and superimposed based on the captured image of the index image. In this way, it is possible to display augmented reality as if the virtual object is placed or glued on the index-projected real object.

  In the above, in the first HMD 100, the projection direction of the index image by the projection unit 103 and the imaging direction of the index image by the imaging unit 102 coincide with the viewpoint (position and orientation) of the first user. In the first HMD 100, based on index image source data used for projection and index image imaging data obtained by imaging the projected index image, the inclination or the like of the projection surface on which the index image is projected is specified. . Then, the inclination and the like are reflected on the position and orientation of the virtual object instructed to be “shared” and superimposed and displayed in the vicinity of the projection plane, thereby performing the augmented reality display.

Note that the projection of the index image may be by a ray visible to the human eye or by an invisible ray (for example, infrared rays) as long as the imaging unit 102 of the HMD 100 can capture an image. Alternatively, instead of continuous projection, for example, it may project discontinuously at fixed time intervals.
Thus, for example, a pattern or the like is drawn on a real object (projection plane), and it is difficult to distinguish between a projected index image and a projection plane pattern in continuous projection using visible light, Even when recognition is difficult, recognition of the index image can be facilitated. By removing the influence of a pattern or the like by projection / recognition by infrared light, or by projecting discontinuously at fixed time intervals and extracting the projected image from the difference between the captured image before and after the projection, the index image It becomes possible to make recognition easy.

Next, in step S212, the information (three-dimensional shape data, position and orientation data, etc.) related to the virtual object and the information (index image original data, index image imaging) related to the index image from the communication unit 180 of the first HMD 100 Data) and send.
In step S213, the display unit 104 displays a message indicating that “sharing setting is in progress”. This display may display, for example, characters such as "in sharing setting", or may display graphics of a virtual object in a color or transparency different from before or blink them. It may be something.

And it waits until it receives a confirmation signal transmitted from other HMD100 (2nd HMD100) (Step S214). The confirmation signal will be described later.
However, a predetermined end condition is provided, and when the condition is satisfied (for example, when a predetermined time has elapsed), the standby is ended (step S215). When the confirmation signal is received, the display of "in sharing setting" is erased (step S216).

<Flow of determination / process of second information processing terminal>
The flow of the determination and processing of the second HMD 100 (receiving terminal) will be described with reference to FIG.
In step S301, when the power is turned on by the second user who uses the second HMD 100, the second HMD 100 starts imaging of the physical space by the imaging unit 102.
Then, in step S302, the second HMD 100 transmits information (three-dimensional shape data, position and orientation data) related to the virtual object instructed to be shared and information related to the index image (index information) transmitted from the other first HMD 100. It waits until the original image data, the index image imaging data) is received.
However, a predetermined termination condition is provided, and when the condition is satisfied (for example, when a predetermined operation by the user is made), the standby is terminated (step S303).

  In step S304, when the second HMD 100 receives the information related to the virtual object instructed to be shared and the information related to the index image, the second HMD 100 displays this information on the display unit (second display unit) of the second HMD 100. Is displayed to the second user. At this time, if the position of the first HMD 100 can be acquired, a display prompting the user to turn in that direction may be displayed.

Then, in step S305, it is determined whether there is an index image projected by the projection unit of the first HMD in the image captured by the imaging unit (second imaging unit) of the second HMD.
In step S306, when there is an index image, first, a “confirmation signal” to that effect is transmitted to the first HMD 100. This is to notify the first HYMD 100 (and its user) that the virtual object has been shared. The imaging direction by the imaging unit 102 of the second HMD 100 matches the viewpoint and line of sight of the user who uses this.

In step S307, the position, size, and orientation (direction and inclination) of the index image in the captured image are acquired.
For example, if the pattern of the index image is neither point-symmetrical nor line-symmetrical, the actual image on which the index image is projected (the projection surface is obtained by comparing the received index image original data with the first index image imaging data And the position and orientation of the first imaging means with respect to the projection plane can be specified.
Further, the position and orientation of the second imaging means with respect to the projection plane can be specified by comparing the received index image original data with the second index image pickup data.
Then, if the relationship between the position and orientation of the first imaging means, the position and orientation of the projection plane, and the position and orientation of the second imaging means is specified,
Identify the relationship between the position and orientation of the first viewpoint in the virtual space, the position and orientation of the virtual object displayed superimposed on or near the projection plane, and the position and orientation of the second viewpoint in the virtual space be able to.

Then, in step S308, rendering is performed based on information (shape data, data of position and orientation) related to the virtual object instructed to be shared.
In step S309, the image of virtual information (image of virtual graphics) obtained by rendering is superimposed on the image of real space captured by the imaging unit 102 and displayed on the display unit 104.
In this way, the position and orientation are specified, and a virtual object displayed superimposed on or near the projection surface (for example, a virtual object disposed about 1 cm above the projection surface and whose right side can be seen from the first viewpoint) The second display unit 104 displays an image obtained by viewing the image from the second viewpoint in the virtual space.
For example, in the positional relationship as shown in FIG. 5C, the display portion of the user P3 is disposed about 1 cm above the projection plane, and a virtual object (shoes) viewed from the "right" side is displayed. In the display section of, the virtual object viewed from the “left” side is displayed about 1 cm above the projection plane.

  In the second HMD 100, sharing of a virtual object (virtual object) is performed through the above-described steps S307, S308, and S309 through the index image projected by the first HMD 100. Steps S307, S308, and S309 constitute step S311 for the sharing process, and realize the sharing means. Thereby, the predetermined virtual object (step S211) displayed on the first HMD 100 is also displayed on the second HMD 100 based on the viewpoint, and sharing is realized.

After the second HMD 100 displays the image of the virtual information related to the sharing instructed virtual object in a superimposed manner, the image of the virtual information is displayed as if it were present at a specific position in the real space. (Step S310). That is, when, for example, the second HMD 100 recognizes that the second HMD 100 has approached the specific position by recognizing the feature points of the captured image obtained by capturing the real space, the image is re-rendered so that the graphics of the object become large. Superimposed on
Also, temporarily, if the above specific position deviates from the user's field of view (i.e., the captured image), the graphics of the virtual object are erased once. However, when the specific position is again in the field of view of the user, the graphics of the virtual object are redisplayed as existing at that position, based on the information of the feature points stored in the memory 160.

<Operation / Effect of Embodiment>
To summarize the above, when the first HMD 100 instructs the user (user) to “share” the virtual object displayed in augmented reality, the direction in which the augmented reality is displayed Project the index image towards the The second HMD 100 captures the projected index image, and displays the virtual object on an augmented reality display based on the index image, and information on the transmitted sharing instructed virtual object and information on the index image. Do.
The above-described index image in the captured image of the second HMD 100 is, as a matter of course, viewed from the viewpoint of the second HMD 100, and reflects the inclination or the like of the projected surface of the real object (for example, "table" etc.) It is a thing.

That is, the second HMD 100 generates graphics of a virtual object of a position and orientation corresponding to the position, size, and orientation (direction and inclination) of the index image in the captured image, and displays augmented reality on the display unit. Thus, the first HMD 100 and the second HMD 100 can display the same object from the respective viewpoints on the respective display units 104 as if they were at the same position in the real space. .
Therefore, for example, when the first user points the virtual object by a real object such as a finger, the virtual object and the real object are superimposed and displayed on the second display unit, and the first user is notified to the first user. Appear to be pointing at the virtual object. Thus, an augmented reality display suitable for the first user and the second user to "share" a virtual object is possible.

At this time, the position and orientation of the first viewpoint when drawing a virtual object displayed on the first display unit, and the position of the second viewpoint when drawing a virtual object displayed on the second display unit. The relationship with the viewpoint substantially matches the relationship between the position and orientation of the first imaging unit and the position and orientation of the second imaging unit.
Although it is possible to give the same appearance by so-called geotags, the information processing terminal of the present invention (for example, the HMD 100 described in the above embodiment) gives the virtual object information of latitude and longitude in advance. There is no need. In addition, it is not necessary to acquire the latitude and longitude information of each terminal.
In the system using the information processing terminal of the present invention, there is a real object capable of projecting an index image in the real space where the first user and the second user exist, and the first terminal is projected on the projection plane It is sufficient if the first terminal and the second terminal can each capture an index image.
By imaging the common index image from the position and orientation of each of the first terminal and the second terminal, the position and orientation of the first imaging unit in the physical space, the position and orientation of the projection plane, and the second The relative relationship (first relative relationship) with the position and orientation of the imaging unit can be specified.
Then, the relative relationship (second relative relationship) between the position and orientation of the viewpoint of the first terminal in the virtual space, the position and orientation of the virtual object, and the position and orientation of the viewpoint of the second terminal is the first The position and orientation of the viewpoint of the second terminal are specified so as to be substantially the same as the relative relationship.
Thus, a virtual object displayed superimposed in the vicinity of the projection plane is displayed on the first display means as viewed from the first viewpoint, and a second display of the same virtual object viewed from the second viewpoint is displayed. It has the effect of the invention in that it can be displayed on the means and shared augmented reality display is possible.

<Display transition in display unit of first information processing terminal>
Next, with reference to FIGS. 4A and 4B, an operation example of the first HMD 100 for projecting an index image when a user instructs to share an image of virtual information will be described. In FIGS. 4A and 4B, display transition in the display unit 104 of the first HMD 100 is shown by a series of display states (401) to (408). The first four display states (401) to (404) are shown in FIG. 4A, and the last four display states (405) to (408) are shown in FIG. 4B.

  The display state (401) shows an example of display immediately after power on in the first HMD 100. In the display state (401), reference numeral 4011 indicates a captured image of the physical space captured by the imaging unit 102. Further, reference numeral 4012 is virtual graphics superimposed on the captured image 4011, and here, a storage icon representing a storage area related to the management of the user of the first HMD 100 is displayed.

The display state (402) represents a mode in which the user is going to instruct / operate the storage icon 4012 by a "finger pinch" gesture. The user's hand 4023 is a part of the captured image of the real space captured by the imaging unit 102. Here, with respect to the virtual graphic 4022, the back side (nail side) portion 4024 of the user's thumb is displayed in front, and the belly side (nail side) portion 4025 of the user's index finger is displayed in back Display control is done.
This is done to give the user a visual sense of pinching graphics with a finger, and an image that identifies and extracts the back side portion and the ventral side portion of the human finger from the captured image, respectively. It is based on processing technology. Such an image processing technique is already known as disclosed in, for example, Japanese Patent Application Laid-Open No. 2012-53674.

The display state (403) represents a mode in which "display the contents of storage (so-called" open ")" is instructed on the storage icon 4031 by a predetermined gesture operation of the user. Here, the predetermined gesture operation is a gesture that “the icon is pinched with a finger and then shifted with the thumb and forefinger attached”. This gesture is common to the operation of opening the bundled cards in a fan-like manner, so it is easy to recall the word “open”, and is suitable as a gesture for instructing so-called “open”.
Of course, there is no problem if another gesture is assigned as a so-called "open" gesture.

  In the display state (404), the lower-level hierarchy of the storage area represented by the storage icon 4031 is displayed as a result of the user's instruction to "open" the storage icon 4031 in the display state (403). It represents the displayed mode. In this display state (404), the icons 4041 to 4048 are icons for displaying data or files stored in the storage unit of the first HMD 100, respectively. For example, the icon 4048 is for displaying one file of application software for displaying a three-dimensional model on the display unit 104.

  The display state (405) shown in FIG. 4B represents a mode in which "(Open application software)" is instructed to the file icon 4052 by the user's predetermined gesture operation. Here, the predetermined gesture operation is a gesture similar to that performed in the display state (403), "a gesture of pinching the icon with a finger and moving with the thumb and forefinger attached".

  In the display state (406), application software for displaying a solid model is activated, the file indicated by the icon 4052 is expanded in the memory 160, and the "shoe" object 4062 is displayed superimposed on the captured image 4061 in the real space Represents the manner in which

  The display state (407) represents a state in which the user has made a gesture "squeezing an object with a finger and shifting with the thumb and forefinger still applied" on the solid model 4062 in the display state (406). ing. When this gesture is made, a menu of commands that can be instructed by the user on the object is displayed according to the gesture. The commands include "COPY", "CUT", "EDIT", "SHARE" and the like. After displaying the menu, the user can further shift the thumb to indicate / select one command.

In the display state (408), an augmented reality display in which the object is associated with position coordinates in the real space as a result of the command "SHARE" being selected by the user and the "share" of the object being instructed Represents That is, in the drawing, the captured image 4081 is a captured image of the real world, and the virtual object 4082 is a virtual object displayed in augmented reality.
For example, by displaying a shadow under the virtual object, the virtual object “shoes” is displayed as if it were placed on the “table” of the real object, so that augmented reality is displayed. It can not distinguish between virtual and real.

  When transitioning from the display state (407) to the display state (408), the display of the virtual object is changed from so-called normal superimposed display to augmented reality display. The display mode is changed in such a manner so as to display an augmented reality display for an object indicating a shared file, and not an augmented reality display for an object indicating a file not shared. This is because the display conditions were set. The reason why the display mode according to the augmented reality or the display mode according to the non-augmented reality is different depending on whether the file is shared or not shared is as follows.

  That is, when a plurality of users (users) gather at a specific place to perform a meeting or collaboration, if the augmented reality display is performed as if the object is actually present, the effect of the meeting or collaboration And improve efficiency. For example, when one user gives a presentation pointing to a virtual object, the other users can simultaneously see the virtual object and a presenter pointing to it.

On the other hand, if the display object is an object related to a "non-shared" file not shared with other users, the display of the object can not be related to the position coordinates in the real space if the non-augmented reality display is used. . Therefore, for example, when the user does not face the direction of the position coordinates, the object is not displayed.
In other words, it is rational to change the display mode so as not to display the augmented reality display for the object related to the file “non-shared” and to display the augmented reality display for the object related to the file “shared”.

<A series of procedures based on the usage state of the first and second information processing terminals>
Next, with reference to FIGS. 5A, 5B, and 5C, an example of a series of procedures for “sharing” virtual objects using the first and second HMDs 100 described above will be described. In this example, a plurality of users wearing the HMD 100 on their heads gather at a specific place to hold a meeting or the like. In such a case, one user instructs "sharing" of the virtual object, and the other user displays the virtual object in relation to the common position coordinates from each viewpoint and displays the augmented reality, It is represented schematically.

  Of the above sequence, FIG. 5A shows the setting step (501) and the starting step (502). FIG. 5B shows a display mode as viewed from the first HMD 100, and shows a step of sharing instruction (503) and a projection step (504) of projecting an index from the first HMD 100. FIG. 5C shows a state (505) in which a virtual object can be shared with the second HMD 100 based on the index projected from the first HMD 100.

  In the setting step (501), the users P1 and P2 wearing the above-described HMD 100 perform user group setting by a predetermined method. Here, a gesture in which the users P1 and P2 wearing the HMD 100 shake hands with each other is a method of group setting. About making this gesture the method of group setting, it has the following meaning.

First of all, "handshake" is natural as a pre-meeting act. Second, it is possible to prevent careless sharing of objects because identification of whether or not a gesture has been made is reliable as compared with other gestures (for example, "bow"). Third, by setting groups based on the fact that the handshake was actually performed (not only the appearance that the handshake was performed but also the actual physical contact), the unintended person is put into the group. It is possible to set a group with high security without fear of adding it.
In order to realize the third significance, it is necessary for the HMDs 100 to be provided with means for detecting the body contact between the wearers of the HMDs 100, for example, by the transition of the electric field state or the like. In addition, the technique which detects a body contact by the transition of an electric field state is a well-known technique, as Unexamined-Japanese-Patent No. 2010-193129 etc. disclose, for example.

  The start step (502) shows that three group-set HMD 100 users P 1, P 2, P 3 surround the real object table 5021. In this state, three HMDs 100 display different virtual objects because there are no virtual objects related to the three sharing.

  Step (503) is the display mode as viewed from one user (wearer) P 3 of one HMD 100. Here, the virtual object 5031 is displayed only for the user P3. In step (503), a gesture operation is performed to designate “sharing” of the virtual object by the finger 5032 of the user P 3.

  The projection step (504) shows that the index image 5041 is projected on the table 5021 from the HMD 100 equipped to the user P3.

In the sharing step (505), the second HMDs 100 of the other users P1 and P2 respectively recognize the index image 5041, and based on that, an image from each viewpoint of the virtual object 5051 related to the sharing is displayed on each display unit. It shows schematically how augmented reality is displayed. That is, the virtual object 5051 of the shoe in the sharing stage (505) is not displayed (invisible) except for the users (wearers) P1 to P3 of the HMD 100.
However, on each display means of the users (wearers) P1 to P3 of the HMD 100, virtual objects 5051 viewed from the viewpoints of the respective users are displayed as if they were actually placed on the table 5021. The thing is shown schematically.

For the users P1 to P3 belonging to the group, the virtual objects 5051 are displayed in common as if they existed, which is convenient for meeting and collaborating within the group. For example, in order to explain the design of "shoes", when one of the users changes the direction of the "shoes" object while pointing in various ways, pointing at it with a finger and adding a description such as "this shape is ..." The object is displayed with the finger pointed to other users.
For this reason, the contents of the explanation become easy to understand. Alternatively, if another user points out an opinion while pointing with a finger or the like, such as "this one is better ...", it is also clear which part of the opinion it is. In addition, when it is possible for the user of the group to perform an operation of editing and modifying the virtual object 5051, for example, it is possible to use it to advance the meeting while changing the color and the pattern.

  In addition, how the user of the group can operate the object set as “shared” can be made the same as the “shared” way of thinking of the HMD 100 in general. That is, it is based on the setting of the “authority” of the group / user at the time of setting the group. For example, the authority such as “view only” or “modifiable / overwritable” is set in advance for the group / user. In addition, when it is determined that the group release is performed, the group release condition can be, for example, when the HMDs 100 are separated by a predetermined distance or more, in addition to the release operation.

  To summarize the above, the information processing terminal (first HMD 100) according to the use of the user who instructs sharing of the image of the virtual information projects the index image when the sharing instruction is given. As described above, since the other information processing terminal (second HMD 100) displays the image of the virtual information related to the sharing instruction based on the projected index image, the other information is not received until the sharing instruction is given. The processing terminal can display the image of the virtual information. That is, since the image of the virtual information is not displayed on the other information processing terminal when there is no sharing instruction, the information can be managed with high security.

Other Embodiments
In this embodiment, an example of the HMD 100 that transmits a predetermined confirmation signal when an index image related to an instruction to share an image of virtual information is captured will be described.
When the HMD 100 according to the present embodiment captures an index image as described in step S306 in FIG. 3, the HMD 100 transmits a “confirmation signal” to that effect to the HMD 100 projecting and displaying the index image.

  An index image projection unit that projects an index image is a part of the HMD 100 according to the present embodiment, and is mounted on the head of the user by the head mounting unit. For this reason, until the other HMD 100 "confirms" the index image, that is, until the index image is captured by the imaging means of the other HMD 100, the user of the HMD 100 who projects the index image moves the head greatly. It must not be. However, after the indicator image has been "confirmed" by another HMD 100, it is rationally notified that the user can immediately move the head.

  When the HMD 100 according to the present embodiment captures an index image relating to an instruction to share an image of virtual information, a predetermined confirmation signal is sent to perform this notification, and this confirmation signal is used as follows. can do.

For example, the HMD 100 that projects the index image may display that the user does not move the head while projecting the index image and until receiving the “confirmation signal”.
Alternatively, in the case where three or more HMDs 100 are set as a group, it is possible to infer that the projection state of the index image is not appropriate when the “confirmation signal” is not received from many of them. Therefore, the HMD 100 that projects the index image may be displayed to the user to improve the projection state.
Conversely, when there are three or more HMDs 100 set as a group, the indicator of the HMD 100 that does not transmit the "confirmation signal" when the "confirmation signal" is received from many of them and the "confirmation signal" is not received from a small part It can be inferred that the imaging condition of the image is not appropriate. In this case, display may be made to improve the imaging state to the user of the small number of HMDs 100.

  To summarize the above, the HMD 100 associated with the use of the user who shares the image of the virtual information (the side that captures the index image) transmits a predetermined confirmation signal when the index image is captured. Based on whether the number of confirmation signals is larger or smaller than the number of the entire HMD 100, it is possible to make a display prompting appropriate use to the user projecting the index image or the user capturing the index image. Become.

  As described above, according to the information processing terminal (HMD 100 or the like) of the present invention, based on the index image projected by one information processing terminal on a real object, another information processing terminal is instructed to share It is possible to display the object on its display unit in augmented reality. Therefore, it is possible to display augmented reality that is optimal for "sharing" data and files.

<Other Embodiments>
The present invention is also realized by executing the following processing. That is, a computer program for realizing the functions of the above-described embodiment is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU or MPU etc.) of the system or apparatus reads and executes the program. It is a process.
The configurations, shapes, sizes, and arrangement relationships described in the above embodiments are merely schematically shown to the extent that the present invention can be understood and implemented. Therefore, the present invention is not limited to the described embodiments, and can be modified in various forms without departing from the scope of the technical idea shown in the claims.

100 HMD (information processing terminal)
101 head mounted unit 102 imaging unit 103 projection unit 104 display unit 130 display control unit 140 imaging control unit 150 CPU
160 Memory 170 Power Supply Unit 180 Communication Unit 190 Projection Control Unit 4082 Virtual Object 5021 Table 5031 Virtual Object 5041 Index Image 5051 Virtual Object

Claims (12)

A transmitting terminal mounted on the head or face of a first user and transmitting three-dimensional shape data for displaying a virtual object,
Selection means for selecting whether to share the virtual object;
A first display means for displaying a real object and displaying an image of a virtual object which can be selected or selected to be shared by the selection means in the vicinity of the real object from a first viewpoint in a virtual space When,
Group setting means for setting a group relating to sharing of the virtual object based on physical contact between the wearers;
A transmitting terminal characterized by having:   The transmitting terminal according to claim 1, further comprising: group releasing means for releasing a group relating to sharing of the virtual object based on the measured distance between the terminals. Projection means for projecting an index image onto a real object based on the index image source data;
And transmitting means for transmitting the three-dimensional shape data and the index image source data.
The transmission terminal according to claim 1, wherein the projection unit projects the index image when it is selected to be shared by the selection unit.   3. The transmission according to claim 2, wherein the position and orientation of the projection means and the position and orientation of the first viewpoint sequentially follow changes in the position and orientation of the head or face of the first user. Terminal.   5. The transmission terminal according to claim 3, further comprising: a first imaging unit configured to capture an index image projected by the projection unit to obtain first index image imaging data.   The said selection means selects whether it shares the said virtual object based on the gesture regarding the virtual object which the said display means is displaying, The any one of the Claims 1-5 characterized by the above-mentioned. Sending terminal. A receiving terminal for receiving three-dimensional shape data attached to the head or face of a second user for displaying a virtual object from the transmitting terminal according to any one of claims 1 to 6, ,
Receiving means for receiving the three-dimensional shape data and index image source data for projecting an index image;
A second imaging unit for capturing an index image projected on the real object to obtain second index image imaging data;
Second display means for displaying the real object and displaying an image of the virtual object selected to be shared by the selection means in the vicinity of the real object as viewed from a second viewpoint in the virtual space;
Based on the index image original data and the second index image imaging data,
A relationship between the position and orientation of the index image and the position and orientation of the second imaging unit in a physical space;
Calculating means for determining the position and orientation of the second viewpoint such that the relationship between the position and orientation of the virtual object and the position and orientation of the second viewpoint in the virtual space is substantially the same;
A receiving terminal characterized in that the position and orientation of the second imaging means and the position and orientation of the second viewpoint sequentially follow changes in the position and orientation of the head or face of the second user. The imaging apparatus further comprises transmission means for transmitting a predetermined confirmation signal to the transmission terminal when the index image projected onto the real object is captured by the second imaging means of the reception terminal. The receiving terminal according to claim 7. The indicator image is projected by an invisible ray,
The first image pickup means of the transmission terminal and the second image pickup means of the reception terminal can pick up the index image projected by the invisible light beam, respectively. Receiving terminal described.   A transmission / reception system comprising: the transmission terminal according to any one of claims 1 to 6; and the reception terminal according to any one of claims 7 to 9.   The computer program for functioning a computer as each means of the transmission terminal of any one of Claims 1-6.   The computer program for functioning a computer as each means of the receiving terminal of any one of Claims 7-9.

Images