JP6711885B2 - Transmission terminal, reception terminal, transmission/reception system, and its program - Google Patents

Description

The present invention relates to a transmission terminal, a reception terminal, a transmission/reception system, and a program thereof, and relates to an augmented reality or mixed reality technology for displaying a virtual object in association with position coordinates in a real space.

In recent years, when the same physical space is viewed from different viewpoints by two or more information processing terminals, the virtual objects are different on the two or more information processing terminals as if they exist in the physical space. Techniques for displaying in the orientation (angle of view) have been proposed. So-called geotags are needed to implement this technology.

The geotag is a virtual object to which latitude, longitude (in some cases, altitude), and direction information have been added in advance. If the position and orientation on the display of the virtual object is determined based on the geotag information when the real space including the latitude and longitude is viewed through the information processing terminal, the virtual object is displayed as if it exists in the real space. It becomes possible to display. Moreover, if common geotag information is made available to two or more information processing terminals, even if each information processing terminal views a real space from different viewpoints, is it as if a virtual object exists in that real space? , It can be displayed in different directions (angles of view).

By the way, since this geotag is given in advance to the virtual object in association with the position coordinates in the physical space, it is desired to be commonly displayed “on the spot” on two or more information processing terminals. It cannot be used for virtual objects.

On the other hand, for example, in Japanese Patent Laid-Open No. 2004-242242, the position data of a virtual object displayed on one image processing device is notified to another image processing device and can be displayed on the other image processing device without inconsistency. A technique for doing so is disclosed. Specifically, a technique is disclosed in which a terminal that originally displayed a virtual object notifies another terminal of information about the position and direction of the virtual object. In other words, the geotag information is added to the data that was not previously attached with geotag information at some point during virtual display on the original terminal, and when the data is shared, the geotag information is shared with the other party. Notify another terminal of.

JP, 2011-175439, A

However, in the method disclosed in Patent Document 1, each information processing terminal needs to be able to acquire position and direction information with high accuracy. However, it is said that there is an error of several meters in the position information obtained by the currently used GPS (Global Positioning System), and in some cases, the position cannot be acquired indoors or underground due to radio wave interruption.

An object of the present invention is to provide a technology that enables sharing with another information processing terminal when “sharing” is instructed for a virtual object displayed in augmented reality on one information processing terminal.

A transmission terminal according to the present invention is a transmission terminal that is attached to the head or face of a first user and that transmits three-dimensional shape data for displaying a virtual object,
Selecting means for selecting whether to share the virtual object;
A first display in which the physical object is displayed, and an image of a virtual object that can be or is selected to be shared by the selection means in the vicinity of the physical object is viewed from a first viewpoint in a virtual space. Means and
Group setting means for setting a group related to sharing of the virtual object based on body contact between the wearers,
It is characterized by having.

The transmitting terminal according to any one of claims 1 to 6, wherein the receiving terminal according to the present invention is attached to a head or a face of a second user and has three-dimensional shape data for displaying a virtual object. A receiving terminal that receives from
Receiving means for receiving the three-dimensional shape data and index image original data for projecting the index image;
Second image capturing means for capturing second index image captured data by capturing the index image projected on the physical object,
Second display means for displaying the real object and displaying an image of the virtual object selected by the selection means in the vicinity of the real object and viewed from a second viewpoint in the virtual space;
Based on the index image original data and the second index image captured data,
A relationship between the position and orientation of the index image and the position and orientation of the second imaging unit in the physical space,
And a calculation unit that determines the position and orientation of the second viewpoint so that the relationship between the position and orientation of the virtual object and the position and orientation of the second viewpoint in the virtual space are substantially the same.
The position/orientation of the second image pickup unit and the position/orientation of the second viewpoint are sequentially followed by changes in the position/orientation of the head or face of the second user.

A transmission/reception system according to the present invention is characterized by including the transmission terminal and the reception terminal.

According to the present invention, it is possible to display an augmented reality of a virtual object instructed to be sharedly displayed at a common position in the real space without requiring a highly accurate position measuring unit or another projection projecting unit. You can

It is a figure which shows the typical embodiment of the information processing terminal of this invention used with the information processing system of this invention. It is a flow chart which shows the flow of judgment and processing of the 1st information processing terminal which constitutes the information processing system of this embodiment. It is a flow chart which shows the flow of judgment and processing of the 2nd information processing terminal which constitutes the information processing system concerning this embodiment. It is a transition diagram showing an example of display transition on the display unit of the first information processing terminal and showing four display states (401) to (404) in the first half. It is a transition diagram showing an example of display transition on the display unit of the first information processing terminal and showing four display states (405) to (408) in the latter half. It is a state diagram which shows typically the use condition of the information processing terminal of this embodiment by a series of procedures, and shows a setting stage (501) and a starting stage (502). It is a state diagram which shows the use condition of the information processing terminal of this embodiment typically by a series of procedures, and shows the stage (503) of sharing instruction|indication, and the projection stage (504) which projects an index. It is a state diagram which shows the use condition of the information processing terminal of this embodiment typically, and shows the stage (505) which became shareable. It is a block diagram which shows the structure of the electronic circuit incorporated in the main body of the information processing terminal of this embodiment.

The transmission/reception terminal (information processing terminal) of the transmission/reception system according to the present embodiment has both a function as a transmission terminal and a function as a reception terminal. However, for convenience of description, the transmission terminal and the reception terminal will be described separately.

The transmission terminal is mounted on the head or face of the first user, and has a projection unit, a first imaging unit, a first display unit, and a transmission unit.
The projection means projects the index image on the physical object based on the index image original data.
The first image capturing unit captures the index image projected on the physical object by the projecting unit to obtain first index image captured data.
The physical object is displayed on the first display unit, and an image of the virtual object viewed from a first viewpoint in the virtual space is displayed in the vicinity of the physical object.
The transmitting means transmits 1) three-dimensional shape data for displaying a virtual object, 2) the index image original data, and 3) the index image captured data.
The position and orientation of the projection unit, the position and orientation of the first image pickup unit, and the position and orientation of the first viewpoint sequentially follow changes in the position and orientation of the head or face of the first user.

The receiving terminal is attached to the head or face of the second user, and has a receiving means, a second imaging means, a second display means, and a calculating means.
The receiving unit receives 1) three-dimensional shape data for displaying a virtual object, 2) the index image original data, and 3) the first index image pick-up data transmitted from the transmitting terminal.
The second image capturing means captures the index image projected on the physical object to obtain second index image captured data.
The real object is displayed on the second display means, and an image of the virtual object viewed from a second viewpoint in the virtual space is displayed near the real object.
The calculation means, based on the index image original data, the first index image captured data, and the second index image captured data,
A relationship between the position and orientation of the index image and the position and orientation of the second imaging unit in the physical space,
The position and orientation of the second viewpoint are determined so that the relationship between the position and orientation of the virtual object and the position and orientation of the second viewpoint in the virtual space are substantially the same.
Then, the position and orientation of the second imaging unit and the position and orientation of the second viewpoint sequentially follow changes in the position and orientation of the head or face of the second user.

Preferred embodiments (examples) of the present invention will be described below with reference to the drawings.
An information processing terminal (transmission/reception terminal) according to this embodiment will be described with reference to FIG. 1, FIG. 2, FIG. 3, FIG.

<External structure of information processing terminal (transmission/reception terminal)>
The information processing terminal (transmission/reception terminal) is configured with a computer having an information processing function as a basic element. In the following description, an example of an HMD (Head Mounted Display) worn on the head of a person will be described as an information processing terminal. In the following description, for convenience of explanation, “HMD” is described in principle instead of “information processing terminal”. Each of the two HMDs includes a mounting means to be mounted on the head of the user, a projection means, an imaging means, a display means, and an information processing means.

1A, 1B, and 1C are external views of the HMD 100. 1A is an external perspective view as seen from the front side, FIG. 1B is an external perspective view as seen from the rear side, and FIG. 1C is a side view in a state of being worn on the head of the user. is there. Generally, the HMD has a so-called goggle type in which the eyepiece is attached and fixed around the user's eye by an expandable and retractable head mounting member, or the entire head of the user is covered. There are various types such as the so-called helmet type, which is designed to be fixed at the jaw. Further, in recent years, in consideration of wearability, a relatively lightweight spectacle frame type in which a frame is hung and fixed on the user's ear has also appeared.
The HMD 100 shown in FIGS. 1A to 1C corresponds to the so-called goggle type described above.

In the HMD 100, the head mounting unit 101 mounts the HMD 100 on the head of the user, and includes a plurality of constituent members denoted by reference numerals 121 to 125 as shown in FIG. 1. To mount the HMD 100 on the head H of the user as shown in FIG. 1C, first, the adjuster 122 covers the head H with the length adjusting portion 123 loosened. Then, the length adjusting unit 123 is squeezed by the adjuster 122 so that the forehead mounting unit 125 is brought into close contact with the forehead of the head H and then the temporal mounting unit 121 and the occipital mounting unit 124 are respectively brought into close contact with the temporal region and the occipital region. ..

The image pickup unit 102 is a so-called digital camera, and is composed of a plurality of members denoted by reference numerals 112, 116, and 117 as shown in FIG. 1C, and has a direction in which a user wearing the HMD 100 on the head H faces. Images are taken in substantially the same direction and sequentially follow changes in the position and orientation of the head of the user. Specifically, the light incident from the outside of the HMD 100 through the incident window 112 is guided to the inside of the HMD 100 by the optical prism 117, and is received by the image sensor 116. The members 112, 116, 117 each have elements corresponding to the left eye and the right eye, and in the figure, they are distinguished by adding symbols “L” and “R”, respectively.

The projection unit 103 is a so-called projector, and includes a projection lens, an image display element, a light emitting element, and the like.
The display unit 104 is composed of a plurality of members numbered 110, 114, and 115 as shown in FIGS. The symbols "L" and "R" indicate the left eye and right eye as described above. It is provided at a position corresponding to a spectacle lens in spectacles so as to face the position of the user's eyes. Specifically, the image displayed by the color liquid crystal display 114 is guided by the optical prism 115, and the screen 110 is displayed. To display.

Here, there are a glass see-through (video see-through) system and a video see-through system as a system for superimposing and displaying graphics on a scene in the real world. The glass see-through method projects graphics using a transparent or translucent spectacle lens as a screen, and optically superimposes the graphics on a real world scene. The video see-through method electrically superimposes graphics on a real-world scene imaged by an imaging device.

When the HMD 100 is the above-described video see-through type, the screen 110 does not have transparency in principle and displays the guided image as it is.
On the other hand, when the HMD 100 is the above-mentioned glass see-through type, the screen 110 is composed of a so-called half mirror. That is, it has a certain degree of transparency so that the user can optically see the scene in the real space. At the same time, the image of the graphics displayed by the color liquid crystal display 114 and guided by the optical prism 115 is reflected in the direction of the user's eye by the mirror provided on the front surface, the back surface, or inside. That is, in the case of the glass see-through type, the scene of the physical space and the image of the virtual object (virtual object) are optically superimposed on the screen 110.

The output light from the optical prism 115 of the display unit 104 and the input light to the optical prism 117 of the image pickup unit 102 match the optical axis of the user's pupil. A physical space image viewed according to a position and a direction (position and orientation) is captured.
In the case of the video see-through HMD 100, the color liquid crystal display 114 displays an augmented reality image in which the physical space image acquired by the image sensor 116 and the image of the virtual object are electrically superimposed (image combination).
On the other hand, in the case of the glass see-through type HMD 100, the color liquid crystal display 114 basically displays only the image of the virtual object.

<Internal (electronic circuit) configuration of information processing terminal>
As shown in FIG. 6, the HMD 100 further includes a display control unit 130, an imaging control unit 140, a CPU 150, a memory 160, a power supply unit 170, a communication unit 180, and a projection control unit 190 as electronic circuit elements. These electronic circuit elements are embedded inside the main body of the HMD 100 (including the head mounting portion 101) and are not shown in appearance.
In the above, the CPU 150, the memory 160, the display control unit 130, the imaging control unit 140, and the projection control unit 190 form the information processing unit of the HMD 100.

The display control unit 130 controls the display of the display unit 104. For example, the size, position, direction, color, transparency, and movement of the virtual object to be superimposed and displayed (image combination) on the real space image, and movement and change in brightness associated with the change of the real space image are controlled.

The image pickup control unit 140 performs exposure control, distance measurement control, and the like based on the calculation result of a predetermined calculation process using the image pickup data. As a result, AF (auto focus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and the like are performed. When the image pickup unit 102 includes a mechanism for inserting and removing the optical filter from the optical path, a vibration isolation mechanism, and the like, the filter insertion and removal, the image stabilization, and the like are controlled according to the imaging data and other conditions.

The CPU 150 performs arithmetic processing of the entire HMD 100. By executing the program recorded in the memory 160, which will be described later, each processing of the present embodiment is realized.
The memory 160 includes a work area and a non-volatile area. In the work area, a program read from the non-volatile area and constants and variables for system control are expanded. In addition, the data of the image of the virtual object displayed in a superimposed manner in the physical space is held for display. In addition, the imaged data that is imaged by the imaging unit 102 and A/D converted is developed for the purpose of image analysis, image processing, and the like.

The power supply unit 170 includes a primary battery such as an alkaline battery and a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, and a Li battery, an AC adapter, and the like, and supplies power to the entire HMD 100. A power switch for switching the power on and off according to user operation and other conditions is provided.
Under the control of the CPU 150, the communication unit 180 communicates with another HMD 100 according to a standard such as WiFiDirect, and transmits/receives three-dimensional shape data, position/orientation data, and the like.
The projection control unit 190 controls the projection operation of the projection unit 103.

<Flow of information processing terminal judgment and processing>
FIG. 2 shows a flow of judgments and processes executed by the first HMD 100 having the above structure, and FIG. 3 shows a flow of judgments and processes executed by the second HMD 100 (reception terminal) having the above structure. There is. The determination and the processing are realized by the CPU 150 expanding the program recorded in the nonvolatile area of the memory 160 in the work area and executing the program.

<Flow of judgment/processing of first information processing terminal>
First, with reference to FIG. 2, a flow of determination/processing of the first HMD 100 (transmission terminal) relating to the use of a user (user) who gives an instruction to share data will be described.
In step S201, the first HMD 100 starts image pickup in the physical space by the image pickup unit 102 when the user who uses the HMD 100 turns on the power.

In step S202, the image of the virtual object is displayed on the display unit 104 by being superimposed on the captured image of the physical space under predetermined display conditions (size, position, orientation, color, transparency, etc.). Depending on the specifications of the information processing terminal, the virtual graphics displayed immediately after the power is turned on are, for example, graphics (stereoscopic icons) that symbolically represent the storage area of the nonvolatile area related to the management of the user of the HMD 100. Good. Further, the graphics displayed before the power is turned off may be recovered and used.

In step S203, the first HMD 100 determines whether any of the virtual graphics displayed by the user has been instructed/operated. In this case, the virtual graphics displayed by being superimposed on the image in the physical space corresponds to a so-called graphical user interface (hereinafter referred to as “GUI”).
For example, if a stereoscopic icon (storage icon) that symbolically represents a storage area related to user management is displayed, and if "open" is instructed by the user's gesture operation, etc., it is stored in that storage area. Display graphics that represent files, etc. Further, for example, when an icon that symbolically represents the design data of “shoes” is displayed, and when “open” is instructed by a user's gesture operation or the like, the stereo model of the “shoes” is changed to the real space. It is displayed by being superimposed on the image.

By the way, when virtual graphics are superposed on an image in the real space, the virtual graphics are displayed in association with the position coordinates in the real space, that is, when the so-called augmented reality display is performed and the position coordinates in the real space are not associated. There is a case where it is displayed in (when the augmented reality display is not performed). The case in which the augmented reality display is performed and the case in which the augmented reality display is not performed are based on a predetermined condition. The setting of the condition of whether or not this augmented reality display is performed will be described later.

When the user gives an instruction/operation to the virtual graphics (YES in step S203), it is determined whether the instruction/operation is an instruction/operation to “share” the data indicated by the graphics ( Step S204). If the instruction/operation is other than “share”, the content of the instruction is executed (step S205).

In step S209, when the user's instruction/operation is this “share” instruction/operation, first, the projection unit 103 displays the virtual graphics whose sharing is instructed and the display position where the virtual graphics are displayed. The index image is projected in the same direction on the physical space. However, at this time, as shown in steps S206 to S209 below, the index image is so-called so that the index image is similar to that projected in the vertical or horizontal direction based on the projection direction angle detected by the angle sensor. Correct the keystone.

For example, when the HMD 100 projects the index image toward a substantially vertical surface such as a wall surface in a direction that forms an angle θ with respect to the horizontal direction, the direction along the projection direction of the index image (from the HMD 100 The so-called up-down direction when viewed and the upper side with respect to the lower side when viewed from the HMD 100 are respectively transformed to 1/cos θ times. Therefore, if the original image is corrected to be multiplied by cos θ, it becomes the same as that projected horizontally regardless of the angle of the information processing terminal.

Further, when the HMD 100 projects the index image toward a substantially horizontal surface such as a table surface in a direction forming an angle θ with respect to the horizontal direction, a direction along the projection direction of the index image (HMD100 The so-called up-down direction when viewed from above and the upper side with respect to the lower side when viewed from the HMD 100 are respectively transformed by −1/sin θ times. Therefore, if the original image is corrected by -sin θ times, it becomes the same as that projected vertically regardless of the angle of the HMD 100.

For example, when the attitude (angle θ with respect to the horizontal direction) of the HMD 100 is −45°≦θ≦45°, it can be estimated that the image is projected toward a substantially vertical plane.
Further, when the attitude (angle θ with respect to the horizontal direction) of the HMD 100 is θ≦−45° or θ≧45°, it can be estimated that the image is projected onto a substantially horizontal surface.

That is, in step S206, the angle of the projection direction is acquired by the angle sensor, and in step S207, it is estimated which of the substantially vertical surface or the substantially horizontal surface the image should be projected.
In step S208, so-called trapezoidal correction is performed on the index image depending on whether the projection surface is a substantially vertical surface or a substantially horizontal surface.
In step S209, the corrected index image is projected on the projection surface by the projection unit 103.

Next, in step S210, the image of the projected index image is captured by the image capturing unit 102. Since the original image of this index image has been subjected to the trapezoidal correction similar to that projected in the vertical or horizontal direction by the process described above, the inclination of the captured index image depends on the imaging direction of the HMD 100 and the projection direction. It is a reflection of the inclination with respect to the surface. The size of the captured index image reflects the distance between the HMD 100 and the projection surface by performing correction in consideration of the spread angle of the projection image and the magnification of the imaging lens (so-called zoom ratio). Become.

Next, in step S211, the position, orientation (position/orientation), and size of the virtual graphics for which sharing is instructed are changed based on the picked-up image of the index image and superimposed and displayed. By doing so, it is possible to perform augmented reality display as if the virtual object is placed on or adhered to the real object on which the index projection is performed.

In the above description, the projection direction of the index image by the projection unit 103 and the imaging direction of the index image by the imaging unit 102 in the first HMD 100 match the viewpoint (position/orientation) of the first user. In the first HMD 100, the inclination and the like of the projection surface on which the index image is projected is specified based on the index image original data used for projection and the index image imaging data obtained by imaging the projected index image. .. Then, the tilt and the like are reflected in the position and orientation of the virtual object for which “sharing” is instructed, and the virtual object is superimposed and displayed in the vicinity of the projection surface to perform augmented reality display.

Note that the index image may be projected by a light ray visible to the human eye or an invisible light ray (for example, infrared ray) as long as the image pickup unit 102 of the HMD 100 can image it. Alternatively, instead of continuous projection, for example, projection may be discontinuous at regular time intervals.
As a result, for example, a pattern is drawn on a real object (projection surface), and it is difficult to distinguish the projected index image and the pattern on the projection surface by continuous projection with visible light. Even if the recognition is difficult, the index image can be easily recognized. By removing the influence of patterns, etc. by projection/recognition with infrared light, or by projecting discontinuously at regular time intervals and extracting the projected image from the difference between the captured images before and after projection, the index image It becomes possible to facilitate the recognition.

Next, in step S212, the communication unit 180 of the first HMD 100 causes the information (three-dimensional shape data, position and orientation data, etc.) related to the virtual object and the information related to the index image (index image original data, index image capturing). Data) and send.
In addition, in step S213, a display indicating that “shared setting is in progress” is displayed on the display unit 104. This display may be, for example, a character such as "sharing is set", or the graphics of the virtual object may be displayed in a different color or transparency, or may be displayed by blinking. It may be one.

Then, it waits until it receives a confirmation signal transmitted from another HMD 100 (second HMD 100) (step S214). The confirmation signal will be described later.
However, a predetermined ending condition is provided, and when the condition is satisfied (for example, when a predetermined time has elapsed), the standby is ended (step S215). When the confirmation signal is received, the display of "setting for sharing" is erased (step S216).

<Flow of determination/processing of second information processing terminal>
The flow of determination/processing of the second HMD 100 (reception terminal) will be described with reference to FIG.
In step S301, the second HMD 100 starts image pickup in the physical space by the image pickup unit 102 when the power is turned on by the second user who uses the second HMD 100.
Then, in step S302, the second HMD 100 transmits the information (three-dimensional shape data, position/orientation data) related to the virtual object for which sharing is instructed and the information related to the index image (index) transmitted from the other first HMD100. It waits until it receives (image original data, index image captured data).
However, a predetermined ending condition is provided, and when the condition is satisfied (for example, when the user performs a predetermined operation), the standby is ended (step S303).

In step S304, when the second HMD 100 receives the information related to the virtual object for which sharing is instructed and the information related to the index image, the "information" is displayed on the display unit (second display unit) of the second HMD 100. Is received" is displayed to the second user. At this time, if the position of the first HMD 100 can be acquired, a display prompting the user to face the direction may be displayed.

Then, in step S305, it is determined whether or not the index image projected by the projection unit of the first HMD is included in the images captured by the imaging unit (second imaging unit) of the second HMD.
In step S306, when there is an index image, first, a "confirmation signal" to that effect is transmitted to the first HMD 100. This is to notify the first HYMD 100 (and its user) that the virtual object has been shared. The image capturing direction of the image capturing unit 102 of the second HMD 100 matches the viewpoint and line of sight of the user who uses the image capturing unit 102.

In step S307, the position, size, and orientation (direction and inclination) of the index image in the captured image are acquired.
For example, if the pattern of the index image is neither point-symmetrical nor line-symmetrical, by comparing the received index-image original data with the first index-image captured data, the physical object (projection surface) onto which the index image is projected is projected. ) And the position and orientation of the first imaging unit with respect to the projection surface can be specified.
Further, by comparing the received index image original data with the second index image captured data, the position and orientation of the second image capturing means with respect to the projection plane can be specified.
Then, if the relationship among the position and orientation of the first image capturing unit, the position and orientation of the projection surface, and the position and orientation of the second image capturing unit is specified,
The relationship between the position and orientation of the first viewpoint in the virtual space, the position and orientation of the virtual object superimposed and displayed on or near the projection surface, and the position and orientation of the second viewpoint in the virtual space is specified. be able to.

Then, in step S308, rendering is performed based on information (shape data, position/orientation data) related to the virtual object for which sharing is instructed.
In step S<b>309, the image of virtual information (image of virtual graphics) obtained by the rendering is superimposed on the image of the physical space captured by the image capturing unit 102 and displayed on the display unit 104.
In this way, the virtual object whose position and orientation are specified and which is superimposed and displayed on or near the projection surface (for example, a virtual object which is located approximately 1 cm above the projection surface and whose right side is visible from the first viewpoint). The image viewed from the second viewpoint in the virtual space is displayed on the second display unit 104.
For example, in the case of the positional relationship as shown in FIG. 5C, the virtual object (shoes) that is placed about 1 cm above the projection surface and is viewed from the “right” side is displayed on the display unit of the user P3, and the user P1 is displayed. The virtual object, which is arranged approximately 1 cm above the projection surface and is viewed from the "left" side surface, is displayed on the display unit of.

Through steps S307, S308, and S309 described above, sharing of the virtual object (virtual object) is executed in the second HMD 100 via the index image projected by the first HMD 100. The steps S307, S308, and S309 compose step S311 for sharing processing, and realize a sharing means. As a result, the predetermined virtual object displayed on the first HMD 100 (step S211) is displayed on the second HMD 100 based on the viewpoint, and sharing is realized.

After the second HMD 100 superimposes and displays the image of the virtual information related to the virtual object for which sharing is instructed, the second HMD 100 displays the image of the virtual information in an augmented reality display as if the image actually exists at a specific position in the physical space. (Step S310). That is, when the second HMD 100 recognizes that the second HMD 100 approaches the specific position by recognizing the feature points of the captured image obtained by capturing the real space, the captured image is re-rendered so that the graphics of the object become large. Superimpose on.
Further, if the above-mentioned specific position deviates from the field of view (that is, the captured image) of the user, the graphics of the virtual object is once deleted. However, when the specific position comes into the user's field of view again, the graphics of the virtual object are displayed again as if they actually exist at that position based on the information of the characteristic points stored in the memory 160.

<Operations and effects of the embodiment>
In summary, when the user (user) instructs the first HMD 100 to “share” the virtual object displayed in augmented reality, the direction in which the virtual reality object is displayed in augmented reality is displayed. The index image is projected toward. The second HMD 100 captures the projected index image, and displays the virtual object in an augmented reality based on the index image and the transmitted information about the virtual object instructed to be shared and the information about the index image. To do.
The above-mentioned index image in the captured image of the second HMD 100 is of course viewed from the viewpoint of the second HMD 100, and also reflects the inclination of the projected surface of the real object (eg, “table”). It is a thing.

That is, the second HMD 100 generates graphics of a virtual object having a position and orientation corresponding to the position, size, and orientation (direction and inclination) of the index image in the captured image, and displays the augmented reality on its display unit. As a result, the first HMD 100 and the second HMD 100 can be displayed on the respective display units 104 from the respective viewpoints, as if the same object were at the same position in the real space. ..
Therefore, for example, when the first user points the virtual object by a physical object such as a finger, the virtual object and the physical object are displayed in a superimposed manner on the second display unit, and the second user displays the first object. User appears to be pointing to the virtual object. Therefore, an augmented reality display suitable for the first user and the second user to “share” the virtual object is possible.

At this time, the position and orientation of the first viewpoint when drawing the virtual object displayed on the first display unit, and the position of the second viewpoint when drawing the virtual object displayed on the second display unit The relationship with the viewpoint substantially matches the relationship between the position and orientation of the first imaging unit and the position and orientation of the second imaging unit.
Although it is possible to make a similar appearance by using a so-called geotag, the information processing terminal of the present invention (for example, the HMD 100 described in the above embodiment) adds the latitude/longitude information to the virtual object in advance. No need. Also, it is not necessary to acquire the latitude/longitude information of each terminal.
In the system using the information processing terminal of the present invention, a physical object capable of projecting an index image exists in the physical space in which the first user and the second user exist, and is projected from the first terminal onto the projection surface. It suffices that each of the first terminal and the second terminal can capture the index image.
By capturing the common index image from the respective positions and orientations of the first terminal and the second terminal, the position and orientation of the first imaging unit in the physical space, the position and orientation of the projection surface, and the second orientation A relative relationship (first relative relationship) with the position and orientation of the imaging unit can be specified.
The relative relationship (second relative relationship) between the position and orientation of the viewpoint of the first terminal, the position and orientation of the virtual object, and the position and orientation of the viewpoint of the second terminal in the virtual space is the first. The position/orientation of the viewpoint of the second terminal is specified so as to be approximately the same as the relative relationship.
As a result, the virtual object displayed in the vicinity of the projection surface is displayed on the first display means as an image viewed from the first viewpoint, and the image displayed on the same virtual object as viewed from the second viewpoint is displayed as the second display. It has the effect of the invention in that it can be displayed on the means and a shared augmented reality display is possible.

<Display transition on the display unit of the first information processing terminal>
Next, with reference to FIGS. 4A and 4B, an operation example of the first HMD 100 that projects an index image when a user gives an instruction to share an image of virtual information will be described. 4A and 4B, the display transition on the display unit 104 of the first HMD 100 is shown by a series of display states (401) to (408). The four display states (401) to (404) in the first half are shown in FIG. 4A, and the four display states (405) to (408) in the latter half are shown in FIG. 4B.

The display state (401) shows an example of display immediately after the power is turned on in the first HMD 100. In the display state (401), reference numeral 4011 indicates a captured image of the physical space captured by the image capturing unit 102. Reference numeral 4012 is a virtual graphic superimposed on the captured image 4011, and here, a storage icon representing a storage area related to the management of the user of the first HMD 100 is displayed.

The display state (402) represents a state in which the user is instructing/instructing the storage icon 4012 by a “finger pinch” gesture. The user's hand 4023 is a part of the captured image in the physical space captured by the image capturing unit 102. Here, with respect to the virtual graphics 4022, the instep side (the side with the nail) 4024 of the user's thumb is displayed in the front, and the ventral side (the side without the nail) 4025 of the user's index finger is displayed in the back. Display control is done.
This is performed in order to give a user a visual sensation of pinching graphics with a finger, and an image for identifying and extracting the instep side and the ventral side of a human finger from the captured image. It is based on processing technology. Such an image processing technique is already known as disclosed in, for example, JP 2012-53674 A.

The display state (403) represents a state in which "display the contents of the storage (so-called "open")" is instructed to the storage icon 4031 by a predetermined gesture operation of the user. Here, the predetermined gesture operation is a gesture of “pinching the icon with a finger and then tampering with the thumb and forefinger”. Since this gesture is common to the operation of opening the bundled cards in a fan shape, it is easy to recall the word “open” and is suitable as a gesture for instructing so-called “open”.
Of course, there is no problem even if another gesture is assigned as the so-called "open" gesture.

The display state (404) indicates that the lower level hierarchy of the storage area represented by the storage icon 4031 is as a result of the user's instruction to open the storage icon 4031 in the display state (403). It shows the displayed state. In this display state (404), the icons 4041 to 4048 are icons that respectively display data or files stored in the storage unit of the first HMD 100. For example, the icon 4048 is for displaying one file of application software for displaying a stereo model on the display unit 104.

The display state (405) shown in FIG. 4B represents a state in which "open (starts application software) to open file" is instructed to the file icon 4052 by a predetermined gesture operation by the user. Here, the predetermined gesture operation is the same gesture as that performed in the display state (403), that is, "the icon is pinched with a finger, and then the thumb and the index finger are put on and misled".

In the display state (406), the application software that displays the stereo model is activated, the file indicated by the icon 4052 is expanded in the memory 160, and the “shoe” object 4062 is displayed by being superimposed on the captured image 4061 in the physical space. It shows the state of being.

The display state (407) represents a state in which the user makes a gesture of “picking up an object with a finger and then encrusting with a thumb and forefinger” on the stereo model 4062 in the display state (406). ing. When this gesture is made, a menu of commands that the user can give to the object according to the gesture is displayed. The commands include "COPY", "CUT", "EDIT", and "SHARE". After displaying the menu, the user can further move his/her thumb to instruct/select one command.

In the display state (408), as a result of the user selecting the command "SHARE" and instructing "sharing" of the object, the state in which the augmented reality display in which this object is associated with the position coordinates in the physical space is displayed. Is represented. That is, in the figure, the captured image 4081 is a captured image of the real world, and the virtual object 4082 is a virtual object displayed in augmented reality.
For example, by displaying a shadow below the virtual object, the virtual object "shoes" is displayed in augmented reality as if it were placed on the "table" of the real object. It is difficult to distinguish between virtual and real.

When transitioning from the display state (407) to the display state (408), the display of the virtual object is changed from so-called normal superimposed display to augmented reality display. In this way, the display mode is changed so that the object showing the shared file is displayed in the augmented reality display, and the object showing the file not shared is not the augmented reality display, that is, the so-called normal superimposed display is performed. This is because the display condition was set. The reason why the display mode relating to the augmented reality or the display mode relating to the non-augmented reality is made different depending on whether the file is shared or non-shared is as follows.

That is, when a plurality of users (users) gather in a specific place for a meeting or a collaboration, if the augmented reality display is performed as if the object were actually there, the effect of the meeting or the collaboration is obtained. And efficiency is improved. This is because, for example, when one user gives a presentation while pointing to a virtual object, other users can simultaneously see the virtual object and the presenter who points to the virtual object.

On the other hand, if the object to be displayed is an object related to a "non-shared" file that is not shared with other users, non-augmented reality display prevents the object display from being related to the position coordinates in the real space. .. Therefore, for example, when the user does not face the direction of the position coordinates, the object is not displayed.
That is, it is rational to change the display mode such that the objects related to the “non-shared” file are not displayed in augmented reality and the objects related to the “shared” file are displayed in augmented reality.

<A series of procedures based on the usage states of the first and second information processing terminals>
Next, with reference to FIGS. 5A, 5B, and 5C, an example of a series of procedures for “sharing” virtual objects using the above-described first and second HMDs 100 will be described. In this example, a plurality of users wearing the HMD 100 on their heads gather at a specific place for a meeting or the like. In such a case, one user may instruct to “share” a virtual object, and another user may display the virtual object in an augmented reality display in association with common position coordinates from their respective viewpoints. It is a schematic representation.

FIG. 5A shows a setting step (501) and a starting step (502) in the above series of procedures. FIG. 5B shows a display mode viewed from the first HMD 100 and shows a sharing instruction step (503) and a projection step (504) for projecting an index from the first HMD 100. FIG. 5C shows a state (505) in which the virtual object can be shared with the second HMD 100 based on the index projected from the first HMD 100.

In the setting stage (501), the users P1 and P2 wearing the above-mentioned HMD 100 respectively perform user group setting by a predetermined method. Here, the gesture of the users P1 and P2 wearing the HMD 100 shaking hands is a group setting method. The use of this gesture as a group setting method has the following significance.

First, "handshake" is a natural act before the meeting. Second, compared to other gestures (for example, "bow"), it is possible to reliably identify whether or not the gesture is performed, and thus it is possible to prevent the object from being shared carelessly. Third, by setting a group based on the fact that a handshake was actually performed (not only the appearance as if a handshake had been made, but also the actual physical contact), unintended persons are grouped. It is possible to set a highly safe group without the risk of adding it.
In order to realize the third meaning, each of the HMDs 100 needs to be provided with means for detecting body contact between the wearers of the HMDs 100, for example, by a change in the electric field state. The technique for detecting body contact based on the change in the electric field state is a known technique as disclosed in, for example, Japanese Patent Laid-Open No. 2010-193129.

The start stage (502) shows that the users P1, P2, and P3 of the three HMDs 100 who are set as a group surround the table 5021 of real objects. In this state, since there are no virtual objects related to sharing by the three people, the three HMDs 100 display different virtual objects.

The stage (503) is a display mode viewed from the user (wearer) P3 of one HMD 100. Here, the virtual object 5031 is displayed only for this user P3. At the stage (503), a gesture operation for instructing “sharing” of the virtual object is performed by the finger 5032 of the user P3.

The projection step (504) represents a state in which the index image 5041 is projected on the table 5021 by the HMD 100 equipped to the user P3.

In the sharing step (505), the second HMDs 100 of the other users P1 and P2 recognize the index images 5041, respectively, and based on that, the images from the respective viewpoints of the virtual objects 5051 related to this sharing are displayed on the respective display units. Is schematically shown as being displayed in augmented reality. That is, the virtual object 5051 of the shoe in the sharing stage (505) is not displayed (not visible) except for the users (wearers) P1 to P3 of the HMD 100.
However, the virtual objects 5051 from the viewpoint of each user are displayed on each display means of the users (wearers) P1 to P3 of the HMD 100 as if they were actually placed on the table 5021. This is schematically shown.

For the users P1 to P3 belonging to the group, the virtual object 5051 is commonly displayed as if it actually exists, which is convenient for meetings and collaborations within the group. For example, in order to explain the design of a "shoe", if one of the users changes the direction of the "shoe" object in various directions and points with his/her finger, he/she adds "the shape here..." , The object is displayed to other users with the pointed finger.
Therefore, the contents of the explanation are easy to understand. Alternatively, if another user gives an opinion while pointing with a finger or the like, "This is the one here...", it is clear which part of the opinion it is. Further, when the user of the group can edit/modify the virtual object 5051, the virtual object 5051 can be used, for example, to proceed with the meeting while changing the color or the pattern.

Note that how the user of the group can operate the object set to “shared” can be the same as the concept of “shared” of the HMD 100 in general. That is, it is based on the setting of the “authority” of the group user when the group is set. For example, authority such as “view only” or “modify/overwrite” is preset for the group user. Regarding when to cancel the group, the condition for canceling the group can be set not only when the cancel operation is performed, but also when the HMDs 100 are separated from each other by a predetermined distance or more.

In summary, the information processing terminal (first HMD 100) used by the user who gives the instruction to share the virtual information image projects the index image when the sharing instruction is given. As described above, since the other information processing terminal (second HMD 100) displays the image of the virtual information related to the sharing instruction based on the projected index image, the other information is not provided until the sharing instruction is given. The processing terminal can display the image of the virtual information. In other words, when there is no sharing instruction, the image of the virtual information is not displayed on the other information processing terminal, so that the information can be managed with high security.

<Other Embodiments>
In this embodiment, an example of the HMD 100 that transmits a predetermined confirmation signal when an index image related to a virtual information image sharing instruction is captured will be described.
As described in step S306 of FIG. 3, when the HMD 100 according to the present embodiment captures the index image, the HMD 100 transmits a “confirmation signal” to that effect to the HMD 100 that is projecting and displaying the index image.

The index image projection unit that projects the index image is a part of the HMD 100 according to the present embodiment, and is mounted on the head of the user by the head mounting unit. Therefore, until another HMD 100 "confirms" the index image, that is, until the index image is captured by the image capturing means of the other HMD 100, the user of the HMD 100 projecting the index image moves his/her head greatly. Don't However, it is rational that after the index image is “confirmed” by the other HMD 100, the fact is promptly notified and the user can recognize that the head may be moved any more.

The HMD 100 according to the present embodiment transmits the predetermined confirmation signal when the index image related to the sharing instruction of the image of the virtual information is captured, for this notification, and the confirmation signal is used as follows. can do.

For example, the HMD 100 that projects the index image may be displayed so that the user does not move his/her head until the “confirmation signal” is received after the HMD 100 projects the index image.
Alternatively, when there are three or more HMDs 100 set as a group, it can be inferred that the projection state of the index image is not appropriate when the “confirmation signal” is not received from many of them. Therefore, the HMD 100 that projects the index image may be displayed to the user so as to improve the projection state.
On the contrary, when there are 3 or more HMDs 100 set as a group, when the "confirmation signal" is received from most of them and the "confirmation signal" is not received from a small part, the indicator of the HMD 100 that does not transmit the "confirmation signal" It can be inferred that the image capturing state is not appropriate. In this case, the display may be displayed so as to improve the imaging state for a small number of users of the HMD 100.

To summarize the above, the HMD 100 used by the user who shares the image of virtual information (the side that captures the index image) transmits a predetermined confirmation signal when the index image is captured. Based on whether the number of the confirmation signals is large or small with respect to the total number of HMDs 100, it is possible to perform a display prompting the user who projects the index image or the user who captures the index image to use appropriately. Become.

As described above, according to the information processing terminal (HMD 100 or the like) of the present invention, based on the index image projected by one information processing terminal on the actual object, the other information processing terminals are instructed to share the virtual image. The object can be displayed on the display unit in augmented reality. Therefore, it is possible to display augmented reality, which is optimal for "sharing" data and files.

<Other embodiments>
The present invention is also realized by executing the following processing. That is, a computer program that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program. Processing.
The configurations, shapes, sizes, and positional relationships described in the above embodiments are merely schematic ones so that the present invention can be understood and implemented. Therefore, the present invention is not limited to the embodiments described above, but can be modified into various forms without departing from the scope of the technical idea shown in the claims.

100 HMD (information processing terminal)
101 Head Mount Unit 102 Imaging Unit 103 Projection Unit 104 Display Unit 130 Display Control Unit 140 Imaging Control Unit 150 CPU
160 memory 170 power supply unit 180 communication unit 190 projection control unit 4082 virtual object 5021 table 5031 virtual object 5041 index image 5051 virtual object

Claims (8)

A transmission terminal that is attached to the head or face of a first wearer and that transmits three-dimensional shape data for displaying a virtual object, and the head or face of another wearer, and the virtual object. A transmission /reception system including one or more receiving terminals capable of receiving three-dimensional shape data for displaying from the transmitting terminal ,
The transmitting terminal is
Selecting means for selecting whether to share the virtual object;
First display means for displaying a real object, and displaying an image of a virtual object that can be selected or shared by the selecting means in the vicinity of the real object, as viewed from a first viewpoint in a virtual space. When,
If there is a predetermined type of physical contact between the first wearer and a second wearer wearing one of the one or more receiving terminals on the head or face, then the first wearer Group setting means for setting the wearer and the second wearer as belonging to a group of wearers sharing the virtual object,
Each of the one or more receiving terminals has a second display unit capable of displaying an image including the physical object and a virtual object,
Among the one or more receiving terminals, the receiving terminal worn by the second wearer set to belong to the group of wearers sharing the virtual object,
In the second display means, the real object is displayed, and an image of the virtual object selected to be shared by the selection means in the vicinity of the real object is viewed from a second viewpoint in the virtual space. Is displayed,
Among the one or more receiving terminals, a receiving terminal not set to belong to a group of wearers sharing the virtual object,
The transmitting/receiving system , wherein the virtual object selected to be shared by the selecting means is not displayed on the second display means . The transmitting terminal is
The transmission/reception system according to claim 1, further comprising a group cancellation unit that cancels a group related to sharing of the virtual object based on a distance between the transmission terminal and the reception terminal. The transmission/reception system according to claim 1, wherein the predetermined type of physical contact is a gesture of shaking hands. Based on the imaged image of the physical space imaged by the imaging unit and the detection result of the means for detecting the body contact between the wearers, the predetermined distance between the first wearer and the second wearer. 4. The transmission/reception system according to claim 1, further comprising a determination unit that determines whether or not there has been a body contact of the form. The transmitting terminal is
Projection means for projecting the index image on the physical object based on the index image original data,
A transmitting unit that transmits the three-dimensional shape data and the index image original data,
The transmission/reception system according to claim 1, wherein the projection unit projects the index image when the selection unit selects the sharing. In the transmitting terminal,
The transmission/reception according to claim 5, wherein the position/orientation of the projection unit and the position/orientation of the first viewpoint sequentially follow changes in the position/orientation of the head or face of the first wearer. system. The transmitting terminal is
7. The transmission/reception system according to claim 5, further comprising a first imaging unit that captures the first index image captured data by capturing the index image projected by the projection unit. A computer program for causing a computer to function as each unit of the transmission/reception system according to claim 1.

Images