WO2022172335A1 - Virtual guide display device, virtual guide display system, and virtual guide display method - Google Patents

Abstract

This virtual guide display device uses sensor output from a direction sensor for detecting the direction the virtual guide display device is facing in the world coordinate system and from a position sensor for detecting the position of the virtual guide display device in the world coordinate system as a basis to generate a virtual guide extending in the same direction as a first viewpoint from the virtual guide display device via a position specified in a first visual field image viewed from the first viewpoint, superposes the virtual guide on a second visual field image viewed from a second viewpoint differing from the first viewpoint and displays the result on a display, receives an operation specifying a position on the virtual guide displayed on the display, converts the specified position to a position in the world coordinate system, and outputs the result.

Description

VIRTUAL GUIDE DISPLAY DEVICE, VIRTUAL GUIDE DISPLAY SYSTEM, AND VIRTUAL GUIDE DISPLAY METHOD

The present invention relates to a virtual guide display device, a virtual guide display system, and a virtual guide display method for specifying a position in a three-dimensional space.

In recent years, Augmented Reality (AR) technology, which adds digital information to the real world and reflects and augments virtual objects (virtual objects) in the virtual space created with CG (Computer Graphics), etc., has become widespread. 2. Description of the Related Art Virtual guide display devices and virtual guide display systems that can easily handle virtual objects while recognizing a three-dimensional real space are widely used. As an example of its use, applications such as remote work support are also expanding, so there are more opportunities than ever for users who are not familiar with the device to specify a position in a three-dimensional space. Improvement is desired. For example, a worker takes a picture of the work place with a camera such as a head-mounted display, and a support instructor who is in a remote location away from the work place looks at the picture of the work place on a display, etc. When specifying and giving work instructions, it is difficult for a support instructor who is unfamiliar with the apparatus to accurately specify the position of the target object. In addition, even when the user himself/herself takes a picture with a camera and, while looking at the photographed image, designates the position of the target object on the photographed image and, for example, informs other nearby users of the position of the target object, the user must be familiar with the apparatus. For a user who does not have this function, it is difficult to perform a position specifying operation. Position specification in a three-dimensional space is mainly performed when specifying an object (physical object or virtual object) that already exists in the space, or arranging a virtual object at a specified position.

Here, as a method for specifying a virtual object already existing in a three-dimensional space, Patent Document 1 discloses "display means for displaying an image of a virtual space in which virtual object information is common among a plurality of virtual guide display devices. An information processing system comprising a plurality of virtual guide display devices communicably connected to a head-mounted display, which acquires the position and orientation of the head-mounted display in the real space, and based on the acquired position and orientation information , to specify a position in a virtual space, and an information processing system that controls to display an arrow from a head-mounted display to the specified position (summary excerpt)." That is, by displaying the line of sight of the first user through a head mounted display (HMD) as a virtual object on the HMD of the second user, the second user can identify the object ahead of the line of sight. is shown.

JP 2017-33575 A

In Patent Document 1, the position of the place where the line of sight hits is specified from one direction. There is no consideration for specifying a different position on the back side, and there is a problem that an arbitrary position on the three-dimensional space cannot be specified.

In view of the above problems, the present invention provides a virtual guide display device, a virtual guide display system, and a virtual guide display method that can easily and accurately designate an arbitrary position in a three-dimensional space.

In order to solve the above problems, the present invention has the configuration described in the claims. As an example, the present invention is a virtual guide display device comprising: a direction sensor for detecting a direction in the world coordinate system to which the virtual guide display device is directed; a position sensor that detects a position; a display; an operation input device that receives an operation to specify an arbitrary point on an image displayed on the display; and a processor connected to each of said processors, based on sensor outputs from said direction sensor and said position sensor, on a first field of view image viewed from a first viewpoint from said virtual guide display device. generating a virtual guide extending in the same direction as the first viewpoint via the specified position of, and displaying the virtual guide on a second view image viewed from a second viewpoint different from the first viewpoint is superimposed and displayed on the display, the operation input device receives an operation to specify a position on the virtual guide displayed on the display, and the specified position is converted to a position in the world coordinate system output.

According to the present invention, it is possible to provide a virtual guide display device, a virtual guide display system, and a virtual guide display method that can easily and accurately designate any position in a three-dimensional space. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

FIG. 2 is a diagram schematically showing the appearance of a configuration example of a virtual guide display device and a virtual guide display system according to the present embodiment; FIG. 2 is a diagram for explaining a position designation operation on a field-of-view image from a first viewpoint in the embodiment shown in FIG. 1; FIG. 4 is a diagram showing a screen when a position is specified on the field-of-view image from a second viewpoint in the embodiment shown in FIG. 1; FIG. 4 is a diagram for explaining a position designation operation on a field-of-view image from a second viewpoint in the embodiment shown in FIG. 1; FIG. 11 is a diagram for explaining another example of a position designation operation on a field-of-view image from a second viewpoint in the embodiment shown in FIG. 1; FIG. 2 is a diagram showing a state in which a position is specified in the embodiment shown in FIG. 1; FIG. 2 is an image diagram for explaining a case where a virtual object is arranged at a designated position in the embodiment shown in FIG. 1; FIG. 2 is an image diagram for explaining a case in which an object at a designated position is specified in the embodiment shown in FIG. 1; FIG. 2 is an image diagram for explaining a case in which an object at a designated position is specified in the embodiment shown in FIG. 1; FIG. 2 is a diagram schematically showing the appearance of a configuration example of a virtual guide display device and a virtual guide display system according to the present embodiment; 4 is a flowchart for explaining the basic operation of the virtual guide display device and virtual guide display system according to the present embodiment; 1 is a block diagram showing a configuration example of a virtual guide display device according to this embodiment; FIG. FIG. 10 is a diagram for explaining an operation when specifying a range instead of a point when specifying a position on the field-of-view image in the virtual guide display device according to the present embodiment; FIG. 10 is a diagram for explaining an operation when specifying a range instead of a point when specifying a position on the field-of-view image in the virtual guide display device according to the present embodiment; FIG. 10 is a diagram for explaining an operation when specifying a range instead of a point when specifying a position on the field-of-view image in the virtual guide display device according to the present embodiment; FIG. 10 is a diagram for explaining an operation when specifying a range instead of a point when specifying a position on the field-of-view image in the virtual guide display device according to the present embodiment; FIG. 10 is a diagram for explaining an operation when specifying a range instead of a point when specifying a position on the field-of-view image in the virtual guide display device according to the present embodiment; FIG. 10 is a diagram for explaining an operation when specifying a range instead of a point when specifying a position on the field-of-view image in the virtual guide display device according to the present embodiment; FIG. 10 is a diagram for explaining an operation when specifying a range instead of a point when specifying a position on the field-of-view image in the virtual guide display device according to the present embodiment; FIG. 5 is a diagram for explaining a position specifying operation when arranging a virtual object at a specified position in a three-dimensional space in the virtual guide display device according to the embodiment; FIG. 5 is a diagram for explaining a position specifying operation when arranging a virtual object at a specified position in a three-dimensional space in the virtual guide display device according to the embodiment; FIG. 5 is a diagram for explaining a position specifying operation when arranging a virtual object at a specified position in a three-dimensional space in the virtual guide display device according to the embodiment; FIG. 5 is a diagram for explaining a position specifying operation when arranging a virtual object at a specified position in a three-dimensional space in the virtual guide display device according to the embodiment; FIG. 4 is a diagram for explaining a case where two field-of-view images viewed from a first viewpoint and a second viewpoint are displayed on two screens in the virtual guide display device according to the present embodiment; FIG. 4 is a diagram for explaining a case where two field-of-view images viewed from a first viewpoint and a second viewpoint are displayed on two screens in the virtual guide display device according to the present embodiment; FIG. 10 is a view for explaining position designation by tag selection, in which tags are displayed at intersections between the virtual guide and objects in the visual field image in the virtual guide display device according to the present embodiment; FIG. 10 is a view for explaining position designation by tag selection, in which tags are displayed at intersections between the virtual guide and objects in the visual field image in the virtual guide display device according to the present embodiment;

Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. The same reference numerals are given to the same configurations throughout the drawings, and redundant explanations are omitted.

<First Embodiment>
FIG. 1 is a diagram schematically showing the appearance of a configuration example of a virtual guide display device and a virtual guide display system according to this embodiment. 2, 3, 4A, 5, and 4B are diagrams for explaining the three-dimensional position designation operation in the embodiment shown in FIG. In the following, an example of using a head-mounted display (hereinafter referred to as "HMD") 100 worn on the user's head as the first virtual guide display device and a tablet terminal 110 as the second virtual guide display device. will be explained.

In FIG. 1, the HMD 100 comprises a camera 104. The angle of view of the camera 104 is assumed to be the same as the first viewpoint 101 of the first user 10 wearing the HMD 100 . A camera 104 captures a view scene in a three-dimensional space from a first viewpoint 101 to obtain a first view image viewed from the first viewpoint 101 . In this embodiment, the first field-of-view image is an image obtained by photographing the field-of-view scenery in the actual three-dimensional space, but it may be an image of virtual reality viewed from the first viewpoint 101 .

When the first user 10 moves from the position of the first viewpoint 101 to the position of the second viewpoint 102 as indicated by an arrow 105 , the HMD 100 captures the scenery from the camera 104 at the second viewpoint 102 of the first user 10 . is captured to obtain a second view image viewed from a second viewpoint 102 .

On the other hand, the tablet terminal 110 is operated by the second user 20 . The tablet terminal 110 receives field-of-view images obtained by the HMD 100 and viewed from the first viewpoint 101 and the second viewpoint 102 through wireless communication with the HMD 100 . The tablet terminal 110 displays the received field-of-view image on the screen 106 of the tablet terminal 110 .

In addition, the server 120, which can process and store a large amount of information at high speed, transmits and receives various types of information such as view images and virtual guide information to and from the HMD 100 and the tablet terminal 110 through wireless communication.

In such a state, it is necessary to specify an arbitrary position on the 3D space in order to place the virtual object at an arbitrary position on the 3D space or to specify the object at that position.

For example, to designate a position 130 in the empty three-dimensional space shown in FIG. A field-of-view image 140 (see FIG. 2) at 101 is transmitted to the tablet terminal 110 .

The tablet terminal 110 displays the received view image 140 on the screen 106. The second user 20 designates the position 141 of the point when viewed from the first viewpoint 101 on the displayed field-of-view image 140 . A position 141 of a point on the field-of-view image 140 when viewed from the first viewpoint 101 corresponds to a position 130 on the three-dimensional space.

Furthermore, the tablet terminal 110 generates a virtual object (hereinafter referred to as a "virtual guide 150") that extends in the same direction as the first viewpoint 101 via a position 141 specified on the view image 140. A point corresponding to the position 130 to be specified in the three-dimensional space exists on the generated virtual guide 150 .

Next, in the HMD 100 where the first user 10 has moved from the first viewpoint 101 to the second viewpoint 102 (moved along the arrow 105), the camera 104 shoots the scenery from the second viewpoint 102, A view image 160 (see FIG. 3) captured at the second viewpoint 102 is transmitted to the tablet terminal 110 .

A view image 160 as shown in FIG. 3 is displayed on the screen 106 of the tablet terminal 110 . Further, on the tablet terminal 110, the generated virtual guide 150 is superimposed on the view image 160 and displayed.

On the tablet terminal 110, as shown in FIG. 4A, the second user 20, as shown in FIG. Specifies the position 151 at . Therefore, a position 130 in the three-dimensional space to be specified is identified as a position 161 corresponding to the position 151 specified on the virtual guide 150 on the view image 160, as shown in FIG. That is, a virtual guide 150 is generated that extends in the same direction as the first viewpoint 101 via a specified position 141 (see FIG. 2) on the visual field image viewed from the first viewpoint 101, and By displaying the generated virtual guide 150 on the viewed visual field image and specifying the position 151 (see FIG. 4A) on the displayed virtual guide 150, it is possible to identify any position in the three-dimensional space. be possible.

To summarize, the virtual guide 150 is displayed and arranged in space according to the spatial coordinates from the first viewpoint 101, and the virtual guide 150 fixed in the space is viewed from the second viewpoint 102 to create the final three-dimensional space. It determines the top position 130 . Note that the virtual guide 150 may be generated by using the HMD 100 or the server 120 capable of handling a large amount of information other than the tablet terminal 110 . In this case, the server 120 distributes virtual guide information for displaying the virtual guide 150 on the tablet terminal 110 by wireless communication or the like. The process of generating the virtual guide and the process of specifying the position may also be displayed on the HMD 100 so that the first user 10 can see it.

It should be noted that a position near the virtual guide 150 may be specified as an extended method of specifying the position. Description will be made with reference to FIG. 4B.

When a position 152 deviated from the virtual guide 150 is specified, the distance in the depth direction along the second viewpoint 102 cannot be determined simply by specifying the position, and the position 152 cannot be determined. Regarding this point, a rule is established in advance, for example, the position at which the distance to the virtual guide 150 is the shortest, that is, the shortest distance between the straight line in the direction of the second viewpoint 102 passing through the position 152 and the straight line indicated by the virtual guide 150. Then, the position in the depth direction can be determined according to the rule.

Another processing method when the second user 20 designates a position not on the virtual guide 150 is, for example, on the virtual guide 150 closest to the position 152 on the visual field image of the second viewpoint 102 . It is also possible to specify the position of

Also, a plurality of processing rules may be defined when the second user 20 designates a position not on the virtual guide 150 so that the second user 20 can designate which rule to apply. As a result, the degree of freedom in specifying the position is improved, and the convenience for the user is improved.

Next, the case of arranging a virtual object at a specified position in the three-dimensional space will be explained using FIG. FIG. 6 is an image diagram for explaining a case where a virtual object is arranged at the position specified in this embodiment shown in FIG.

In FIG. 6, the tablet terminal 110 determines the designation of, for example, a position 130 in the three-dimensional space by the spatial position designation process described above, and generates a virtual object 170 to be placed on the position 130 . The spatial coordinates of the position 130 are associated with the virtual object 170 .

Furthermore, the HMD 100 displays the generated virtual object 170 with spatial coordinates on the field image 180 (see FIG. 6) viewed from the third viewpoint 103 (see FIG. 1), which is a bird's eye view, for example. FIG. 6 shows, as an example of the virtual object 170, a virtual object labeled "This Point" for indicating a destination or gathering place. As a result, the first user 10 wearing the HMD 100 can easily visually recognize the destination and meeting place indicated by the second user 20 operating the tablet terminal 110 . That is, by arranging the virtual object 170 at the specified position 130 in the three-dimensional space, the virtual object 170 is generated, and the virtual object 170 is generated for the first user 10 different from the second user 20 who is arranged at the position 130 . Useful information such as instructions indicated by 170 can be accurately and easily notified at the specified position 130 .

Note that the virtual object may be generated by using the server 120 other than the tablet terminal 110, and is distributed to the virtual guide display device (HMD 100 in the above example) that can display the virtual object 170 by wireless communication or the like. Become. Alternatively, the server 120 may specify only the spatial coordinates, transmit and receive the spatial coordinate data, and display the virtual object 170 at the spatial coordinate position specified by the HMD 100 . The server 120 may specify the type and display direction of the object, and the HMD 100 may display the object.

Also, a case of specifying a physical object or a virtual object by specifying a position in a three-dimensional space will be described with reference to FIG. 7A. FIG. 7A is an image diagram illustrating a case of identifying an object at a designated position in the embodiment shown in FIG. In FIG. 7A, in the tablet terminal 110, for example, positions 191 and 192 in the three-dimensional space are designated by the spatial position designation process described above.

The HMD 100 uses the specified position 191 to identify the tree 193 of the physical object on the specified position 191 on the field image 180 seen from the third viewpoint 103, for example.

Also, in the HMD 100, using the specified position 192, the virtual object 194 located on the specified position 192 is specified on the field image 180 viewed from the third viewpoint 103, for example.

FIG. 7A shows an example of a screen displaying a virtual object 194 labeled "This is a memorial tree" for explaining the tree 193. FIG. Note that FIG. 7A shows a case in which a physical object such as a tree 193 is specified when viewed from the third viewpoint 103. However, since the first user 10 who is viewing the physical space walks on the ground, the first user 10 It is difficult for the robot 10 to move to the position of the third viewpoint 103 shown in FIG. 7A, and in practice, the physical object may be identified by viewing from a viewpoint that is possible in the physical space.

In the description of FIG. 7A, when specifying a physical object in the physical space, the image of the third viewpoint 103 is merely an example for explanation. However, when an image of the physical space and distance information are captured and treated as a virtual space, it is possible to easily move to the third viewpoint 103, so the physical object may be identified from the third viewpoint 103. .

In addition, as shown in FIG. 7B, in addition to the virtual guide 150 set at the first viewpoint 101, a virtual guide 151a based on the position 151 specified at the second viewpoint 102 is displayed, and the position other than the intersection of the two guides is displayed. It may be configured so that the point 162 can be specified. At this time, regarding the position of the point 162 in the depth direction at the third viewpoint 103, for example, the rule that the sum of the distances to the virtual guide 150 and the virtual guide 151a is the smallest is set in advance and determined. Just do it. This technique improves the convenience of position designation, such as allowing designation of a position deviated from the intersection of the plurality of virtual guides 150 and 151a.

Also, this allows the first user 10 wearing the HMD 100 to accurately and easily confirm the physical object or virtual object pointed by the second user 20 operating the tablet terminal 110 . That is, by going back and forth between the first viewpoint and the second viewpoint, or by recognizing the designated point from the difference in appearance between the first or second viewpoint and the n-th viewpoint, the position in the three-dimensional space can be determined. By specifying, it is possible to accurately and easily indicate a physical object or virtual object at a specific position to a user viewing the physical object or virtual object.

According to the examples described with reference to FIGS. 6, 7A, and 7B, a worker corresponding to the first user 10 photographs a work place or the like with a camera 104 mounted on an HMD, A support instructor corresponding to the second user 20 in the office can easily specify a desired position in the three-dimensional space while viewing the camera-captured image of the work place, etc. on the screen 106 of the tablet terminal 110, etc. . Therefore, information such as work instructions can be given to a worker by a virtual object placed at a specified desired position, or an object at a specified position can be identified and pointed to the worker to inform the worker of the information. Therefore, a support instructor who is unfamiliar with the apparatus can accurately and conveniently provide support such as instructions to remote workers.

Next, another configuration example of the virtual guide display device and virtual guide display system according to this embodiment will be described with reference to FIG. FIG. 8 is a diagram schematically showing the appearance of another configuration example of the virtual guide display device and the virtual guide display system according to this embodiment. The difference from the configuration example shown in FIG. 1 is that the tablet terminal 110 is equipped with a camera 801 and uses the camera 801 to photograph the scenery from a first viewpoint 101 and a second viewpoint 102 . 2 to 7B are used as the drawings used for explaining the position designation.

In FIG. 8, the tablet terminal 110 is provided with a camera 801 that captures a field of view scenery, captures the field of view of the second user 20 from the first viewpoint 101 with the camera 801, and displays the captured field of view image 140 on the screen. 106 and a position 141 (see FIG. 2) is specified on the field of view image 140 displayed by the second user 20 .

Furthermore, the tablet terminal 110 generates a virtual guide 150 (see FIG. 3) that extends in the same direction as the first viewpoint 101 via a position 141 specified on the field-of-view image 140 .

Furthermore, when the second user 20 moves from the position of the first viewpoint 101 to the position of the second viewpoint 102 as indicated by an arrow 105 , the camera 801 of the tablet terminal 110 moves the second user 20 from the second viewpoint. The field of view scenery at the viewpoint 102 is photographed, and the photographed field of view image 160 (see FIG. 3) is displayed on the screen 106. - 特許庁

When the second user 20 is on the virtual guide 150 displayed on the field of view image 160 and sees the position 130 in the three-dimensional space from the second viewpoint 102, the position 151 on the field of view image 160 is the field of view. Specified on image 160 . Therefore, a position 130 on the three-dimensional space to be specified can be specified as a position 161 on the field-of-view image 160 .

This makes it possible for users who are not familiar with the device to easily specify the position in the three-dimensional space while viewing the captured image from the camera themselves. Therefore, other users sharing the space on the spot or remotely can be shown a virtual object arranged at a specified position in the three-dimensional space to be notified of the information indicated by the virtual object, or It is possible to accurately and easily indicate an object at the specified position of the user. Note that the above operation may be performed using the HMD 100 instead of the tablet terminal 110 . At that time, the camera and input device mounted on the HMD 100 are used.

1 and 8, the HMD 100 and the tablet terminal 110 are used as examples of the first virtual guide display device and the second virtual guide display device. You can use any device such as a smartphone, a personal computer, etc.

In the above explanation example, both the first viewpoint 101 and the second viewpoint 102 use field-of-view images of the current landscape. may be specified to generate and display a virtual guide, and a position on the virtual guide may be specified at the current viewpoint as the second viewpoint 102 . In this case, it is possible to designate a coordinate position in the three-dimensional space without moving the place to change the viewpoint.

Similarly, as a first viewpoint, a position is specified using a field image taken at the current viewpoint, a virtual guide is generated and displayed, and a field image taken in the past is used as a second viewpoint. You can specify the position. That is, as the first viewpoint and the second viewpoint, past field-of-view images may be appropriately combined and used.

Also, past visual field images may be used for both the first viewpoint and the second viewpoint. It goes without saying that this past field-of-view image must be associated with spatial coordinate data. Also, satellite photographs and map data may be used as long as they have spatial coordinate data. Image data in the server 120, which can store a large amount of information data 1024, may be used as the past view image. In this case, the capacity load on the first virtual guide display device and the second virtual guide display device can be greatly reduced.

In addition, the server 120 may execute not only processing such as holding past field-of-view images and virtual objects, but also a part of other operation processing such as position designation in a three-dimensional space.

FIG. 9 is an example of a flowchart explaining the basic operation of the virtual guide display device and virtual guide display system according to this embodiment.

In FIG. 9, when a visual field image at a first viewpoint is obtained by the first virtual guide display device (S901), and a position is specified on the obtained visual field image at the first viewpoint (S902), A virtual guide is generated that extends in the same direction as the first viewpoint via the specified position (S903). An example algorithm for generating and displaying a virtual guide is described below.

Next, from the first virtual guide display device, the virtual guide information including the shape information of the virtual guide, the position information indicating the position of the world coordinate system, and the direction information indicating the direction of the world coordinate system is sent to the second virtual guide display device. Send. In the second virtual guide display device, a virtual guide is displayed on the visual field image at the second viewpoint (S904), and a position is designated on the displayed virtual guide (S905).

The position in the three-dimensional space can be specified by the process 900 for specifying the position from the above two viewpoints. The second virtual guide display device converts the specified position into a position in the world coordinate system and outputs the position. This output mode may be displayed on the display 1034 of the second virtual guide display device, or may be displayed in the virtual guide generation display processing unit 1012 (FIG. 10) inside the processor 1010 (FIG. 10) of the second virtual guide display device. 10) to the virtual object generation processing unit 1013 and output to each processing unit that uses the specified position, such as arranging the virtual object at this position. Thereafter, the process proceeds to a process using the output specified position (S906).

For example, there are utilization processes such as arranging a virtual object at a specific position, using it to specify a physical object or virtual object at a specific position, and transmitting and using coordinate information of a specific position to another terminal. be done.

For example, when arranging a virtual object at a specified position, a virtual object representing information such as an instruction is arranged, and useful information such as an instruction is given to a user who can visually recognize the virtual object in the three-dimensional space. be able to.

Further, for example, when specifying the position of an object at a specified position, a physical object or virtual object at the specified position is specified, and the user viewing the physical object or virtual object is notified of the physical object or virtual object. You can point it out. Subsequently, if the position specification in the three-dimensional space is to be repeated, the process returns to S901 (S909: Yes).

On the other hand, if the position designation in the three-dimensional space is not required (S909: No), the series of processing ends.

By the above operation, after the virtual guide is generated in accordance with the spatial coordinates from the first viewpoint, the final position of the generated virtual guide can be specified by viewing the generated virtual guide from the second viewpoint. can be accurately and easily specified.

Furthermore, for a user wearing a virtual guide display device that can display and visually recognize a specific position in a three-dimensional space, useful information can be displayed by a virtual object placed at the specified position, or an object at the specified position can be displayed to the user. By pointing, it is possible to implement user-friendly instruction support to the user.

Next, a configuration example of the virtual guide display device according to this embodiment will be described using FIG. FIG. 10 is a functional block diagram of a configuration example of a first virtual guide display device and a second virtual guide display device according to this embodiment.

In FIG. 10, the first virtual guide display device (HMD 100) and the second virtual guide display device (tablet terminal 110) include a processor 1010, a memory 1020, a camera 104 (a camera 801 in the tablet terminal 110), and a left eye line detection sensor. 1032, a right eye line of sight detection sensor 1033, a display 1034, an operation input device 1035, a microphone 1036, a speaker 1037, a vibrator 1038, a communication I/F (communication device) 1039, a sensor group 1040, etc. They are interconnected via bus 1050 .

The processor 1010 is composed of a CPU, ROM, RAM, etc., and constitutes a controller of the first virtual guide display device (HMD 100) and the second virtual guide display device (tablet terminal 110).

The processor 1010 executes processing according to an operating system (OS) 1022 stored as a control program 1021 in the memory 1020 and an application program 1023 for operation control, thereby performing each function in the processor 1010. It controls the parts and implements the functions of the OS, middleware, applications, etc. and other functions.

Functional units configured by execution by the processor 1010 include a position designation processing unit 1011, a virtual guide generation display processing unit 1012, and a virtual object generation processing unit 1013.

The memory 1020 is composed of a non-volatile storage device or the like, and stores various programs 1021 and information data 1024 handled by the processor 1010 and the like. The information data 1024 includes coordinate position information 1025 indicating a spatial coordinate position such as a designated position, virtual guide information 1026 required to generate and display a virtual guide, virtual object information 1027 representing a virtual object, and physical object information 1027. The view field image information 1028 of the photographed scenery and the like are stored.

The cameras 104 and 801 take images of the field of view around the front, and acquire the field of view image by converting the light incident from the lens into an electrical signal with an imaging device. In the case of the HMD 100 of the optical see-through type, the first user 10 obtains a field image by photographing the field of view with the camera 104 while visually recognizing an actual object in the front surrounding field of view with his/her own eyes.

The left-eye line-of-sight detection sensor 1032 and the right-eye line-of-sight detection sensor 1033 detect the line of sight by capturing the movements and orientations of the left and right eyes, respectively. For the processing of detecting the line of sight, a well-known technology that is generally used as eye tracking processing may be used. A known technique is to detect the line of sight based on the position of the pupil with respect to the position of the corneal reflection, with the position of the reflected light (corneal reflection) on the cornea taken by an infrared camera and the position of the corneal reflection as a reference point. Also known is a method of capturing an eye with a visible light camera, setting the inner corner of the eye as a reference point and the iris as a moving point, and detecting the line of sight based on the position of the iris relative to the inner corner of the eye.

In the tablet terminal 110, the display 1034 includes a screen 106 configured by a liquid crystal panel, and displays a visual field image, notification information to the user such as an alarm, and the like on the screen 106.

The operation input device 1035 may be, for example, a capacitive touch pad stacked on the screen 106 . The touch pad detects an approach or contact operation (touch operation) by a finger, touch pen, or the like. The position can be easily specified by the user performing a touch operation on the position to be specified on the display image.

As the display 1034, in the case of the HMD 100, in the case of an optical see-through type, for example, a projector that projects a virtual object, notification information to the user, and the like, and a transparent transparent display that displays the projected virtual object, etc. in front of the eyes. A half mirror may be used. As a result, the user can visually recognize both the real object in the field of view in front of the user and the imaged virtual object as if they are floating.

Also, when the HMD 100 is of the video see-through type, it is configured using a display 1034 such as a liquid crystal panel that displays together the physical object in front of the eye photographed by the camera 104 and the virtual object. As a result, the user can use the HMD 100 to superimpose and visually recognize the physical object and the virtual object within the visual field in front of the user's eyes.

The operation input device 1035 may be, for example, a keyboard, key buttons, touch keys, or the like. The operation input device 1035 may be provided in a position and form in which the first user 10 can easily perform an input operation in the HMD 100, or may be separated from the main body of the HMD 100 and connected by wire or wirelessly.

Alternatively, the input operation screen may be displayed on the screen 106 of the display 1034, and the input operation information may be captured based on the position on the input operation screen to which the line of sight is directed detected by the left-eye line-of-sight detection sensor 1032 and the right-eye line-of-sight detection sensor 1033. Alternatively, a pointer may be displayed on the input operation screen and input operation information may be obtained by operating the pointer with the operation input device 1035 .

Also, the operation input device 1035 may utter a voice indicating an input operation by the user, collect the sound with the microphone 1036, and capture the input operation information.

The microphone 1036 collects voice from the outside or the user's own voice and converts it into voice data. Instruction information uttered by the user can be taken into the virtual guide display device, and an operation in response to the instruction information can be conveniently executed.

A speaker 1037 outputs sound based on the sound data. Thereby, the notification information to the user can be notified by voice. Speaker 1037 can be replaced with headphones.

The vibrator 1038 generates vibration under the control of the processor 1010, and converts notification information to the user transmitted by the virtual guide display device into vibration. In particular, in the HMD 100, by causing the vibrator 1038 to vibrate the user's head to which the vibrator 1038 is closely attached, it is possible to reliably transmit the notification to the user. Examples of information to be notified to the user include designation of the position at the first viewpoint 101, generation and display of the virtual guide 150, placement of the virtual object 170, and finally specification of the position in the three-dimensional space. There are notifications and notification contents, and these notification information can improve usability.

The communication I/F 1039 is a communication interface that performs wireless communication with other nearby information terminals by short-range wireless communication, wireless LAN, base station communication, or the like. Including antennas, etc. The communication I/F 1039 performs wireless communication between the first virtual guide display device and the second virtual guide display device and with the server 120 . As short-range wireless communication, Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or Wi-Fi (registered trademark) It is performed using a wireless LAN such as. Further, as base station communication, long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access, registered trademark) and GSM (Global System for Mobile Communications) may be used. Although not shown, the communication I/F 1039 may use other methods such as optical communication and sound wave communication as means for wireless communication. In that case, a light emitting part, a light receiving part, a speaker and a microphone are used instead of the transmitting/receiving antenna. In addition, when handling high-definition video, etc., the amount of data used is dramatically large, and in this case, high-speed large-capacity communication networks such as 5G (5th Generation: 5th generation mobile communication system) and local 5G are used for wireless communication. This can dramatically improve usability.

The distance sensor 1041 is a sensor that measures the distance between the virtual guide display device and a real object in the outside world. The distance measurement sensor 1041 may use a TOF (Time Of Flight) sensor, or may use a stereo camera or another method. From the three-dimensional data created using the distance measuring sensor 1041, it is possible to create a virtual space in which physical objects are virtually arranged, and it is also possible to arrange virtual objects by designating three-dimensional coordinates in the virtual space. becomes.

The acceleration sensor 1042 is a sensor that detects acceleration, which is a change in speed per unit time, and can detect movement, vibration, impact, and the like.

The gyro sensor 1043 is a sensor that detects the angular velocity in the rotational direction, and can capture the state of vertical, horizontal, and diagonal postures.

The posture and movement of the head of the first user 10 can be detected using the acceleration sensor 1042 and the gyro sensor 1043 mounted on the HMD 100 .

The geomagnetic sensor 1044 is a sensor that detects the magnetic force of the earth, and detects the direction in which the virtual guide display device is facing. It is also possible to detect the movement of the virtual guide display device by using a 3-axis type that detects geomagnetism in the vertical direction as well as the front and back directions and the left and right directions, and by capturing changes in geomagnetism with respect to the movement of the virtual guide display device. The gyro sensor 1043 and the geomagnetic sensor 1044 function as direction sensors that detect the orientation of the HMD 100 in the world coordinate system.

By using the sensor output of each of these sensors, it is possible to detect the posture and movement fluctuations of the virtual guide display device in detail, which can help confirm the position in the three-dimensional space.

The GPS sensor 1045 receives signals from GPS (Global Positioning System) satellites in the sky and detects the current position of the virtual guide display device. This makes it possible to fix the position of the moving viewpoint. GPS sensor 1045 is a position sensor that detects a position in the world coordinate system.

The position designation processing unit 1011 performs processing for designating a position on the field-of-view image displayed on the display 1034 using the operation input device 1035 . The position of a point or range desired by the user is designated on the field of view image viewed from the first viewpoint 101, and the desired position on the displayed virtual guide 150 is designated on the field of view image viewed from the second viewpoint 102. specify. Also, a position 152 around the virtual guide may be specified using the virtual guide 150 as a guideline, not only on the virtual guide 150 .

The virtual guide generation display processing unit 1012 generates a virtual guide 150 that extends in the same direction as the first viewpoint 101 via points 141 and range positions specified on the field of view image at the first viewpoint 101 . In addition, processing for displaying a virtual guide 150 on the field-of-view image at the second viewpoint 102 is performed.

The virtual object generation processing unit 1013 generates a virtual object 170 that is an object in a virtual space different from the real space. Note that the virtual object 170 generated by the external server 120 may be imported into the virtual guide display device through wireless communication.

With the above configuration, in the tablet terminal 110, the position designation processing unit 1011 designates the position 141 on the view image 140 viewed from the first viewpoint 101, and the virtual guide generation display processing unit 1012 passes through the designated position 141. A virtual guide 150 extending in the same direction as the first viewpoint 101 is generated.

Furthermore, the virtual guide generation and display processing unit 1012 displays the virtual guide 150 generated on the visual field image 160 viewed from the second viewpoint 102, and the position on the virtual guide 150 displayed by the position designation processing unit 1011 is specified. do. Therefore, the virtual guide 150 is generated according to the space coordinates at the first viewpoint 101, and the final position in the three-dimensional space is specified while viewing the virtual guide 150 fixed in the space from the second viewpoint 102. be able to. That is, by specifying a position in space from two different viewpoints while using the virtual guide 150, it is possible to determine an arbitrary position in the three-dimensional space accurately, easily and conveniently.

The field-of- view images 140 and 160 viewed from the first viewpoint 101 and the second viewpoint 102 may be images captured by the camera 104 of the HMD 100 or images captured by the camera 801 of the tablet terminal 110 .

Furthermore, the virtual object 170 generated by the virtual object generation processing unit 1013 is arranged at the specified coordinate position in the three-dimensional space, and the virtual object with spatial coordinates is displayed on the display 1034 of the first and second virtual guide display devices. By displaying, it is possible to inform the user who operates the first and second virtual guide display devices of the useful information indicated by the virtual object 170 at the appropriate spatial coordinate position.

In addition, by specifying the specified position in the three-dimensional space on the display 1034 of the first and second virtual guide display devices, the user can be pointed to the physical object or virtual object at the specified position, It is possible to easily give an accurate object indication to the user.

As a virtual guide display system having the first and second virtual guide display devices, the HMD 100 captures the field of view seen from the first viewpoint 101 and the second viewpoint 102 with the camera 104, and 1. Transmit the second field-of-view image to the tablet terminal 110 .

Alternatively, in the tablet terminal 110, the field of view viewed from the first viewpoint 101 and the second viewpoint 102 is photographed by the camera 801, and the photographed first and second field of view images are acquired.

In the tablet terminal 110, based on the first field-of-view image transmitted from the HMD 100 or the first field-of-view image acquired by the tablet terminal 110, the position designation processing unit 1011 displays the first field-of-view image viewed from the first viewpoint 101. A virtual guide generation display processing unit 1012 generates a virtual guide 150 extending in the same direction as the first viewpoint 101 via a position 140 specified on the field of view image.

After that, in the tablet terminal 110 , based on the second field-of-view image transmitted from the HMD 100 or the second field-of-view image acquired by the tablet terminal 110 , the virtual guide generation display processing unit 1012 generates images from the second viewpoint 102 . The generated virtual guide 150 is displayed on the viewed second view image, and a position 151 on the virtual guide 150 displayed by the position designation processing unit 1011 is designated. As a result, the second virtual guide display device can accurately and easily determine an arbitrary position in the three-dimensional space with good usability.

Further, in the tablet terminal 110, the virtual object generation processing unit 1013 generates a virtual object 170 at a position 151 on the virtual guide 150 specified by the tablet terminal 110, and transmits the generated virtual object with spatial coordinates to the HMD 100. , the HMD 100 receives the virtual object with spatial coordinates transmitted from the tablet terminal 110, and displays the received virtual object with spatial coordinates on the visual field image according to the spatial coordinates.

Therefore, it is possible for the user holding the tablet terminal 110 to inform the user equipped with the HMD 100 of useful information indicated by the virtual object at a suitable spatial coordinate position.

In addition, the tablet terminal 110 transmits to the HMD 100 spatial coordinate position information indicating a position 151 on the virtual guide 150 specified by the position specification processing unit 1011 . The HMD 100 receives the spatial coordinate position information transmitted from the tablet terminal 110 and displays the position indicated by the received spatial coordinate position information on the visual field image. Therefore, the user holding the tablet terminal 110 can point to the user wearing the HMD 100 a physical object or a virtual object at a designated position in the three-dimensional space.

<Second embodiment>
The second embodiment is an embodiment in which a range is specified instead of a point when specifying a position on the view field image in the virtual guide display device. 11A to 11G are diagrams for explaining an operation when specifying a range instead of a point when specifying a position on the field-of-view image viewed from the first viewpoint 101 and the second viewpoint 102. FIG. In FIGS. 11A to 11G, the parts shown in FIGS. 2 to 7 and denoted by the same reference numerals have the same operations as those already described in FIGS. 2 to 7, so detailed descriptions thereof will be omitted. .

As illustrated in FIG. 11A, a range 1101 on the field of view image 140 is specified by the position specification processing unit 1011 . When the range 1101 is designated, as shown in FIG. 11B, the virtual guide 1102 displayed on the field-of-view image 160 is generated as a three-dimensional object having the range 1101 as a cross section.

When the position designation processing unit 1011 acquires a position designation operation on the virtual guide 1102, it is possible to specify the position in the three-dimensional space with a range. Therefore, when arranging a wide-spread virtual object, it is possible to specify a range corresponding to the spread, or to specify the range of a physical object or a virtual object and point to that object.

Here, as shown in FIG. 11C, by making the virtual guide 1102 translucent, an object hidden behind the virtual guide 1102 can be seen and the background of the virtual guide 1102 can be confirmed, which is convenient. can improve.

Further, as shown in FIG. 11D, when the position designation processing unit 1011 places a finger (pointer) or the like on the visual field image 160 at the second viewpoint 102 and designates it, a ring object framing the vicinity of the designated position is displayed. It may be displayed as 1103, and the specified position can be visually recognized stereoscopically.

Also, the range to select and specify may be distorted. That is, not only when the shape of the range at the first viewpoint 101 is distorted, but also at the second viewpoint 102, a range having a unique shape different from that at the first viewpoint 101 is specified. However, when combined with the range seen from the first viewpoint 101 side, it may become stereoscopically distorted. In this way, when the range to be selected and specified becomes distorted, the sphere object 1106 including the range to be selected and specified may be displayed as the specified range.

For example, as shown in FIG. 11E, when the virtual guide 1104 has an distorted range 1105 and the position designation processing unit 1011 acquires a designation operation of the distorted range 1105 on the field image 160 at the second viewpoint 102, , as shown in FIG. 11F, a sphere object 1106 surrounding a three-dimensional distorted range 1105 is specified as the range.

As a result, as shown in FIG. 11G, a sphere object 1106 including a selected and specified range 1105 can be displayed as a specified range. The identification can be performed accurately and conveniently.

Also, the object may be a closed three-dimensional area other than the spherical object 1106, and may be an object of other geometric shape such as a rectangular parallelepiped.

Next, an operation example when arranging a virtual object in the virtual guide display device according to this embodiment will be described. FIG. 12 is a diagram for explaining a position specification operation when arranging a virtual object at a specified position in a three-dimensional space. In FIG. 12, the parts shown in FIGS. 2 to 7 and denoted by the same reference numerals have the same operations as those already explained in FIGS. 2 to 7, so detailed explanations thereof will be omitted. As an example, a case of arranging a virtual object representing a new car for advertisement at a specified position in a three-dimensional space will be described.

When a position is specified on the view image 140 at the first viewpoint 101, as shown in FIG. 12A, a new car-shaped virtual object 1201 viewed from the first viewpoint 101 is displayed instead of a point.

Furthermore, as shown in FIG. 12B, on the field-of-view image 160 at the second viewpoint 102, a virtual guide 1202 is generated and displayed that has a shape of a new car of the virtual object 1201 and extends in the same direction as the first viewpoint 101 in cross section. When the user designates a position 1203 on the displayed virtual guide 1202, the position designation processing unit 1011 acquires the designation operation.

12C and 12D show the field of view of a virtual object 1204 arranged at a specified position in the three-dimensional space and having a cross-sectional shape representing the shape of a new car, viewed from the second viewpoint 102 and the third viewpoint 103. It is an image. As a result, it becomes possible to specify the position in the three-dimensional space while visually recognizing the virtual object 1204 to be arranged, and the user can greatly improve usability when specifying the position. That is, a shape object and a virtual guide reflecting the shape and size of the virtual object 1204 to be placed when the position is specified at the first viewpoint 101 and the second viewpoint 102 are displayed to obtain further convenience. .

Next, in the virtual guide display device according to the present embodiment, a case of displaying both field-of- view images 140 and 160 viewed from the first viewpoint 101 and the second viewpoint 102 on two screens will be described. 13A and 13B show, in addition to the field-of-view image 160 viewed from the second viewpoint 102, a field-of-view image viewed from the first viewpoint 101 on a part of the screen 106 (a screen area smaller than the screen area of the field-of-view image 160). 140 is a diagram showing displaying 140. FIG. In FIGS. 13A and 13B, the parts shown in FIGS. 2 to 7 and denoted by the same reference numerals have the same operations as those already explained in FIGS. 2 to 7, so detailed explanations thereof will be omitted. .

In FIG. 13A , in addition to the field-of-view image 160 viewed from the second viewpoint 102 when specifying a position on the field-of-view image 160 viewed from the second viewpoint 102 , a It shows a state in which the viewed field image 140 is displayed in a reduced form.

Also, in FIG. 13B, the field image 160 viewed from the second viewpoint 102 is displayed on the left side of the screen 106, and the field of view image 140 viewed from the first viewpoint 101 is displayed on the right side of the screen 106.

By displaying the field-of- view images 140 and 160 viewed from the first viewpoint 101 and the second viewpoint 102 on two screens in this way, when specifying a position on the field-of-view image 160 viewed from the second viewpoint 102, The state in which the position has been designated by the first viewpoint 101 can also be visually recognized before that, and the ease of use can be improved.

In addition, when displaying both field-of- view images 140 and 160 viewed from the first viewpoint 101 and the second viewpoint 102 on two screens, the vicinity of the position designation portion at the first viewpoint 101 may be enlarged and displayed, A transition video from the first viewpoint 101 to the second viewpoint 102 may be used. If playback control such as repetition is enabled for the transition moving image, the state of the transition can be easily grasped, which is effective in improving the accuracy of position designation.

Also, the state designated by the second viewpoint 102 may be synthesized with the field-of-view image 140 of the first viewpoint 101 and reflected. For example, when the position is specified from the second viewpoint 102, if the setting is made to display the actual virtual object at the coordinates, the view image 140 from the first viewpoint 101 is also viewed from the angle of the first viewpoint 101. Synthesize images of virtual objects. Therefore, it is possible to obtain an effect that it is possible to confirm how the object looks from the first viewpoint 101 without having to bother to return to the first viewpoint 101 .

As a method of displaying the virtual guide, if processing (so-called occlusion processing) is performed so that the virtual guide 150 is not displayed in the shadow of the real object when viewed from the current viewpoint, the position in the depth direction with respect to the real object is displayed. The relationship becomes easy to understand and the position specification becomes easy. However, if the physical objects are densely packed, the virtual guide 150 is difficult to see, and when the position of the point is specified from the second viewpoint 102, if the virtual guide is a line, it may be thin and difficult to see. On the other hand, the virtual guide 150 may be displayed thicker, blinked, displayed in front of the physical object that shields the virtual guide 150, or may be combined.

It may also be displayed in combination with the display of the original virtual guide 150. For example, the original display of the virtual guide 150 and the one displayed in front of the real object may be combined and displayed alternately, or a combination of the thick display and the original thin display may be used.

You may also use the view image from another camera. When there is another camera having spatial coordinate data that can be shared with the camera 104 photographed from the first viewpoint 101, instead of moving the photographer to the second viewpoint 102, the field of view image photographed by the other camera may be used as the view image 160 at the second viewpoint 102 to specify the spatial coordinate position. Other cameras include, for example, surveillance cameras, cameras used by other people on site, and the like. Needless to say, the field-of-view image must be associated with the spatial coordinate data. Also, satellite photographs and map data may be used as long as they have spatial coordinate data.

Furthermore, it may be applied to the virtual space based on the acquired 3D data. For example, a virtual space in which physical objects are virtually arranged can be created from three-dimensional data created using the distance measuring sensor 1041, and a virtual object can be arranged by designating three-dimensional coordinates in the virtual space. is possible.

Also, in the case of specifying a position within the virtual space, the second viewpoint 102 may be obtained by rotating the virtual space. That is, by rotating the three-dimensional data of the object in the virtual space, the position to be specified can be changed by changing the viewpoint without moving from the first viewpoint 101 to the second viewpoint 102 . It becomes possible to specify.

Also, when specifying a physical object, a mark may be used to identify a candidate for specification. For example, as shown in FIG. 14A, when a plurality of physical objects 1401, 1402, 1403, 1404, and 1405 exist on a virtual guide 1400, the virtual guide 1400 and the plurality of physical objects 1401 and 1402 are shown in FIG. 14B. , 1403, 1404, and 1405 are displayed, and by selecting tags 1411, 1412, 1413, 1414, and 1415, the tagged physical objects are specified. As a result, even when a plurality of physical objects are densely packed on the virtual guide 1400 and it is difficult to select the designated object, the position can be designated by the tag selection operation, and the desired physical object can be pointed accurately and easily. It becomes possible.

<Algorithm for how to set the virtual guide>
A method of setting a virtual guide by changing the viewpoint will be described. First, regarding the coordinate system, the coordinate system of the target space in which the virtual guide is set is assumed to be the world coordinate system. The target space may be a real space or a virtual space. Then, consider a local coordinate system with each viewpoint as the coordinate origin. The viewpoint may or may not be the same as the position of the actual virtual guide display. In any case, it is assumed that an image seen from a viewpoint used as a reference for the display is displayed on the virtual guide display device.

The orientation of the local coordinate system with respect to the world coordinate system is represented by the rotation matrix R. In the world coordinate system, the coordinate axis direction rotated by R becomes the coordinate axis direction of the local coordinate system. Also, the origin of the local coordinate system, that is, the viewpoint is represented by O, which is the position coordinate in the world coordinate system. Here, assuming that the position coordinate in the local coordinate system corresponding to the position coordinate X in the world coordinate system is U, the relationship is expressed by the following equation (1).
U=R ⁻¹ (X−O) (1)
The rotation matrix R and the origin O of the local coordinate system are updated as the viewpoint moves after initial setting.

Next, write down a specific formula expressing the reference line (hereinafter simply referred to as the reference line) that serves as a reference for drawing the virtual guide in the world coordinate system. Take FIGS. 2 and 3 as an example. First, the direction of the virtual guide viewed from the first viewpoint in FIG. 2 is determined by designating point 141 . The direction is determined as the direction vector N1 in the local coordinate system (hereinafter referred to as the _first local coordinate system) corresponding to the first viewpoint. Here, assuming that the rotation matrix representing the orientation of the _first local coordinate system at this time is _R1 , the direction vector W1 of the reference line in the world coordinate system is given by the following equation (2).
_W1 = _R1N1 ( ₂ )

Assuming that the coordinate of the _first viewpoint in the world coordinate system is O1, the reference line in the world coordinate system can be expressed as _a straight line passing through the _point O1 and extending in the same direction as the direction vector W1. . When written down using the real number parameter _k , the point X1 on the reference line is obtained by the following equation (3).
X1= _O1 + _kW1 ( ₃ )

By using the information of _O1 and W1, the reference line of the virtual guide ₁₅₀ in the local coordinate system corresponding to the second viewpoint 102 (hereinafter referred to as the second local coordinate system) can be configured. Let _R2 be the rotation matrix representing the orientation of the second local coordinate system in the world coordinate system, and _O2 be the position of the second viewpoint 102 in the world coordinate system. Then, if a point on the reference line of the virtual guide 150 in the _second local coordinate system is represented by U2, _U2 is represented by the following equations ((1), (3 ) formula).
U ₂ =R ₂ ⁻¹ (O ₁ −O ₂ +kW ₁ ) (4)

The range of the real number parameter k is the range required for drawing on the display screen displaying the virtual guide. Exceptionally, when the direction of the virtual guide is perpendicular to the display screen, only the section of the virtual guide corresponding to the point R ₂ ^-1 (O ₁ -O ₂ ) is displayed.

In the other virtual guide display device (tablet terminal 110), similarly to the above, the coordinates in the world coordinate system of the viewpoint used as the display reference in the virtual guide display device (tablet terminal 110) and the world coordinates in the local coordinate system A rotation matrix representing orientation in the system can be used to construct the reference line.

The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

In addition, part or all of the above configurations, functions, processing units, processing means, etc. may be realized by hardware, for example, by designing them as integrated circuits. Moreover, each of the above configurations, functions, and the like may be realized by software by the processor 1010 interpreting and executing the program 1021 for realizing each function. Information such as programs, tables, and files that implement each function may be stored in the memory 1020, a recording device such as a hard disk, SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. Alternatively, it may be stored in a device on a communication network.

In addition, control lines and information lines indicate what is considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In practice, it may be considered that almost all configurations are interconnected.

10: first user 20: second user 100: HMD
101: first viewpoint 102: second viewpoint 103: third viewpoint 104, 801: camera 105: arrow 106: screen 110: tablet terminal 120: servers 140, 160, 180: view images 150, 151a, 1102, 1104, 1202, 1400: virtual guides 170, 194, 1201, 1204: virtual objects 191, 192: designated position 193: tree 1010: processor 1011: position designation processing unit 1012: virtual guide generation display processing unit 1013: virtual object generation processing Unit 1020 : Memory 1021 : Program 1023 : Application program 1024 : Information data 1025 : Coordinate position information 1026 : Virtual guide information 1027 : Virtual object information 1028 : View image information 1032 : Left-eye line-of-sight detection sensor 1033 : Right-eye line-of-sight detection sensor 1034 : Display 1035: operation input device 1036: microphone 1037: speaker 1038: vibrator 1039: communication I/F
1040: Sensor group 1041: Ranging sensor 1042: Acceleration sensor 1043: Gyroscope sensor 1044: Geomagnetic sensor 1045: GPS sensor 1050: Buses 1101, 1105: Range 1103: Ring object 1106: Sphere object 1401, 1402, 1403, 1404, 1405 : real object 1411, 1412, 1413, 1414, 1415: tag

Claims (16)

1. A virtual guide display device,
   a direction sensor that detects the direction in the world coordinate system to which the virtual guide display device is directed;
   a position sensor that detects the position of the virtual guide display device in the world coordinate system;
   a display;
   an operation input device that receives an operation to specify an arbitrary point on the image displayed on the display;
   a processor connected to each of the orientation sensor, the position sensor, the display, and the operation input device;
   The processor
   Via a specified position on a first field-of-view image seen from the first viewpoint from the virtual guide display device based on sensor outputs from the direction sensor and the position sensor, the first viewpoint and generate virtual guides extending in the same direction,
   superimposing the virtual guide on a second view image viewed from a second viewpoint different from the first viewpoint and displaying the virtual guide on the display;
   The operation input device receives an operation to specify a position on the virtual guide displayed on the display,
   converting the specified position into a position in the world coordinate system and outputting it;
   A virtual guide display device characterized by:
2. The virtual guide display device according to claim 1,
   further comprising a camera connected to the processor;
   the direction sensor detects a direction in which the camera is facing;
   the position sensor detects the position of the camera;
   The first field-of-view image is a field-of-view image captured by the camera from the first viewpoint,
   A virtual guide display device characterized by:
3. The virtual guide display device according to claim 2,
   The second field-of-view image is a field-of-view image captured by the camera from the second viewpoint,
   A virtual guide display device characterized by:
4. The virtual guide display device according to claim 3,
   further comprising a communication device that transmits and receives data to and from another communication device;
   The processor
   transmitting virtual guide information including shape information of the generated virtual guide and position information indicating the position of the world coordinate system to the other communication device;
   receiving virtual guide information to which position information specified on the virtual guide is added from the other communication device;
   superimposing and displaying the virtual guide and the specified position on the display based on the received virtual guide information;
   A virtual guide display device characterized by:
5. The virtual guide display device according to claim 1,
   The first field-of-view image is a current field-of-view image or a past field-of-view image,
   The second field-of-view image is a field-of-view image captured at a time different from the time at which the first field-of-view image was captured,
   A virtual guide display device characterized by:
6. The virtual guide display device according to claim 1,
   The operation input device receives an operation to specify a range on the first field-of-view image,
   The processor superimposes and displays the virtual guide, which is a three-dimensional object having a cross-sectional shape corresponding to the range, on the second field-of-view image.
   A virtual guide display device characterized by:
7. The virtual guide display device according to claim 6,
   The processor translucently superimposes a virtual guide composed of the three-dimensional object on the second field-of-view image.
   A virtual guide display device characterized by:
8. The virtual guide display device according to claim 6,
   The operation input device receives an operation of specifying a position on the virtual guide composed of the three-dimensional object displayed on the second field-of-view image,
   The processor superimposes a ring object containing a position specified on the virtual guide or a sphere object containing the specified position on the virtual guide,
   A virtual guide display device characterized by:
9. The virtual guide display device according to claim 1,
   The processor
   When designation of a position on the first field-of-view image is accepted by the operation input device, the first field-of-view image is displayed on the display, and a virtual object is placed at the designated position in the first field-of-view image. superimposed on the
   A virtual guide display device characterized by:
10. The virtual guide display device according to claim 9,
    The processor superimposes and displays the virtual object superimposed on the first field-of-view image on the second field-of-view image using a shape viewed from the second viewpoint.
    A virtual guide display device characterized by:
11. The virtual guide display device according to claim 1,
    The processor displays the second field-of-view image on the display and displays the first field-of-view image on a screen area smaller than the second field-of-view image.
    A virtual guide display device characterized by:
12. The virtual guide display device according to claim 1,
    The processor blinks the virtual guide.
    A virtual guide display device characterized by:
13. The virtual guide display device according to claim 1,
    The processor performs occlusion processing between the virtual guide and the object imaged in the second view image, and superimposes it on the second view image.
    A virtual guide display device characterized by:
14. The virtual guide display device according to claim 1,
    The processor places a mark tag at the intersection of the virtual guide and the object captured in the first view image or the intersection of the virtual guide and the object captured in the second view image. display and
    The operation input device receives a selection operation of the tag,
    A virtual guide display device characterized by:
15. A virtual guide display system in which a first virtual guide display device and a second virtual guide display device are connected for communication,
    The first virtual guide display device,
    a first direction sensor for detecting a direction in a world coordinate system to which the first virtual guide display device is directed;
    a first position sensor that detects the position of the first virtual guide display device in the world coordinate system;
    a first display;
    a first operation input device that receives an operation to specify an arbitrary point on the image displayed on the first display;
    a first communication device for communicating with the second virtual guide display device;
    a first processor connected to each of the first direction sensor, the first position sensor, the first display, the first operation input device, and the first communicator;
    The second virtual guide display device,
    a second direction sensor that detects the direction in which the second virtual guide display device is facing;
    a second position sensor that detects the position of the second virtual guide display;
    a second display;
    a second operation input device that receives an operation to specify an arbitrary point on the image displayed on the second display;
    a second communication device for communicating with the second virtual guide display device;
    a second processor connected to each of the second direction sensor, the second position sensor, the second display, the second operation input device, and the second communicator;
    The first processor
    Based on the sensor outputs from the first direction sensor and the first position sensor, a specified position on the first field-of-view image viewed from the first viewpoint from the first virtual guide display device is determined. generating a virtual guide extending in the same direction as the first viewpoint, and transmitting virtual guide information including shape information of the generated virtual guide and position information indicating the position of the world coordinate system to the first communication device; from to the second virtual guide display device,
    The second virtual guide display device,
    receiving the virtual guide information from the second communicator;
    The second processor displays the virtual guide on the second display based on the received virtual guide information on a second view image viewed from a second viewpoint different from the first viewpoint. death,
    The second operation input device receives an operation for designating a position on the virtual guide displayed on the second display,
    converting the specified position into a position in the world coordinate system and outputting it;
    A virtual guide display system characterized by:
16. A virtual guide display method executed by a virtual guide display device having a processor, comprising:
    The processor
    obtaining information indicative of the position and orientation of the world coordinate system of the first viewpoint;
    obtaining a specified position on the field-of-view image viewed from the first viewpoint;
    generating a virtual guide that passes through the specified position and extends in the same direction as the first viewpoint;
    displaying the virtual guide on a field-of-view image viewed from a second viewpoint different from the first viewpoint;
    receiving a designation of a position on the displayed virtual guide;
    a step of converting the specified position into a position in the world coordinate system and outputting the position;
    A virtual guide display method characterized by executing
    

Images