JP2020074066A - Image display device and control method of image display device - Google Patents

Abstract

To provide a technique capable of easily and accurately specifying a virtual object in a sharing video on a display screen of the apparatus when a video in a composite real space seen by a user wearing a head-mounted display device such as an HMD is displayed and shared in other apparatuses.SOLUTION: An image display device acquires a video in a composite real space displayed on a display device in a first mode and displays it, and generates an image of a virtual object and displays it when the first mode is switched to a second mode. The image display device transmits, to a holding device, information on parts corresponding to a portion specified by the user for the virtual object displayed in the second mode of the parts constituting the virtual object. The holding device receives the information and updates data on the parts corresponding to the information of the parts constituting the virtual object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a mixed reality presentation technique.

  In manufacturing fields such as automobiles and aircraft, it is required to shorten the design process period and reduce costs. As a technology that meets this demand, there is a mixed reality (MR) system that seamlessly combines a real space and a virtual space. According to the mixed reality system, the product under development can be superimposed and displayed as a CG (computer graphics) image on the real world. Then, by checking and examining from a free viewpoint, it is possible to study the design and the like without producing a full-scale model.

  In the mixed reality system, a head-mounted display (hereinafter, referred to as HMD: head mounted display) in which an image pickup device such as a video camera and a display are integrated is used as one of image display devices. A video see-through method is one form of HMD. This is a method of superimposing and displaying an image of a virtual space generated in accordance with the position and orientation of the image pickup device on an image of the physical space imaged by the image pickup device mounted on the HMD. Examples of the image in the virtual space include, for example, a virtual object drawn by CG and character information. Another image display device is an optical see-through system. This is a method of displaying an image of a virtual space generated according to the position and orientation of the observer's viewpoint on an optical transmissive display mounted on the head of the experiencer.

  In the manufacturing design process, reviews with many participants are often held. In a review using a mixed reality system, in order to allow a large number of people to experience a mixed reality space (a space in which a virtual space and a real space are combined), in addition to the HMD, the system is combined with a handheld information terminal such as a tablet. Composed. This system simultaneously delivers / displays the video of the mixed reality space viewed by the HMD wearer on a plurality of tablet screens at the same time. This allows multiple people to share the field of view of the HMD wearer at the same time and participate in design / study.

  Patent Document 1 discloses an example of collaborative work using a mixed reality system. In this example, multiple participants can remotely share the mixed reality space that workers are experiencing, and while changing viewpoints, they can perceive real objects and virtual objects in a seamless state and collaborate. To

  Further, Non-Patent Document 1 discloses a technique for switching between an augmented reality (AR) display mode and a virtual reality display mode (VR: Virtual Reality) display mode in a navigation system. As a result, the user can appropriately grasp the surrounding situation and support the action. In addition, a commercial graphics library is provided with a function of finding the correspondence between an arbitrary point on the screen (display image plane) and a component forming a three-dimensional model in virtual space by inverse perspective projection transformation. ..

JP 2006-293604 A

Makita, et. Al. "Mixed Reality Navigation on a Tablet Computer for Supporting Support Machine Maintenance in Wide-area Indo Environment", Proc. of ICC Serv 2014, pp. 41-47. , 2014.

  Consider a case where a tablet user specifies parts of a virtual object displayed on the screen of the tablet in a situation where the video of the HMD user is shared by the tablet using the mixed reality system. At this time, the HMD user voluntarily changes the viewpoint position / orientation, and therefore the tablet user who is sharing the video cannot control the viewpoint position / orientation with respect to the virtual object. That is, the tablet display screen changes regardless of the intention of the tablet user, and it is difficult to accurately specify a desired component.

  The present invention has been made in view of such a problem, and displays an image of a mixed reality space viewed by a user wearing a head-mounted display device such as an HMD on another device. Provided is a technique that enables a virtual object in a shared video to be easily and accurately designated on the display screen of the device when sharing.

  One aspect of the present invention is a system including an image display device, a holding device that holds data of a virtual object, and a display device that displays an image of a mixed reality space including an image of a virtual object based on the data. Then, the image display device acquires the image of the mixed reality space displayed on the display device and the display device when the first mode is set, and displays the image on the display device. When the mode is switched from the first mode to the second mode, a display control unit that generates an image of a virtual object using the data and displays the image on the display unit; In the mode, a transmission unit that specifies a part corresponding to a portion specified by the user with respect to the virtual object displayed on the display unit, and transmits information of the specified part to the holding device. The holding device includes a receiving unit that receives the information, and an updating unit that specifies a part corresponding to the information among the parts forming the virtual object and updates the data of the specified part. Is characterized by.

  According to the configuration of the present invention, when a video image of a mixed reality space viewed by a user wearing a head-mounted display device such as an HMD is displayed and shared on another device, the shared image is shared. It is possible to easily and accurately specify a virtual object in a moving image on the display screen of the device.

The figure which shows the structural example of a system. The figure which shows a mode that the image of mixed reality space is shared. The figure which shows the transition of the screen of a tablet terminal device. The figure explaining the process of step S912. The block diagram which shows the function structural example of a system. The flowchart of the process which a system performs. The figure explaining designation of a virtual part. The block diagram which shows the function structural example of a system. The flowchart of the process which a system performs. The block diagram which shows the function structural example of the system which concerns on 3rd Embodiment.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In addition, the embodiment described below shows an example of a case where the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
The system according to the present embodiment is a system including an image display device, a holding device that holds data of a virtual object, and a display device that displays an image of a mixed reality space including an image of a virtual object based on the data. is there. In the system having such a configuration, the image display device, when the first mode is set, acquires and displays the image of the mixed reality space displayed on the display device, and displays the image. When the mode is switched to the second mode, an image of the virtual object is generated and displayed using the data (display control), and among the parts forming the virtual object, the virtual object displayed in the second mode. With respect to, the part corresponding to the location designated by the user is specified, and the information of the specified part is transmitted to the holding device. On the other hand, the holding device receives the information, specifies the part corresponding to the information among the parts forming the virtual object, and updates the data of the specified part.

  First, a configuration example of the system according to this embodiment will be described with reference to FIG. As shown in FIG. 1, the system according to the present embodiment includes HMDs 150 and 160, HMD control devices 110 and 120, a scene data management server 130, a wireless LAN access point 170, and tablet terminal devices 180 and 190.

  The HMDs 150 and 160 are examples of head-mounted display devices, and present an image of a mixed reality space that is a combination of an image of a real space and an image of a virtual space in front of the eyes of a user who wears the HMD 150 or 160 on the head. It is a device that does.

  The HMD control devices 110 and 120 are devices for controlling basic operations of the HMDs 150 and 160, including power management of the HMDs 150 and 160 and communication by the HMDs 150 and 160, respectively. Communicate.

  The scene data management server 130 is a device that holds data (scene data) 131 necessary for constructing a virtual space (scene). The HMD control devices 110 and 120 and the scene data management server 130 may be separate devices as shown in FIG. 1 or may be one integrated device. Examples of devices applicable to the HMD control devices 110 and 120 and the scene data management server 130 include general PCs (personal computers).

  The wireless LAN access point 170 functions as a wireless LAN access point for the tablet terminal devices 180 and 190, and the tablet terminal devices 180 and 190 transmit the scene data management server 130 and the HMD 150, via the wireless LAN access point 170. Data communication with 160 (via HMD control devices 110 and 120) is performed.

  The tablet terminal devices 180 and 190 are devices having a touch panel screen, displaying various information on the touch panel screen, and having a function of detecting a user's operation on the touch panel screen. The tablet terminal devices 180 and 190 may be any devices as long as they have a function of receiving various instructions from the user and a function of displaying various information.

  As shown in FIG. 1, the tablet terminal devices 180 and 190 and the wireless LAN access point 170 are connected by a wireless LAN, and the HMD control devices 110 and 120, the scene data management server 130, the wireless LAN access point 170, Are connected to a network so that they can communicate data with each other.

  In the present embodiment, the HMD 150 (160) captures an image of the physical space, generates an image of the virtual space according to the position and orientation of the HMD 150 (160) using the scene data 131, and captures the captured image of the physical space. A mixed reality space image is generated by combining the generated virtual space image with the generated virtual space image, the generated mixed reality space image is displayed, and the mixed reality space image is displayed among the tablet terminal devices 180 and 190. It transmits to one or both associated with the HMD 150 (160). As a result, for example, on the display screen of the tablet terminal device 180, the video of the mixed reality space displayed on the HMD associated with the tablet terminal device 180 is displayed, and as a result, the tablet terminal device 180 is displayed. The device 180 can share an image of the mixed reality space with the HMD. The manner in which the video of the mixed reality space is shared between the HMD and the tablet terminal device will be described with reference to FIG.

  The user 230 wears the HMD on the head, and a projected image obtained by projecting the mixed reality space including the virtual object 220 onto the virtual screen 231 is displayed on the screen of the HMD. The markers 211 and 212 are provided on the wall or floor of the real space in order to obtain the position and orientation of the HMD. Here, the HMD picks up an image of the real space, and the markers shown in the image are used. It seeks to determine its own position and orientation.

  The video of the mixed reality space displayed on the HMD worn on the head of the user 230 is also transmitted to the tablet terminal device 250 associated with the HMD, and displayed on the display screen 215 of the tablet terminal device 250. Is displayed. This allows the user 240 to confirm (share) the image of the mixed reality space that the user 230 is observing on the display screen 215.

  Next, FIG. 5 shows a functional configuration example of each of the scene data management server 130, the HMD 150 (160), and the tablet terminal device 180 (190). Note that in FIG. 5, the HMD control devices 110 and 120 are not shown for the sake of simplicity, and the HMD 150 (160) performs direct data communication with the scene data management server 130 and the tablet terminal device 180 (190). However, in reality, as shown in FIG. 1, the HMD 150 (160) transmits data to and from the scene data management server 130 and the tablet terminal device 180 (190) via the HMD control devices 110 and 120, respectively. Communicate. Each process performed by each functional unit shown in FIG. 5 will be described with reference to the flowchart of FIG.

<Step S621>
The image capturing unit 522 of the HMD 150 (160) captures an image of the physical space.

<Step S622>
The acquisition unit 524 included in the HMD 150 (160) acquires the position and orientation of the HMD 150 (160). There are various possible methods for acquiring the position and orientation of the HMD 150 (160), and any method may be adopted. For example, as shown in FIG. 2, markers are arranged in the real space, and the HMD 150 (using a natural feature) displayed in the image acquired by imaging such a real space in step S621 is used. The position and orientation of 160) may be obtained. Further, the position and orientation of the HMD 150 (160) may be obtained based on the measurement result of a sensor such as an infrared sensor, a magnetic sensor, an ultrasonic sensor, an optical sensor, or a gyro sensor. Further, the position and orientation of the HMD 150 (160) may be obtained using the marker in the image and the measurement result of the sensor. The "position and orientation of the HMD 150 (160)" here means "the world coordinate system (one point in the real space is the origin, and three axes orthogonal to each other at the origin are the x-axis and the y-axis, respectively). The position / orientation of the image capturing unit 522 in the z-axis coordinate system), and may be referred to as “viewpoint position / orientation” below.

<Step S624>
The video composition unit 525 first builds a virtual space using the “scene data 131 acquired (downloaded) from the scene data management server 130 by the acquisition unit 526 in step S630” ”. The scene data 131 defines data of each virtual object forming the virtual space (data such as shape and texture necessary for drawing the virtual object), position and orientation data of each virtual object, and a light source for illuminating the virtual space. Data that defines the virtual space, such as data to be created. Next, the image composition unit 525 generates an image of the virtual space viewed from the viewpoint having the position and orientation acquired in step S622. A technique for generating an image of a virtual space that can be viewed from a viewpoint given a position and orientation is well known, and therefore description thereof will be omitted.

<Step S625>
The image composition unit 525 generates an image of mixed reality space by superimposing the image of the virtual space generated in step S624 on the image of the physical space captured by the image capturing unit 522 in step S621. Since a technique for generating an image in a mixed reality space is well known, description of the technique will be omitted.

<Step S626>
The video composition unit 525 outputs the video of the mixed reality space generated in step S625 to the display unit 523 to display the video on the display unit 523. The display unit 523 is attached to the HMD 150 (160) so as to be positioned in front of the eyes of the user who wears the HMD 150 (160) on the head. As a result, the image of the mixed reality space generated in step S625 can be presented in front of the eyes of the user wearing the HMD 150 (160) on the head.

<Step S627>
The video transmission unit 528 transmits the video of the mixed reality space generated by the video synthesis unit 525 in step S625 to one or both of the tablet terminal devices 180 and 190 associated with the HMD 150 (160). The tablet terminal device to which the video of the mixed reality space generated in the HMD 150 (160) is to be transmitted may be set in advance, or may be set at an arbitrary timing on the HMD 150 (160) side and / or the tablet terminal device side. You can set it with. In addition, such a setting is performed for the HMD control device 110 (120), and the video of the mixed reality space received by the HMD control device 110 (120) from the HMD 150 (160) is transmitted according to the setting. It may be transmitted to the destination. In addition, the video transmission unit 528 transfers the video of the mixed reality space to JPEG or H.264. You may make it transmit after encoding by H.264 or the like. Further, the video of the mixed reality space may be transmitted using a streaming protocol such as RTP.

<Step S651>
The control unit 590 confirms which of the MR mode (shared mode) and the VR mode is set as the operation mode of the tablet terminal device 180 (190). Various methods can be considered as a method of switching the operation mode of the tablet terminal device 180 (190), and the method is not limited to a specific method. For example, when the display unit 536 and the input unit 537 form a touch panel screen, a button image for switching to the MR mode (MR mode button image) and a button image for switching to the VR mode (VR mode button image). Is displayed on the display unit 536. When the input unit 537 detects that the user taps the MR mode button image on the touch panel screen, the control unit 590 switches the operation mode of the tablet terminal device 180 (190) to the MR mode. On the other hand, when the input unit 537 detects that the user taps the VR mode button image on the touch panel screen, the control unit 590 switches the operation mode of the tablet terminal device 180 (190) to the VR mode. Such a mode switching request can be detected by another process (not shown) for detecting a user input event.

  Then, if the operation mode of the tablet terminal device 180 (190) is the MR mode, the process proceeds to step S652, and if it is the VR mode, the process proceeds to step S660.

<Step S652>
The video reception unit 534 receives the video in the mixed reality space transmitted by the video transmission unit 528. When the received mixed reality space video is encoded, the video is decoded.

<Step S653>
The generation unit 535 generates a display screen by converting the video of the mixed reality space received in step S652 into a video suitable for the display unit 536. For example, the size of the video of the mixed reality space received in step S652 is converted according to the resolution of the display unit 536. In this step, in addition to this, a dialog screen including a display area for displaying an image of the mixed reality space and an operation area in which buttons for performing various operations are arranged is displayed on the display screen. May be generated as.

<Step S654>
The generation unit 535 displays the display screen generated in step S653 on the display unit 536. As a result, the image of the same content as the mixed reality space image displayed on the HMD side in step S626 is displayed on the display unit 536, and the image of the mixed reality space can be shared with the HMD. ..

<Step S660>
The video receiving unit 534 stops the operation of receiving the video of the mixed reality space transmitted from the video transmitting unit 528. If the operation mode is the VR mode and the video sharing is already stopped, the process of step S660 may be skipped.

<Step S661>
When the input unit 537 detects an input event indicating that the user has performed an operation input for changing the position and orientation of the locally controllable viewpoint (local viewpoint) on the tablet terminal device 180 (190) side, The process proceeds to step S662, and when the input event indicating that the user has performed the operation input for designating a part (part) of the virtual object displayed on the display unit 536, the process is performed. The process proceeds to step S665, and if no input event is detected, the process proceeds to step S663.

<Step S663>
The generation unit 535 first builds a virtual space using the “scene data 131 acquired (downloaded) from the scene data management server 130 by the acquisition unit 532 in step S640” ”. Next, the generation unit 535 generates an image of the virtual space viewed from the local viewpoint. Note that the control unit 590 determines the position / orientation of the HMD (HMD associated with the tablet terminal device 180 (190)) immediately before switching to the VR mode as the position / orientation of the local viewpoint immediately after switching to the VR mode. Set from the HMD. In this setting, not only the position and orientation but also other parameters such as the angle of view may be set. Further, the HMD controller 110 (120) may manage parameters such as the position and orientation of the HMD 150 (160) and the angle of view. In that case, the control unit 590 controls the position and orientation and the angle of view of the HMD 150 (160). The parameters such as are acquired from the HMD control device 110 (120).

<Step S664>
The generation unit 535 displays the image of the virtual space generated in step S663 on the display unit 536.

<Step S662>
The input unit 537 changes the position / orientation of the local viewpoint in response to a change operation of the position / orientation of the local viewpoint performed by the user (for example, an operation such as dragging on the screen of the display unit 536 or zooming in / out). However, after that, in step S663, an image of the virtual space according to the changed position and orientation is generated, and in step S664, an image of the virtual space according to the changed position and orientation is displayed on the display unit 536. Become.

  For example, in the tablet terminal device in which the MR mode is set, as shown in FIG. 3A, the display unit 536 displays an image of the real space including the marker 311 and an image of the virtual space including the virtual object 312. It is assumed that an image of the mixed mixed reality space is displayed. At this time, when the operation mode of the tablet terminal device is switched to the VR mode, in step S664 immediately after that, as shown in FIG. 3B, the virtual object viewed from the local viewpoint having the same viewpoint position and orientation as in FIG. The image of 312 is displayed on the display unit 536. Here, it should be noted that in the VR mode, the image in the real space is not superimposed and displayed. Then, in the tablet terminal device in which the VR mode is set, the user can perform an operation of changing the position / orientation of the local viewpoint, and when the user performs an operation of changing the position / orientation of the local viewpoint, the user operates in response to the change operation. The position / orientation of the local viewpoint is changed, and the image of the virtual object based on the local viewpoint having the changed position / orientation is displayed on the display unit 536, regardless of the change in the position / orientation of the corresponding HMD. It is possible to browse the image of the virtual object on the display unit 536 by changing the position and orientation of the viewpoint independently on the tablet terminal device side.

<Step S665>
The acquisition unit 538 identifies a virtual part (a virtual part forming a virtual object) displayed at a position (for example, a tapped position) designated by the user on the display screen of the display unit 536. Various kinds of processing can be considered as the processing for the identification, and the processing is not limited to the specific method. For example, a virtual screen (which is provided as a surface whose normal direction is the line-of-sight direction) at a position separated from the position of the local viewpoint by a specified distance in the line-of-sight direction of the local viewpoint. It can be obtained from the position and orientation, and the size (width, height) of the virtual screen and the above “specified distance” are set based on the screen size of the display unit 536, the viewing angle, and the like. Each position of 3 is represented by three-dimensional coordinates, and each position on the display screen of the display unit 536 has a one-to-one correspondence with each position on the virtual screen. Therefore, first, the display screen of the display unit 536 is displayed. A position (three-dimensional coordinate) on the virtual screen corresponding to the designated position on the upper side is specified. Then, a process of obtaining an intersection of a virtual object and a straight line passing through the specified position and the position of the local viewpoint is performed. If there is no intersection, the virtual part is not displayed at the designated position, and the intersection is present. For example, the intersection point closest to the position of the local viewpoint is specified, and the virtual part including the specified intersection point is specified as the "part specified by the user".

  Then, the acquisition unit 538 acquires, from the scene data 131 acquired by the acquisition unit 532 in step S640, identification information unique to the virtual component identified as the “user-specified component” (for example, an ID unique to the virtual component). .. When there are a plurality of virtual objects in the virtual space, this identification information is information for uniquely specifying the virtual part specified as the "part specified by the user" and the virtual object to which the virtual part belongs. ..

<Step S666>
The transmission unit 539 transmits to the scene data management server 130 the identification information of the virtual part, which is the “part specified by the user”, which is acquired from the scene data 131 by the acquisition unit 538 in step S665.

<Step S667>
The control unit 590 switches the operation mode of the tablet terminal device 180 (190) from the VR mode to the MR mode. In response to the switching of the operation mode of the tablet terminal device 180 (190) to the MR mode, the video reception unit 534 restarts the reception operation of the video of the mixed reality space transmitted from the video transmission unit 528.

<Step S611>
The receiving unit 513 receives the identification information transmitted by the transmitting unit 539 in step S666.

<Step S614>
The changing unit 514 recognizes the virtual part corresponding to the identification information in the scene data 131 so that the virtual part corresponding to the identification information received by the receiving unit 513 in step S611 can be visually recognized so as to be distinguishable from other virtual parts. Process the data. For example, the color of the virtual part is changed to a color different from the colors of other virtual parts so that the virtual part corresponding to the identification information is highlighted.

<Step S615>
The sharing unit 515 transmits the scene data 131 to the HMD 150 (160) or the tablet terminal device 180 (190).

  For example, as shown in FIG. 3B, when the user taps the headlight portion while the virtual object 312 is displayed on the display screen of the tablet terminal device, the tapped virtual part is processed by the above process. Since the data of the headlight is updated so that the headlight can be visually recognized so as to be distinguishable from other virtual parts, the headlight portion 333 is highlighted as shown in FIG. 3C. The cursor 334 is displayed at the tap position. After switching to the MR mode, the headlight portion 333 is highlighted in the mixed reality space image as shown in FIG. In this way, since the scene data 131 commonly used by all the tablet terminal devices is updated so that the headlight portion 333 is highlighted, the highlighting of the headlight portion 333 is performed by the HMD and the tablet terminal device. It will be reflected in the video of the observed mixed reality space.

  In the process according to the flowchart of FIG. 6, such highlighting is performed in the MR mode as shown in FIG. 3D, but as shown in FIG. 3C, the highlighting is performed in the VR mode. The display may be performed.

  An information terminal device such as a tablet may be used instead of the HMD used in the present embodiment, or a camera installed at a fixed position may be used.

  As described above, according to the present embodiment, in the system in which the video in the mixed reality space is shared between the HMD and the tablet terminal device, the tablet terminal device side independently sets the viewpoint to observe the virtual object and the tablet terminal device. A virtual part can be designated on the device side, and the designated virtual part can be presented to an observer in another mixed reality space, and a collaborative work using the above system in which a plurality of people participate can be promoted.

  Note that the HMD control device may perform part of the processing described as being performed by the HMD, for example, generation of a virtual space image or generation of a mixed reality space image. In this way, a part of the processing described as being performed by one device may be performed by another device. In general, the system design such as which processing is to be performed by which apparatus and how many apparatuses are used is not limited to the configuration shown in FIG. 5, but as described in the first embodiment. Various configurations are conceivable as long as the processing equal to or higher than the above processing can be realized. This also applies to the second and subsequent embodiments.

<Modification 1>
In the first embodiment, in the VR mode, the virtual object can be observed from various angles by changing the position and orientation of the local viewpoint, but even if the mode is changed to the VR mode in order to obtain the same effect. The position and orientation of the viewpoint may be fixed (the position and orientation of the HMD immediately before shifting to the VR mode), and the virtual object itself may be moved or rotated. At this time, the tablet terminal device configures a virtual space by using the scene data downloaded from the scene data management server 130, and then changes the position and orientation of the virtual object defined by the scene data to generate the virtual object. Movement and rotation may be realized.

<Modification 2>
[Second Embodiment]
In the present embodiment, when the tablet terminal device switches from the MR mode to the VR mode, the virtual part is designated using the video (still image) of the mixed reality space received from the HMD before switching to the VR mode. Hereinafter, the difference from the first embodiment will be mainly described, and unless otherwise stated below, it is the same as that of the first embodiment.

  That is, the system according to the present embodiment includes an image display device, a holding device that holds data of a virtual object, and a display device that displays an image of a mixed reality space including an image of the virtual object based on the data. It is an example of a system. In such a system, the image display device acquires the image of the mixed reality space from the display device and displays it when the first mode is set, and the image display device displays the image from the first mode to the second mode. When the display is switched to, the video of the mixed reality space acquired from the display device is displayed before the switching, and the position designated by the user on the video of the mixed reality space displayed in the second mode is acquired and stored in the holding device. To send. On the other hand, the holding device receives the designated position, identifies the part corresponding to the designated position among the parts forming the virtual object, and updates the data of the identified part.

  First, designation of virtual parts will be described with reference to FIG. The tablet terminal device holds the image of the latest frame of the mixed reality space received from the HMD, and when switched to the VR mode, as shown in FIG. 7A, the image of the mixed reality space held at this point is held. (Video of the mixed reality space received from the HMD before the switching) is displayed. Also in the present embodiment, in the VR mode, the reception of the video of the mixed reality space from the HMD is stopped. On the display screen of the tablet terminal device, the user performs an enlarging operation, a reducing operation, and a parallel moving operation (drag) on the video (still image) in the mixed reality space displayed on the display screen to display the image on the display screen. The displayed image of the mixed reality space can be enlarged, reduced, or moved in parallel. In FIG. 7B, a partially enlarged image of the mixed reality space image received from the HMD is displayed. Then, as shown in FIG. 7B, when the user taps a desired virtual part 723 on the display screen (a cursor 722 is displayed at the tap position), the tap is performed thereafter as in the first embodiment. The virtual component 723 thus created is highlighted as shown in FIG.

  Next, FIG. 8 shows a functional configuration example of the system according to the present embodiment. Further, each process performed by each functional unit of such a system will be described with reference to the flowchart of FIG.

  The processes in steps S921 to S930 are the same as those in steps S621 to S630, and therefore the description of these steps will be omitted. Further, the processing in each step of steps S951 to S960 and S967 is the same as the above-mentioned steps S651 to S660 and S667, respectively, and therefore the description of these steps will be omitted.

<Step S961>
The input unit 537 indicates that the user has performed an operation input (a gesture input on the display screen, for example, a pinch-in / out operation) such as enlarging, reducing, or translating the image of the mixed reality space displayed on the display screen. If an input event is detected, the process proceeds to step S962, and an input event indicating that the user has performed an operation input for designating a part (part) of the virtual object displayed on the display unit 536. If so, the process proceeds to step S965. If no input event is detected, the process proceeds to step S964.

<Step S962>
The generation unit 535 displays the display unit 536 in accordance with the user's operation such as enlargement, reduction, or parallel movement of the image of the mixed reality space (eg, dragging on the screen of the display unit 536, zooming in / out, or the like). The image of the mixed reality space displayed on the display screen is enlarged, reduced, or moved in parallel.

<Step S964>
The generation unit 535 displays the video of the mixed reality space on the display unit 536. In step S964 immediately after shifting to the VR mode, the image (still image) of the mixed reality space received by the image receiving unit 534 from the HMD immediately before shifting to the VR mode is displayed on the display unit 536.

<Step S965>
The acquisition unit 538 acquires a position (for example, a tapped position) designated by the user on the display screen of the display unit 536.

<Step S966>
The transmission unit 539 transmits the designated position acquired by the acquisition unit 538 in step S965 to the scene data management server 130.

<Step S911>
The receiving unit 513 receives the designated position transmitted by the transmitting unit 539 in step S966.

<Step S912>
The identifying unit 817 identifies the virtual component (virtual component forming the virtual object) displayed at the designated position received by the receiving unit 513 in step S911. Various kinds of processing can be considered as the processing for the identification, and the processing is not limited to the specific method. As an example, the method shown in FIG. 4 can be considered.

  For example, it is assumed that the user taps on the screen 450 of the tablet terminal device to point to the headlight portion of the virtual object 451, and the position 452 is I (Iu, Iv). The position 452 is in a coordinate system I (u, v) that defines the u-axis in the horizontal direction and the v-axis in the vertical direction with the origin (0,0) at the upper left corner of the screen 450.

  Next, the line of sight is separated from the position V (Vx, Vy, Vz) of the viewpoint 441 of the HMD mounted on the head of the user 440 by a specified distance in the direction of the line of sight θ (α, β, γ) of the viewpoint 441. The position 431 corresponding to the position 452 is obtained on the virtual screen 430 provided as a surface whose direction is the normal direction. Here, the position V (Vx, Vy, Vz) represents the x coordinate, the y coordinate, and the z coordinate in the world coordinate system, and the line-of-sight direction θ (α, β, γ) is the x axis, the y axis, and the z axis. Indicates the rotation angle around each axis. A method other than this can be considered as a method of expressing the viewpoint direction.

  The position and orientation of the virtual screen 430 in the world coordinate system can be obtained from the position and orientation V, θ of the viewpoint 441. Further, the size (width, height) of the virtual screen 430 and the above “specified distance” are set based on the image size of the image captured by the imaging unit 522, the screen size of the display unit 536, the viewing angle, and the like.

  Here, in a state in which any of the images in the mixed reality space displayed on the screen 450 of the tablet terminal device is not enlarged, reduced, or moved in parallel (for example, a state immediately after switching to the VR mode), for example, If the vertical and horizontal size of the virtual screen 430 is H × W (pixels) and the vertical and horizontal size of the screen 450 is h × w (pixels), the position 431I ′ (Iu ′) corresponding to the position 452I (Iu, Iv) on the screen 450 is displayed. , Iv ′) can be uniquely specified as Iv ′ = H / h × Iv and Iu ′ = W / w × Iu. Here, the position 431I ′ (Iu ′, Iv ′) is the virtual screen coordinate system (the position of the upper left corner of the virtual screen 430 is the origin (0, 0), the u ′ axis is in the horizontal direction, and the v ′ axis is in the vertical direction. In a coordinate system I '(u', v ')) that defines

  On the other hand, when the image of the mixed reality space displayed on the screen 450 of the tablet terminal device is enlarged, reduced, or moved in parallel, what is displayed on the screen 450 is enlarged or reduced on the virtual screen 430. Since it is an image in a window that has a size corresponding to and is translated by a translation operation (actually, an image in the real space is superimposed on this image), it corresponds to the position 452 as described above. The position in the window can be specified.

  For example, the vertical / horizontal size of the virtual screen 430 is H × W (pixels), the vertical / horizontal size of the screen 450 is h × w (pixels), and in the screen 450, an image of the original mixed reality space (displayed on the screen 450 when shifting to the VR mode). The ratio (enlargement / reduction ratio) of the enlargement / reduction performed on the image of the mixed reality space (R) (0 <R), the position of the upper left corner of the window on the virtual screen 430 (a, b), the screen If the position 452 on 450 is (p, q), the position 431 (x, y) can be calculated based on the following formula.

x = a + W × p / (w × R)
y = b + H × q / (h × R)
In this way, even if the image in the mixed reality space is enlarged, reduced, or moved in parallel, the position on the virtual screen 430 and the position on the screen 450 of the tablet terminal device are in one-to-one correspondence. Therefore, the position 431 on the virtual screen 430 corresponding to the position 452 can be uniquely obtained. That is, the two-dimensional coordinates I ′ (Iu ′, Iv ′) in the virtual screen coordinate system of the position 431 corresponding to the position 452 can be uniquely obtained.

  Further, since each position on the virtual screen 430 is also represented by three-dimensional coordinates, the three-dimensional coordinates S (Sx, Sy, Sz) of the position 431 in the world coordinate system and the position V (Vx, Vy, of the viewpoint 441). Vz), the intersection of a straight line passing through and the virtual object 420 is obtained, and the three-dimensional coordinates P (Px, Py, Pz of the intersection 421 that is closest to the position V (Vx, Vy, Vz) of the viewpoint 441 among the obtained intersections. ) Can be specified as “the position on the virtual object 420 corresponding to the position 452 specified by the user on the screen 450”.

<Step S914>
The changing unit 514 processes the data of the specific virtual part in the scene data 131 so that the virtual part (specific virtual part) specified in step S912 can be visually recognized so as to be distinguishable from other virtual parts. For example, the color of the virtual part is changed to a color different from the colors of other virtual parts so that the virtual part specified in step S912 is highlighted.

<Step S915>
The sharing unit 515 transmits the scene data 131 to the HMD 150 (160).

<Modification>
In the second embodiment, one still image (video in mixed reality space) received by the video receiving unit 534 from the HMD was displayed on the display unit 536 immediately before switching to the VR mode, but switching to the VR mode. From the time point to the specified time (or within the specified time before and after that time), the plurality of still images (videos in the mixed reality space) received by the video receiving unit 534 from the HMD, or among them, the input unit 537 by the user. The still image selected by using may be displayed on the display unit 536.

[Third Embodiment]
As described in the first embodiment, some devices may be integrated. For example, as shown in FIG. 10, the HMD 150 (160) and the scene data management server 130 may be integrated into one HMD 150 (160). Of course, the configuration of the HMD 150 (160) in FIG. 10 may be configured by the HMD 150 (160) and the HMD control devices 110 and 120. This is as described in the first embodiment.

[Fourth Embodiment]
Each of the functional units shown in FIGS. 5, 8 and 10 may be entirely configured by hardware, but a part may be configured by software. For example, in the case of the scene data management server 130 of FIG. 5, the receiving unit 513, the sharing unit 515, and the changing unit 514 (in addition to this in the case of FIG. 8, the specifying unit 817) may be implemented by a computer program. In this case, this computer program is stored in the memory of the scene data management server 130, and the CPU of the scene data management server 130 executes the computer program. In addition, for example, in the case of the HMD 150 (160) of FIG. 5, the acquisition unit 526, the acquisition unit 524, the video composition unit 525, and the video transmission unit 528 may be implemented by a computer program. In this case, this computer program is stored in the memory of the HMD 150 (160), and the CPU of the HMD 150 (160) executes the computer program. Further, for example, in the case of the tablet terminal device 180 (190) in FIG. 5, the video reception unit 534, the generation unit 535, the acquisition unit 538, the transmission unit 539, and the acquisition unit 532 may be implemented by a computer program. In this case, this computer program is stored in the memory of the tablet terminal device 180 (190), and the CPU of the tablet terminal device 180 (190) executes the computer program.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  150, 160: HMD 130: Scene data management server 180, 190: Tablet terminal device

One aspect of the present invention is an image display device connected to a holding device that holds data of a virtual object, and a display device that displays an image of a mixed reality space including an image of a virtual object based on the data. ,
Display means,
Display control means for performing the first processing when the first mode is set and performing the second processing when the first mode is switched to the second mode;
Among the parts forming the virtual object, the part corresponding to the portion designated by the user with respect to the virtual object displayed on the display unit in the second mode is specified, and the information of the specified part is specified. A transmitting means for transmitting to the holding device,
In the first processing, the display control means
Acquiring an image of the mixed reality space displayed on the display device and displaying the image on the display means,
In the second processing, the display control means
Unless a user operation for changing the position of the viewpoint is input, an image of a virtual object based on the viewpoint whose position is fixed and the data is displayed on the display means, and when the user operation is input, the user operates. It is characterized in that an image of a virtual object based on the viewpoint and a position of a position changed according to an operation is displayed on the display means .

Claims (12)

A system having an image display device, a holding device for holding data of a virtual object, and a display device for displaying an image of a mixed reality space including an image of a virtual object based on the data,
The image display device,
Display means,
When the first mode is set, an image of the mixed reality space displayed on the display device is acquired and displayed on the display means, and the first mode is switched to the second mode. Display control means for generating an image of a virtual object using the data and displaying the image on the display means,
Among the parts forming the virtual object, the part corresponding to the portion designated by the user with respect to the virtual object displayed on the display unit in the second mode is specified, and the information of the specified part is specified. A transmitting means for transmitting to the holding device,
The holding device is
Receiving means for receiving the information,
An updating unit that specifies a part corresponding to the information among the parts forming the virtual object and updates the data of the specified part.   The image display device according to claim 1, wherein the image display device acquires the data of the virtual object from the holding device. The display control means,
The image display according to claim 1, wherein when the mode is switched to the second mode, the image of the virtual object is generated based on the position and orientation of the display device before the mode is switched to the second mode. apparatus. The display control means,
The image of the virtual object is generated based on the changed position and orientation in response to an operation for changing the position and orientation of the viewpoint in the second mode. The image display device described in 1. A system having an image display device, a holding device for holding data of a virtual object, and a display device for displaying an image of a mixed reality space including an image of a virtual object based on the data,
The image display device,
Display means,
When the first mode is set, an image of the mixed reality space is acquired from the display device and displayed on the display unit, and when the first mode is switched to the second mode, the switching Display control means for displaying an image of the mixed reality space previously acquired from the display device on the display means,
Transmission means for acquiring a position designated by the user on the video of the mixed reality space displayed on the display means in the second mode and transmitting the position to the holding device,
The holding device is
Receiving means for receiving the indicated position,
An updating unit that specifies a part corresponding to the designated position among the parts forming the virtual object and updates the data of the specified part. The image display device further includes
The system according to claim 5, further comprising means for inputting an enlarging operation, a reducing operation, and a parallel moving operation with respect to the image of the mixed reality space displayed on the display means in the second mode. The updating means is
7. The system according to claim 5, wherein the position and orientation of the display device and the designated position are used to identify a part corresponding to the designated position among the parts forming the virtual object. .. The updating means is
The system according to any one of claims 1 to 7, wherein data of the specified part is updated so that the specified part is highlighted.   9. The system according to claim 1, wherein the system includes a plurality of at least one of the image display device and the display device.   10. The system according to any one of claims 1 to 9, wherein the display device is a head-mounted display device. A control method for a system, comprising: an image display device, a holding device for holding data of a virtual object, and a display device for displaying an image of a mixed reality space including an image of a virtual object based on the data,
The image display device,
When the first mode is set, the image of the mixed reality space displayed on the display device is acquired and displayed on the display unit of the image display device, and the second mode is changed from the first mode. When switched to the mode, an image of the virtual object is generated using the data and displayed on the display means,
Among the parts forming the virtual object, the part corresponding to the portion designated by the user with respect to the virtual object displayed on the display unit in the second mode is specified, and the information of the specified part is specified. The holding device transmits to the holding device,
Receiving the information,
A method of controlling a system, characterized in that a part corresponding to the information is specified from among parts forming the virtual object, and data of the specified part is updated. A control method for a system, comprising: an image display device, a holding device for holding data of a virtual object, and a display device for displaying an image of a mixed reality space including an image of a virtual object based on the data,
The image display device,
When the first mode is set, an image of the mixed reality space is acquired from the display device, displayed on the display means of the image display device, and the first mode is switched to the second mode. , Displaying an image of the mixed reality space acquired from the display device before the switching on the display means,
In the second mode, the position designated by the user on the video of the mixed reality space displayed on the display means is acquired and transmitted to the holding device,
The holding device is
Receiving the indicated position,
A method of controlling a system, characterized in that a part corresponding to the designated position is specified from among parts forming the virtual object, and data of the specified part is updated.

Images