JP6765846B2 - Information processing equipment, information processing methods, and programs - Google Patents

Description

The present invention relates to an information processing apparatus, an information processing method, and a program, and is particularly suitable for use in displaying an image in a virtual real space so as to overlap the real space.

In recent years, research on mixed reality (MR) has been actively conducted for the purpose of seamless connection between real space and virtual space. An image display device that presents mixed reality is, for example, a device having the following configuration. That is, an image of the virtual real space generated according to the position and orientation of the image pickup device (for example, a virtual object or character information drawn by computer graphics) on the image of the real space captured by an image pickup device such as a video camera. It is a device that displays an image drawn by superimposing. For such a device, for example, an HMD (head-mounted display, head-mounted display) can be used.

The image display device is also realized by an optical see-through method. That is, the image display device is also realized by displaying an image of the virtual reality space generated according to the position and orientation of the viewpoint of the observer on the optical see-through type display mounted on the observer's head. .. In this case, the image of the virtual real space is displayed so as to overlap the actual real space, not the image of the real space.

As an application of such an image display device, it is conceivable to superimpose a product under development in a manufacturing industry such as an automobile or an aircraft as a computer graphics (CG) image on a real space. In this way, by confirming and examining the product from a free viewpoint, it is possible to examine the design and the like without producing a full-scale model. As a technique for designating the position of such CG, there is a technique described in Patent Document 1. In the technique described in Patent Document 1, a main pointer indicating one point of CG is displayed, and an auxiliary pointer is displayed at a distance from the main pointer. When the position of the primary pointer moves, the auxiliary pointer also moves while maintaining the distance from the primary pointer. When the auxiliary pointer overlaps with the virtual object on the CG, the part of the auxiliary pointer that overlaps with the virtual object is hidden, and the user is made to grasp the distance between the main pointer and the virtual object.

JP-A-6-83935

However, in the technique described in Patent Document 1, the part of the image in the virtual reality space can be specified only in one direction (depth direction). Therefore, in a direction other than the relevant direction, the experiencer may specify an erroneous part. In the technique described in Patent Document 1, there is no means of correction in such a case.
The present invention has been made in view of such a problem, and makes it possible to easily and surely adjust a selected part with respect to an image of a virtual real space displayed overlaid on a real space. The purpose is.

The information processing device of the present invention comprises a designation means for designating a part of an image in a virtual reality space displayed by a display device so as to overlap the real space based on a first instruction by a user, and a position and orientation of the display device. The specific means for specifying the above, the image part of the virtual reality space designated by the designated means , the position and orientation of the display device specified by the specific means, and the content of the second instruction by the user. The second instruction is an instruction different from the redoing of the first instruction, and the user who uses the display device is displayed by the display device. It is characterized in that the instruction can be given while looking at the image of the virtual reality space.

According to the present invention, it is possible to easily and surely adjust the selected portion with respect to the image of the virtual real space displayed overlaid on the real space.

It is a figure which shows the structure of an image display system. It is a figure which shows the 1st example of the functional configuration of a virtual reality display device. It is a flowchart explaining the process of an image display system. It is a figure which shows an example of the arrangement of a virtual object. It is a figure which shows the 2nd example of the functional configuration of a virtual reality display device. It is a figure which shows the structure of a CG model.

Hereinafter, embodiments will be described with reference to the drawings.
(First Embodiment)
First, the first embodiment will be described.
FIG. 1 is a diagram showing an example of the configuration of an image display system. The image display system includes a virtual reality display device 101, an HMD (Head Mounted Display) 102, and a virtual laser device 103.

The virtual reality display device 101 and the HMD 102 are connected to each other by a communication path 104. The communication path 104 is a communication path used for communication between the virtual reality display device 101 and the HMD 102. The communication includes, for example, transmitting an image generated by the virtual reality display device 101 to the HMD 102, and transmitting an image of the real space acquired by the HMD 102 to the virtual reality display device 101. In the communication path 104, for example, an image transmission / reception protocol such as HDMI (registered trademark) or IEEE 1394 is used. In addition, USB (Universal Serial Bus), Ethernet (registered trademark), or the like may be used for the communication path 104. The virtual reality display device 101 and the HMD 102 may be incorporated into the same computer, and the communication path 104 may be used as a communication path between software processes.

Similarly, the virtual reality display device 101 and the virtual laser device 103 are connected to each other by a communication path 105. The communication path 105 is a communication path used for communication between the virtual laser device 103 and the virtual reality display device 101. For example, the position / orientation information of the virtual laser device 103 is transmitted from the virtual laser device 103 to the virtual reality display device 101 via the communication path 105. In the communication path 105, for example, a wireless communication protocol such as Bluetooth (registered trademark) is used. In addition, a wired connection such as USB may be used for the communication path 105. The virtual laser device 103 may be incorporated in the same computer as the virtual reality display device 101, and the communication path 105 may be used as a communication path between software processes.

FIG. 2 is a block diagram showing an example of a functional configuration of the virtual reality display device 101. As described above, the image display system of the present embodiment includes a virtual reality display device 101, an HMD 102, and a virtual laser device 103. The HMD 102 is an example of a display device, and is a head-mounted imaging / display device. In the present embodiment, a head-mounted display with a built-in camera is used, but a hand-held handheld display may be used. In the head-mounted display with a built-in camera, when the experiencer wears the display with a built-in camera on the head, he / she sees an image in which the CG of the virtual real space is superimposed on the image of the real space captured by the camera. Can be done. By wearing a head-mounted display with a built-in camera, the field of vision of the experiencer is limited. On the other hand, in the handheld handheld display, the experiencer can look into the handheld display and see an image in which the CG of the virtual real space is superimposed on the image of the real space captured by the camera. In this case as well, the field of view of the experiencer is limited. Further, an optical see-through type (optical transmission type) HMD may be used. That is, the CG of the virtual real space may be displayed so as to overlap the actual real space instead of the image of the real space captured by the camera. Regardless of which of the above-mentioned display devices is used, only the experienced person (user who uses the display device) can see the image displayed by the display device.

The virtual laser device 103 is used by the experiencer wearing the HMD 102 to select a position on the CG in the virtual reality space. In this embodiment, the case where one person can wear the (one) HMD 102 is shown as an example. The experiencer wearing the HMD 102 holds the virtual laser device 103 in his hand, and while looking at the image displayed by the HMD 102 in which the CG of the virtual real space is superimposed on the image of the real space, the CG is displayed on the CG. Select a position. Here, the virtual reality display device 101 and the HMD 102 may be independent computer devices, or the virtual reality display device 101 may be built in the housing attached to the HMD 102. The virtual reality display device 101 may be configured as software that operates on a computer device built in the HMD 102.

The virtual reality display device 101 includes an imaging unit 201, a head position / orientation specifying unit 202, an MR image display unit 203, a composite part designation unit 204, a CG part designation unit 205, a voice command analysis unit 206, and a designated part correction unit 207. Including, various information processing is performed. The hardware of the virtual reality display device 101 is realized, for example, by using a device including a CPU, ROM, RAM, HDD, and various interfaces.
The imaging unit 201 inputs an image of the field of view that can be seen from the HMD 102 (camera) to the system as an image of the real space through a camera or the like built in the HMD 102.

The head position / posture specifying unit 202 specifies the three-dimensional coordinates and the posture of the HMD 102 in the real space. The head position / orientation specifying unit 202 specifies the three-dimensional coordinates and the posture of the HMD 102 in the real space by using the information of the markers and feature points existing in the image of the three-dimensional real space acquired by the imaging unit 201. Can be done. In addition, the head position / orientation specifying unit 202 identifies the three-dimensional coordinates and posture of the HMD 102 in the real space by using information such as an infrared marker, a magnetic sensor, and a gyro sensor connected to the virtual reality display device 101. May be good. In addition, the head position / posture specifying unit 202 may specify the three-dimensional coordinates and the posture of the HMD 102 in the real space by using the information from a plurality of sensors that detect the HMD 102.

The MR image display unit 203 generates an image in which the shape of a virtual object or the like is superimposed as a CG in the virtual real space on the image in the real space acquired by the imaging unit 201. In the following description, an image in which CG in the virtual real space is superimposed on the image in the real space is referred to as an MR image, if necessary. By viewing this MR image through the HMD 102, the experiencer can experience as if a virtual object exists in the real space. Here, the image generated by the MR image display unit 203 may be output to the HMD 102, or may be displayed on a display device other than the HMD 102. That is, the image generated by the MR image display unit 203 may be output only to the HMD 102 or may be output to both the HMD 102 and a display device other than the HMD 102. A display device other than the HMD 102 is an example of a second display device, which is realized by, for example, an external monitor, a projector, a television, or a computer display. When any of these display devices is used as a display device other than the HMD, a plurality of people (anyone) can simultaneously view the images displayed by the display device.

By displaying the image generated by the MR image display unit 203 on a display device other than the HMD 102 in this way, a person other than the experiencer can see the same image as the experiencer at the same timing. In this case, the virtual reality display device 101 is communicably connected to not only the HMD 102 but also a display device other than the HMD 102.

The CG part designation unit 205 designates a specific part on the virtual object displayed by the MR image display unit 203 based on the position and orientation of the virtual laser device 103 in the real space. The CG site designation unit 205 assumes, for example, a virtual laser beam that travels straight in a specific direction specified in advance from a position in the real space of the virtual laser device 103, and a position where the virtual laser beam comes into contact with a virtual object. Is specified as a specific part on the virtual object. As described above, in the present embodiment, for example, the first instruction is given by the operation of the virtual laser device 103 by the experiencer. However, the instruction (first instruction) for designating a specific part on the virtual object is not limited to such an instruction. For example, instead of a virtual laser beam, a virtual tool is assumed, and a position where the virtual tool and a virtual object come into contact with each other at a position advanced by a certain distance in a specific direction from the position in the real space of the virtual tool. It may be specified as (a specific part of the virtual object). Further, the specific portion on the virtual object may be continuously designated in conjunction with the movement of the virtual laser device 103, or may be designated intermittently each time the switch included in the virtual laser device 103 is pressed. May be.

The voice command analysis unit 206 acquires and analyzes the voice generated by the experiencer, and recognizes it as a voice command. Here, the voice command analysis unit 206 may acquire the voice emitted by the experiencer using the microphone built in the HMD 102, or may use an independent microphone arranged outside the HMD 102 to emit the experiencer. The voice may be acquired. Here, the voice command in the present embodiment includes a relative position as seen from the viewpoint of the experiencer, such as "left" and "right part", and corrects a part designated by the CG part designation unit 205. Instructions for.

As described above, in the present embodiment, for example, a case where the second instruction is given by the voice of the experiencer and the voice command is recognized from the voice of the experiencer will be described as an example. However, if the experiencer can instruct how to correct the part specified by the CG part designation unit 205 while looking at the MR image, the instruction to correct the part specified by the CG part designation unit 205 (second). It is not always necessary to use voice for the instruction). For example, the command may be recognized not by voice but by a gesture instruction by the experiencer's hand or an instruction by operating a virtual button in the MR image. Further, it may be instructed how to correct the part specified by the CG part designation unit 205 without using the viewpoint of the experiencer as a reference.

The designated part correction unit 207 is a CG part based on the content of the voice command recognized by the voice command analysis unit 206 and the information on the position and posture of the HMD 102 in the real space specified by the head position / posture specifying unit 202. The correction instruction to the part designated by the designation unit 205 is specified. Here, it is assumed that the voice command recognized by the voice command analysis unit 206 includes a relative direction instruction from the viewpoint of the experiencer wearing the HMD 102. Therefore, by combining the voice command with the information on the position and posture of the HMD 102 in the real space specified by the head position / orientation specifying unit 202, it is possible to correct the part specified by the CG part specifying unit 205. ..

The composite site designation unit 204 reflects the correction instruction specified by the designated site correction section 207 on the site designated by the CG site designation section 205, and finally determines the designated site. The information of the finally designated portion is displayed on the HMD 102 by the MR image display unit 203 (superimposed on the MR image). As a result, the experiencer can select and view the position to which the correction by the voice command issued by himself / herself is applied.

FIG. 3 is a flowchart illustrating an example of processing of the image display system when the experiencer specifies a specific part in the virtual object by the virtual laser device 103 and then issues a voice command to correct the position. Is.
First, when the operation of the system is started, the CG site designation unit 205 accepts the selection of the component (position) by the virtual laser device 103 (step S301). Here, the component is a component that constitutes a virtual object displayed in the MR image.

When a component is selected by the virtual laser device 103 possessed by the experiencer, the MR image display unit 203 highlights the component selected in step S301 in the MR image (step S302). Highlighting may be done by changing the color or brightness of the selected part, by displaying a virtual frame around the part, or by displaying annotations such as sticky notes. Good. If the display content of the selected part is changed, it is not always necessary to highlight it. For example, the selected part may be hidden or the position of the selected part may be moved.

After highlighting the parts, the voice command analysis unit 206 receives the correction instruction uttered by the experiencer (step S303). This correction instruction is an instruction indicating to which component the component selected in step S301 is to be changed. After that, the voice command analysis unit 206 waits for the utterance of the experiencer for a certain period of time (step S304). If there is no utterance within a certain period of time, it can be determined that the parts selected in step S302 were as intended by the experiencer, and the operation of the system is terminated. Here, the standby time is set in advance before the system is started. However, the standby time may be arbitrarily changed during system startup. In addition, a function may be provided to terminate the operation of the system by recognizing a dedicated voice command or other method, assuming that the selection of parts is appropriate before the set fixed time elapses. Similarly, assuming that the result of selecting the component in step S301 is incorrect by recognition of a dedicated voice command or other method, and that the error exceeds the range in which the component can be corrected by the voice command. A function for canceling the selection of parts may be provided.

In step S304, when the experiencer utters a voice within a certain period of time, the voice command analysis unit 206 analyzes the voice and converts it into a command (step S305).
Subsequently, the head position / posture specifying unit 202 acquires the position and posture of the HMD 102 in the real space (step S306). The position and posture of the HMD 102 in the real space correspond to the position and posture of the experiencer's head in the real space. Here, as the position and posture of the HMD 102 in the real space, the position and posture at the timing during the execution of step S306 can be acquired. Further, the position and posture of the HMD 102 in the real space may be acquired by going back to the timing when the experiencer is uttering in steps S303 to S304. Further, the MR image displayed on the HMD 102 may be stopped at a certain time by a dedicated voice command or other input method, and the position and orientation of the HMD 102 in the real space at that time may be acquired.

Next, the designated portion correction unit 207 calculates the correction direction of the component selected in step S301 (step S307). This calculation is performed using the voice command recognized in step S305 and the position and posture (head position and posture) of the HMD 102 acquired in step S306 in the real space. The method of calculating the correction direction will be described later with reference to FIG.

After the correction direction is calculated, the composite part designation unit 204 changes the selection range of the parts based on the correction direction calculated in step S307 (step S308). For example, the composite site designation unit 204 moves the selection range to the component next to the currently selected component in the direction of the correction calculated in step S307, and updates the highlighting target in step S302. Then, the MR image display unit 203 changes the parts to be highlighted in the MR image to the updated parts.

Here, the designated portion correction unit 207 may adjust the distance for moving the currently selected component according to the type of voice command such as "slightly right" or "still right". For example, by registering the relationship between the content of the voice command and the moving distance in advance, it is possible to adjust the moving distance of the currently selected part according to the type of the voice command. For example, if the voice command contains "a little", it moves to the part next to the currently selected part, and if it contains "yet", it is currently selected. The relationship can be registered to move to the part two next to the part. Further, the designated portion correction unit 207 may widen or narrow the currently selected range without deselecting the currently selected component, depending on the type of voice command. For example, by registering the relationship between the content of the voice command and the selection range of the part in advance, the current selection range can be selected according to the type of voice command without deselecting the currently selected part. Can be widened or narrowed. For example, if the voice command includes "with surrounding parts", you can register the relationship so that the currently selected part and all the parts adjacent to the part are selected. it can.

When the correction of the selected part is completed according to the voice command and the position and posture (head position and posture) of the HMD 102 in the real space, the voice command analysis unit 206 again makes a voice of the experiencer. Accept the correction instruction (step S303).
The above steps S303 to S308 are repeated until the selection range is as intended by the experiencer. Then, if there is no utterance within a certain period of time, the operation of the system is terminated.

As described above, instruct how to correct the part specified by the CG part designation unit 205 by the gesture by the experiencer's hand or the operation of the virtual button in the MR image without using the voice. May be good. When giving an instruction by a gesture by the experiencer's hand, for example, the relationship between the movement of the hand and the content of the change in the selection range of the parts is registered in advance in the virtual reality display device 101 or the like. Then, the designated portion correction unit 207 identifies the movement of the experiencer's hand from the image captured by the camera mounted on the HMD 102, and reads out the content of the change in the selection range of the component associated with the specified movement. , Change the selection range of parts according to the read contents.

Further, when giving an instruction by operating a virtual button in the MR image, the relationship between the content of the operation of the virtual button in the MR image and the content of changing the selection range of the component is described in the virtual reality display device 101 or the like. Register in advance in. The designated part correction unit 207 reads out the content of the change in the selection range of the component associated with the content of the operation performed on the virtual button in the MR image, and changes the selection range of the component according to the read content. To do. The instruction by operating the virtual button in the MR image may be given by using the virtual laser device 103 or by using a device other than the virtual laser device 103.

FIG. 4 is a diagram showing an example of arrangement of virtual objects. Here, a virtual object in which cubic parts 401 to 409 are arranged as shown in FIG. 4 will be described as an example. An example of the process in step S307 (method of calculating the correction direction of the currently selected component) will be described with reference to FIG.

In step S301, it is assumed that the experiencer selects the component 405 by using the virtual laser device 103. Then, in step S302, the component 405 is highlighted. Here, it is assumed that the command "right part" is recognized as a voice command in step S305. Then, the head position / posture specifying unit 202 acquires the position and posture of the HMD 102 in the real space in step S306. In FIG. 4, assuming that the HMD 102 is looking at the virtual object (parts 401 to 409) with the parts 401, 402, and 403 vertically upward from the front direction of the paper surface, the right side of the part 405 as seen from the experiencer. The part in the direction of is part 406. The designated part correction unit 207 specifies the direction of each virtual object when viewed from the HMD 102 from the position and orientation of the HMD 102 in the real space, and in the direction designated by the voice command for the currently selected part. Change the highlighted part to an adjacent part. That is, the part newly selected and highlighted by the processing of steps S307 and S308 becomes the part 406.

Similarly, in FIG. 4, it is assumed that the HMD 102 is looking at the virtual object (parts 401 to 409) in a posture in which the parts 401, 402, and 403 are vertically upward from the direction from the back of the paper to the front. In this case, when the command "right part" is recognized as a voice command, the part 404 is newly selected and the highlighted part is changed from the part 405 to the part 404. Further, it is assumed that the HMD 102 is looking at the virtual object (parts 401 to 409) in a posture in which the parts 407, 404, and 401 are in the vertical direction from the front direction to the back direction of the paper surface. That is, it is assumed that the experiencer who is looking at the virtual object (parts 401 to 409) from the front side to the back side of the paper surface with the parts 407, 404, and 401 in the vertical direction tilts his head 90 degrees to the left. .. In this case, the right direction of the part 405 as seen by the experiencer is the part 402, and the correction by the command "right part" is the part 402.

For the selection (picking) of parts using the virtual laser emitted from the virtual laser device 103, for example, 3DCG (3D computer graphics) is displayed using MR technology, and a plurality of parts displayed by CG are displayed. It is done when reviewing at. At this time, the selected part is highlighted, hidden, or moved to facilitate the review.

When picking, if the experiencer mistakenly selects the next part, it is necessary to ensure that the correct part is selected. Therefore, it is conceivable to re-pick the parts and reselect the parts. However, if it is difficult to select parts by picking due to the arrangement of parts or the like, there is a risk that the correct parts will not be selected even if picking is repeated. Further, instead of re-picking the parts, it is conceivable to reselect the parts by operating a PC different from the image display device that displays the 3DCG. For example, it is conceivable to display a list of parts constituting the 3DCG on the screen of the PC and instruct the image display device to change the selected parts from the PC. In this case, the experiencer cannot see the screen of the PC unless the HMD is removed, so that the desired component cannot be selected. From the above, there is a risk that smooth review of parts will be hindered.

On the other hand, in the present embodiment, the parts (parts of the virtual object) selected by the experiencer are automatically reselected for the MR image based on the position and posture of the HMD 102 and the voice command, and the MR image is displayed. Change the highlighted part to the reselected part. Therefore, the only operation that the experiencer consciously performs to adjust the selected part with respect to the image in the virtual reality space is the utterance of a voice command. Therefore, it is possible to easily and surely adjust the selected portion with respect to the image in the virtual reality space. For example, when the experiencer utters a command such as "a little more left", a desired component can be selected without operating a PC or the like. As a result, it is possible to prevent the review of the MR image (for example, a part) from being hindered by the operation for correcting the picking selection range, and it becomes possible to concentrate on the review.

(Second Embodiment)
Next, the second embodiment will be described. In the first embodiment, a case where the direction of correcting the selected component is instructed by voice has been described as an example. On the other hand, in the present embodiment, a case where the upper / lower relationship of the parts is instructed by voice will be described. As described above, the present embodiment and the first embodiment are mainly different in configuration and processing due to the difference in the content of the voice command. Therefore, in the description of the present embodiment, detailed description of the same parts as those of the first embodiment will be omitted by adding the same reference numerals as those given in FIGS. 1 to 4.

FIG. 5 is a block diagram showing an example of a functional configuration of the virtual reality display device 500. The image display system of the present embodiment includes a virtual reality display device 500, an HMD 102, and a virtual laser device 103. It is a functional block diagram which shows the 2nd Example of this invention.

The virtual reality display device 500 includes an image pickup unit 201, a head position / orientation identification unit 202, an MR image display unit 203, a composite part designation unit 501, a CG part designation unit 205, a voice command analysis unit 502, a CG model storage unit 503, and The designated part correction unit 504 is included. As described above, the HMD 102, the virtual laser device 103, the imaging unit 201, the head position / orientation specifying unit 202, the MR image display unit 203, and the CG site specifying unit 205 are the same as those described in the first embodiment, respectively. ..

The voice command analysis unit 502 acquires and analyzes the voice generated by the experiencer, and recognizes it as a voice command. The voice command in the present embodiment is an instruction regarding the configuration information of the CG model that constitutes the virtual object, such as "one higher component", and is a specific part (for example, for example) on the virtual object designated by the CG part designation unit 205. This is an instruction to correct the position of the part). In the present embodiment, a case where a voice command is recognized from the voice of the experiencer will be described as an example. However, as in the first embodiment, the command may be recognized not by voice but by a gesture by the experiencer's hand or an instruction by operating a virtual button in the MR image.

The CG model storage unit 503 stores the CG model. The CG model is data representing the shape of a virtual object to be superimposed and displayed on an image in real space. This data may be data artificially designed using software such as CAD, or may be data reconstructed from measurement data in real space using a laser scanner or multiple cameras. Good. This data consists of data of a plurality of parts constituting a virtual object, and between each part, there is a tree structure showing the inclusion relationship (that is, the upper / lower relationship) of each part.

FIG. 6 is a diagram showing an example of the structure of the CG model. The CG model has a hierarchical structure in which a plurality of parts are gathered to form one part, and the parts are gathered again to form a higher-level part.
In the example shown in FIG. 6, the copier model 601 is composed of a main body 602 and other parts. In FIG. 6, parts other than the main body 602 constituting the copying machine model 601 are omitted, but in reality, the copying machine model 601 also has parts other than the main body 602. However, the copying machine model 601 may be configured only with the main body 602.

Similarly, the main body 602 is the top panel 603 and some other parts, the top panel 603 is the operation panel 604 and some other parts, and the operation panel 604 is the liquid crystal screen 605, the start button 606, the power LED 607 and others. It consists of several parts. In this way, the hierarchical structure is defined. As described above, in the present embodiment, a case where the CG model storage unit 503 is provided in the virtual reality display device 500 and the CG model is managed by the virtual reality display device 500 is shown as an example. However, it is not always necessary to do this. For example, a CG model may be managed by a device different from the virtual reality display device 500, and the virtual reality display device 500 may refer to the CG model.

The designated part correction unit 504 transmits the content of the voice command recognized by the voice command analysis unit 502 to the part designated by the CG part designation unit 205 based on the CG model stored in the CG model storage unit 503. Identify the correction instruction. Here, it is assumed that the voice command recognized by the voice command 309 includes an instruction for correcting the selection range of the parts in the CG model. Therefore, by combining the voice command with the data structure of the CG model, it is possible to correct the selection range of the part designated by the CG part designation unit 205. The designated portion correction unit 504 can specify which part is displayed at which position in the MR image from, for example, the contents of CG or the MR registered in advance.

For example, in FIG. 6, it is assumed that the voice command recognized by the voice command analysis unit 502 is "expand the selection range to the next higher component" in the state where the start button 606 is selected by the CG part designation unit 205. .. In this case, the designated part correction unit 504 corrects the selection range of the parts to the operation panel 604. If a similar voice command is further recognized in this state, the selection range of the component is expanded to the top panel 603. Further, it is assumed that the voice command recognized by the voice command analysis unit 502 is "reduced to the next lower component" in the state where the upper surface panel 603 is selected by the CG part designation unit 205. In this case, the designated part correction unit 504 corrects the selection range of the parts to the main body 602.

There may be a plurality of correction candidates depending on the configuration of the CG model. For example, when the operation panel 604 is selected by the CG part designation unit 205 and the voice command recognized by the voice command analysis unit 502 is "reduced to one lower component", three correction candidates are available. It will exist. In this case, one part may be uniquely determined from a plurality of correction candidates by using arbitrary information (ID assigned to the part, etc.) in the CG model, or a dedicated voice command may be used. Candidates may be presented in order, and parts may be selected from the presented candidates by the operation of the experiencer. When assigning IDs to parts, for example, a plurality of parts belonging to the same hierarchy can be prioritized, and IDs having a smaller value can be assigned to the plurality of parts as the priority is higher. In this way, when a plurality of correction candidates exist because a plurality of parts belong to the same hierarchy, a part having a small ID value can be selected.

The composite site designation unit 501 reflects the correction instruction specified by the designated site correction section 504 on the site designated by the CG site designation section 205, and finally determines the designated site. The information of the finally designated portion is displayed on the HMD 102 by the MR image display unit 203 (superimposed on the MR image). As a result, the experiencer can select and view the position to which the correction by the voice command issued by himself / herself is applied.
By doing so, the selected component can be corrected in the unit of the hierarchy defined by the CG model, and the same effect as described in the first embodiment can be obtained. The configuration of the present embodiment and the configuration of the first embodiment may be combined. That is, when the voice command indicates the direction of correction, the first embodiment is applied, and when the voice command is related to the configuration information of the CG model, the second embodiment is applied. You may. Further, also in this embodiment, various modified examples described in the first embodiment can be adopted.

(Third Embodiment)
Next, a third embodiment will be described. In the first and second embodiments, the MR image display unit 203 has described the case where the MR image in which a virtual object is superimposed on the image in the real space is displayed on the HMD 102 as an example. On the other hand, the image display system of the present embodiment switches and displays the MR image displayed on each of the plurality of HMDs to a display device other than the HMD. As described above, the present embodiment and the first and second embodiments are mainly different in the configuration and processing for displaying the MR image as a display device other than the HMD. Therefore, in the description of the present embodiment, detailed description of the same parts as those of the first and second embodiments will be omitted by adding the same reference numerals as those given in FIGS. 1 to 6.

In this embodiment, a plurality of HMDs are connected in the image display system. Then, by displaying the MR image of an arbitrary HMD on a display device other than the HMD, members other than those who have experienced the HMD can participate in the review of the MR image (for example, parts). In the present embodiment, a plurality of HMDs and a display device other than the HMDs are communicably connected to the virtual reality display device. A display device other than the HMD is an example of a second display device, which is realized by, for example, an external monitor, a projector, a television, or a computer display. When any of these display devices is used as a display device other than the HMD, a plurality of people (anyone) can simultaneously view the images displayed by the display device.

For example, a review participant operates a terminal device such as a PC or a tablet to select an MR image to be displayed on a display device other than the HMD from among MR images displayed on a plurality of HMDs. The terminal device outputs the information of the HMD that outputs the selected MR image to the virtual reality display device. Then, the virtual reality display device switches and displays the MR image displayed on the display device other than the HMD with the MR image output from the HMD specified by the information. In this case, the position and orientation of the HMD in the real space acquired in step S306 is the position and orientation of the HMD in the real space that is outputting the MR image displayed on the display device other than the HMD at that time. Is. Therefore, the correction direction calculated in step S307 is the correction direction of the parts in the MR image displayed on the display device other than the HMD at that time. Further, the parts highlighted in step S308 are the parts in the MR image displayed on the display device other than the HMD at that time, which are changed based on the correction direction calculated in step S307. is there. The highlighting of the parts is performed not only on the MR image displayed on the HMD but also on the MR image displayed on a display device other than the HMD. As the method of correcting the highlighted parts, the same method as in the first embodiment or the second embodiment can be adopted.

By configuring the image display system as described above, in addition to the effects described in the first and second embodiments, MR images (for example, parts) displayed on a plurality of HMDs by members other than those who have experienced the HMD. ) Has the effect of being able to participate in the review. Also in this embodiment, various modified examples described in the first and second embodiments can be adopted.

Each of the above-described embodiments merely shows an example of embodiment in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101: Virtual reality display device, 102: HMD, 103: Virtual laser device

Claims (14)

A designation means for designating a part of an image in the virtual reality space displayed by the display device so as to overlap the real space based on the first instruction by the user.
Specific means for specifying the position and orientation of the display device, and
A correction means for correcting a portion of an image in the virtual reality space designated by the designated means based on the position and orientation of the display device specified by the specific means and the content of a second instruction by the user. Have,
The second instruction is an instruction different from the redoing of the first instruction, and the user using the display device may give the instruction while looking at the image of the virtual reality space displayed by the display device. An information processing device characterized by being a possible instruction. The information processing apparatus according to claim 1, wherein the second instruction is a voice instruction, a gesture instruction, or an operation instruction on a screen for displaying an image of the virtual reality space. The information processing device according to claim 1, wherein the second instruction is a voice instruction. Each part that constitutes the image of the virtual reality space is managed in a hierarchical structure.
The correction means is characterized in that it corrects a selection range of a portion constituting an image in the virtual reality space based on the position and orientation of the display device, the content of the second instruction, and the hierarchical structure. The information processing apparatus according to any one of claims 1 to 3 . The information processing device according to any one of claims 1 to 4 , wherein the display device superimposes an image of the virtual real space on an image of the real space captured by the imaging means. .. Further, it has a changing means for changing the display of the designated part with respect to the image in the virtual reality space by the designated means.
The changing means is characterized in that, when a part of an image in the virtual reality space is corrected by the correction means, the part to be changed in the display is updated to the corrected part. The information processing apparatus according to any one of 5 . The information processing according to any one of claims 1 to 6 , wherein the display device displays an image of the virtual reality space in a state where only a user who uses the display device can see the display device. apparatus. The information processing device according to any one of claims 1 to 7 , wherein the display device is an HMD (Head Mounted Display). Further, it has an output means for outputting an image of the virtual reality space to the display device and a second display device different from the display device.
The information processing device according to any one of claims 1 to 8 , wherein the second display device displays an image of the virtual reality space in a state where a plurality of people can view it at the same time. An image display system comprising the information processing device according to any one of claims 1 to 8 and the display device. An image display system comprising the information processing device according to claim 9 , the display device, and the second display device. It has a plurality of the display devices and
The second display device displays an image displayed on a display device selected from the plurality of display devices.
The image display system according to claim 11 , wherein the correction means corrects a portion of an image in the virtual reality space displayed by the selected display device. A designation process of designating a part of an image in the virtual reality space displayed by the display device so as to overlap the real space based on the first instruction by the user.
A specific process for specifying the position and orientation of the display device, and
A correction step of correcting a portion of an image in the virtual reality space designated by the designated step based on the position and orientation of the display device specified by the specific step and the content of a second instruction by the user. Have,
The second instruction is an instruction different from the redoing of the first instruction, and the user using the display device may give the instruction while looking at the image of the virtual reality space displayed by the display device. An information processing method characterized by being a possible instruction. A program characterized in that a computer functions as each means of the information processing apparatus according to any one of claims 1 to 9 .