JP7498330B2 - Virtual object operation method and head-mounted display - Google Patents

Description

The present invention relates to a virtual object manipulation method and a head-mounted display.

There is a technology that displays multiple windows on the display unit of an information processing device and switches between the windows in response to operational input (for example, Patent Document 1).

JP 2015-43135 A

The technology disclosed in the above-mentioned Patent Document 1 controls a window displayed on a display unit of an information processing device whose position is fixed.

Here, we consider a head-mounted display that defines absolute coordinates for displaying an application (a window displaying the application). In this case, when a user operates an application, it may be necessary for the user to move in accordance with the position associated with the application being used. As a result, operation may become cumbersome.

The object of the present invention is to provide a head-mounted display that allows users to easily operate applications.

The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

The technology described in the claims is used as a means to solve the above problems.

As an example, the device includes an operation reception unit that receives operation inputs for operating a virtual space or virtual objects, a communication unit that transmits and receives data to and from the outside, and a control unit, and when the operation reception unit receives an external display linking operation, the control unit links an external display device in the real space as a device for operating virtual objects, when the operation reception unit receives a virtual object selection operation, the control unit selects a specific virtual object from virtual objects arranged at any position in the virtual space, and when the operation reception unit receives a virtual object editing request operation, outputs information related to the selected specific virtual object to the linked external display device via the communication unit.

By using the technology of this invention, applications can be easily operated.

FIG. 1 is a diagram showing an overview of a head mounted display according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a visual recognition area. FIG. 2 is a diagram illustrating a hardware configuration of a head mounted display. FIG. 2 is a functional block diagram of a head mounted display. FIG. 13 is a diagram illustrating a method for setting a home position. 13A and 13B are diagrams illustrating another method for specifying the home position. 11A and 11B are diagrams illustrating a method for calculating the distance from the head mounted display to a home position. 10 is a flowchart showing a processing procedure for setting a home position. 1A and 1B are diagrams for explaining a state until a home position is set in a head mounted display worn by a user. 11A and 11B are diagrams for explaining display control of the home position. 13 is a flowchart showing a process for controlling the display of a home position. 11A and 11B are diagrams for explaining a display example of a home position when the home position is set; 11 is a diagram showing an example in which the head mounted display is facing in a direction different from the home position. FIG. FIG. 13 is a diagram showing an example in which the head mounted display moves an application to a home position. FIG. 13 is a diagram illustrating the state after "Call" is selected in the menu. 13 is a diagram illustrating a case where the user faces the direction in which the home position is located. FIG. FIG. 13 is a diagram illustrating a case where the user turns to the left again. 13 is a flowchart showing a processing procedure for application display control. FIG. 11 is a diagram illustrating an example of setting a home position. 13A and 13B are diagrams illustrating an example in which, when a monitor is placed at a home position, the monitor is reset as the home position. 13 is a flowchart showing a processing procedure for linking a home position to a monitor. 10 is a flowchart showing a procedure for controlling the display of an application when the monitor is in the home position.

Below, an example embodiment of the present invention will be described with reference to the drawings.

The overview of the head mounted display according to this embodiment will be described with reference to FIG. 1 (FIGS. 1(a) to 1(c)). FIG. 1 is a diagram showing an overview of a head mounted display 100. Specifically, FIG. 1(a) is a top view of the head mounted display 100 when a user 1 is wearing the head mounted display 100. FIG. 1(b) is a front view of the head mounted display 100. FIG. 1(c) is a side view of the head mounted display 100.

The head mounted display 100 is a device that displays and executes an application so that it is visible to the user 1 when the user 1 wears the display on his/her head. The head mounted display 100 has a first imaging unit 133a and a second imaging unit 133b that can capture an image of the outside world. The head mounted display 100 also has a transparent display unit 131 that is a transparent display means. As shown in FIG. 1(c), the head mounted display 100 also has an audio output unit 141 (e.g., audio output unit 141a). Note that a non-transparent display means may be used instead of the transparent display unit 131. In this case, the head mounted display 100 displays the image capture result on the display means.

The head mounted display 100 stores one or more applications, and displays the applications on the transparent display section 131 in response to an operational instruction from the user 1. The head mounted display 100 stores information indicating the position at which the application is to be displayed (for example, absolute position coordinates), and displays the application based on this information.

Next, the visual recognition area showing the area visible from the user 1 will be described with reference to FIG. 2. FIG. 2 is a diagram for explaining the visual recognition area. Here, the visual recognition area refers to the angle range that the user 1 can see through the transparent display unit 131. Which area of the outside world this visual recognition area corresponds to is determined by which direction the head mounted display 100 faces with respect to the outside world, and the user 1 can see any direction within the visual recognition area by changing the direction of the head. Furthermore, the head mounted display 100 arranges virtual objects (for example, the display screen/operation screen of an application displayed on the above-mentioned transparent display unit 131) in a virtual space that spreads and is superimposed on the outside world space around the user 1. This allows the user 1 to visually recognize the virtual object. Furthermore, the user 1 can operate the virtual object by inputting a predetermined operation to the head mounted display 100. The virtual object is visually recognized as if it is arranged at a specific position in the outside world. Note that when a display means that is not transparent is used, the range imaged by the first imaging unit 133 becomes the visual recognition area.

As described above, the head mounted display 100 executes an application stored in the head mounted display 100. In this embodiment, the head mounted display 100 sets an operation area (home position) in which the user 1 operates the application (e.g., operates an editing process). Here, the positions of the application and the home position are fixed relative to the outside world.

Next, the hardware configuration of the head mounted display 100 will be described with reference to FIG. 3. FIG. 3 is a diagram illustrating the hardware configuration of the head mounted display 100. The head mounted display 100 has a main control unit 101, which controls various devices via a bus 102. The main control unit 101 is a CPU (Central Processing Unit) or the like. The head mounted display 100 also has a RAM 103, a storage 110, a sensor unit 150, a communication unit 160, and an expansion I/F unit 170. The sensor unit 150 has a GPS receiver 151, a gyro sensor 152, a geomagnetic sensor 153, and an acceleration sensor 154.

The head mounted display 100 also has an operation input unit 120, a transmissive display unit 131, an image signal processing unit 132, a first imaging unit 133, a second imaging unit 134, an audio output unit 141, an audio signal processing unit 142, and an audio input unit 143. The operation input unit 120 has an operation key 121 and a touch sensor 122. The above-mentioned components operate to provide the functions of the head mounted display 100 described below.

The main control unit 101 is a processing unit that executes various functions by executing various information processing programs stored in the storage 110. The RAM 103 serves as a work area when the various programs are executed. The storage 110 is a section in which the main control unit 101 stores images captured by the first imaging unit 133 and the second imaging unit 134, as well as various information.

The sensor unit 150 is a part that acquires information for identifying the position and orientation of the head mounted display 100. The sensor unit 150 has a GPS receiver unit 151, a gyro sensor 152, a geomagnetic sensor 153, and an acceleration sensor 154.

The GPS receiver 151 receives GPS signals and acquires GPS information. The gyro sensor 152 has a function of detecting the direction of movement of the head mounted display 100, and acquires gyro information indicating angular velocity data associated with changes in direction of the head mounted display 100. The geomagnetic sensor 153 is a type of orientation detection device that detects an angle representing the absolute orientation of the head mounted display 100 based on geomagnetism. An example of such an angle is the azimuth angle. The acceleration sensor 154 has a function of detecting at least one of the acceleration and tilt angle of the head mounted display 100.

In this way, the sensor unit 150 can detect the position and orientation of the head mounted display 100.

The communication unit 160 is a part that performs wireless communication. For example, it performs short-range wireless communication. The expansion I/F unit 170 is an interface for connecting to peripheral devices.

The operation input unit 120 is a part that accepts operation input. The operation keys 121 are physical keys that are installed at predetermined positions on the head mounted display 100 (e.g., the part that is placed on the ear of the user 1). The touch sensor 122 is an input sensor that is installed on the transmissive display unit 131, etc.

The transmissive display unit 131 is a see-through (transmissive) display device. The image signal processing unit 132 is a part that analyzes the image captured by the first imaging unit 133 or the second imaging unit 134 and outputs the analysis results to the main control unit 101, etc.

The first imaging unit 133 is a part that images the outside world. The second imaging unit 134 is a part that images the eyes of the user 1. The audio output unit 141 is a speaker or the like that outputs audio. The audio signal processing unit 142 is a part that analyzes the audio input by the audio input unit 143. The audio input unit 143 is a part that inputs the audio of the user 1, and is a microphone or the like.

Next, the functions of the head mounted display 100 will be described with reference to FIG. 4. FIG. 4 is a functional block diagram of the head mounted display 100. The head mounted display 100 has a storage unit 71, a setting reception unit 72, a position identification unit 73, an imaging unit 74, a detection unit 75, a setting unit 76, and an application control unit 77 (application control unit).

The storage unit 71 is a part that stores various types of information. For example, the storage unit 71 stores a plurality of applications. The storage unit 71 may also store image data captured by the imaging unit 74.

The setting reception unit 72 is a part that receives an operation area setting request in response to an operation indicating a setting request by the user 1. When the setting reception unit 72 receives the operation area setting request, it notifies the position identification unit 73, the imaging unit 74, and the detection unit 75 that the operation area setting request has been received. When the setting reception unit 72 receives the operation area setting request, it transitions to a home position setting mode. The home position setting mode is a mode in which the home position is set.

In addition, instead of accepting an operation indicating a setting request by the user 1, the setting reception unit 72 may receive a setting request for an operation area setting from a control unit of the device itself (e.g., the main control unit 101) if an operation area has not been set when the head mounted display 100 is powered on.

The setting reception unit 72 is a part that receives a request to use the operation area in response to an operation indicating a use instruction by the user 1. When the setting reception unit 72 receives the request to use the operation area, it notifies the position identification unit 73, the imaging unit 74, and the detection unit 75 that the request to use the operation area has been received. When the setting reception unit 72 receives the request to use the operation area, it transitions to a home position use mode. The home position use mode is a mode in which the home position is used (displayed).

The position identification unit 73 is a part that identifies the position and direction (pointing direction) of the head mounted display 100. The position identification unit 73 is realized by, for example, the sensor unit 150. When the position identification unit 73 receives from the setting reception unit 72 a notice that the operation area setting request has been accepted or that the operation area use request has been accepted, the position identification unit 73 identifies the position and direction of the head mounted display 100. The position identification unit 73 identifies the position and direction of the head mounted display 100, and sends the identified position and direction to the setting unit 76. Furthermore, in the case of the home position use mode, the position identification unit 73 sends the identified position and direction to the application control unit 77.

The imaging unit 74 is a part that images the body parts of the user 1 wearing the head mounted display 100. The imaging unit 74 is realized by the first imaging unit 133, the second imaging unit 134, etc. When the imaging unit 74 receives a notification from the setting reception unit 72 that a request to set an operation area has been received or a request to use an operation area has been received, the imaging unit 74 sends the imaged result to the detection unit 75. The imaging unit 74, for example, images an area including the above-mentioned visual recognition area or images the area around the eyes of the user 1. The imaging unit 74 sends the imaged result to the detection unit 75.

The detection unit 75 is a part that detects the position indicated by the user 1 based on the image captured by the imaging unit 74. When the detection unit 75 receives a notification from the setting reception unit 72 that a request to set an operation area has been accepted or a request to use an operation area has been accepted, and receives the imaged result from the imaging unit 74, it detects the position indicated by the user 1.

When the detection unit 75 acquires an image of a location including the visual recognition area as a result of imaging by the imaging unit 74, the detection unit 75 performs a known image analysis process on the image to identify the movement of the finger included in the visual recognition area. In other words, the detection unit 75 detects a gesture of the user 1 and identifies a position based on the gesture.

For example, the detection unit 75 detects a gesture by the user 1 indicating an area from the image captured by the imaging unit 74, and identifies the area indicated by the detected result. The detection unit 75 detects the movement of the user 1's finger, and identifies the position of the area (operation area) based on the movement of the finger. The detection unit 75 sends information indicating the position of the area to the setting unit 76. The information indicating the position of the area is, for example, information based on the position specified by the finger.

When the detection unit 75 obtains an image of the area around the eyes of the user 1 as a result of imaging from the imaging unit 74, the detection unit 75 performs a known image analysis process on the image to detect the position of the gaze of the user 1. For example, the detection unit 75 detects the position of the gaze of the user 1 on the transparent display unit 131 as the position of the gaze of the user 1. In this case, the detection unit 75 sends information indicating the position to the setting unit 76 as information related to the operation area.

The setting unit 76 is a part that sets information (e.g., a position related to the operation area) that is an area for operating an application stored in the storage unit 71 (home position) based on the result detected by the detection unit 75 and the position identified by the position identification unit 73.

When the setting unit 76 acquires the position and direction identified by the position identification unit 73 and information indicating the area indicated by the results of detection by the detection unit 75, the setting unit 76 sets the information indicating the area and the position and direction identified by the position identification unit 73 in the memory unit 71.

The setting unit 76 sets in the storage unit 71 the position and direction identified by the position identification unit 73 and information about the area that is determined based on the position of the line of sight or the position of the area based on the finger movement as a result of detection by the detection unit 75.

In addition, when the setting unit 76 identifies a monitor corresponding to a position based on the results detected by the detection unit 75, the setting unit 76 may set the position of the monitor as the position of the operation area. For example, the setting unit 76 acquires an image corresponding to the position detected by the detection unit 75 from the imaging unit 74, compares the image with a pre-stored monitor template, and, when a monitor is detected, sets the identification information of the monitor as information related to the operation area.

The application control unit 77 is a part that displays an application stored in the memory unit 71 in the operation area of the transparent display unit 131 in response to an operation request from the user 1. When an editing operation request for a specific application is made in response to an operation request from the user 1, the application control unit 77 displays the application in the operation area and makes the application editable.

In addition, when information about the home position is already stored in the memory unit 71, the application control unit 77 appropriately acquires the position and direction of the head mounted display 100 from the position identification unit 73, and determines whether or not the home position is included in the visual recognition area based on the position and direction. When the application control unit 77 determines that the home position is included in the visual recognition area, it outputs the area of the home position to the transparent display unit 131.

In addition, when information about the operation area has already been set (information about the operation area has already been set), if the application control unit 77 identifies a monitor that corresponds to that position, it resets the monitor to the position of the new operation area.

The application control unit 77 also stores the display position of the application, and displays the application based on the display position. The display position is determined by the user's position, orientation, etc. The storage unit 71 may also store the display position of the application.

<Setting the home position>
Next, a method for setting the home position will be described with reference to Fig. 5. Fig. 5 is a diagram for explaining the method for setting the home position. It is assumed that the home position setting mode is set based on the operation received by the operation input unit 120.

When the home position setting mode is entered as described above, the imaging unit 74 captures an image of the visual recognition area shown in FIG. 5(a) for a predetermined period of time, and captures an image of the finger of the user 1 as shown in FIG. 5(a). For example, the detection unit 75 detects the trajectory of the finger from the upper center of the rectangular area to the lower center of the rectangular area.

That is, the detection unit 75 detects finger movements (gestures) as shown in FIG. 5(a) using the results of imaging by the imaging unit 74, identifies a region (rectangular region) designated by the user 1 based on the finger movements, and sets it as the home position. Furthermore, the detection unit 75 determines a reference position for the home position based on the difference between the visible region and the region of the transmissive display unit 131.

For example, the detection unit 75 determines the reference position of the home position in the transparent display unit 131 based on the positional relationship in the visual recognition area and the difference in size between the visual recognition area and the area of the transparent display unit 131. Here, the method of determining the reference position will be explained using FIG. 5(b). FIG. 5(b) shows the display area in the transparent display unit 131.

The detection unit 75 also calculates the ratios of the height V, the distance VU, the distance VP, and the distance VL. The height V is the height of the transparent display unit 131. The distance VU is the distance from the top of the transparent display unit 131 to the top of the home position. The distance VP is the distance from the top of the home position to the bottom of the home position. The distance VL is the distance from the bottom of the home position to the bottom of the transparent display unit 131.

The detection unit 75 also calculates the ratios of the width H of the transparent display unit 131, the distance HL from the left part of the transparent display unit 131 to the left part of the home position, the distance HP from the left part of the home position to the right part of the home position, and the distance HR from the right part of the home position to the right part of the transparent display unit 131.

The detection unit 75 calculates the distance from the head mounted display 100 to the position of the finger, and determines a reference position P based on this distance and the ratio calculated above. This reference position P indicates the position to be displayed in the transparent display unit 131, and the absolute position of the home position in the outside world can be determined by combining the current position and the pointing direction identified by the position identification unit 73. Basically, the absolute position in the outside world determined here is the home position, but as a modified example, a specific position within the visible area shown in Figure 5 (b) may be set as the home position. In this modified example, the home position can always be viewed.

The detection unit 75 may also determine the position of the specified area without taking the ratio into account as described above.

In FIG. 5(a), an example is given in which a finger is traced from the upper center of the home position to the lower center of the home position, but the home position may be specified by other methods. Here, other methods of specifying the home position are described with reference to FIG. 6. FIG. 6 is a diagram for describing other methods of specifying the home position. For example, as shown in FIG. 6(a), the home position may be specified along the home position area from the upper right to the lower left. Also, as shown in FIG. 6(b), the home position may be specified along the diagonal of the home position area from the upper right to the lower left.

The detection unit 75 also calculates the distance from the head mounted display 100 to the reference position of the home position based on the imaging result. A method for calculating the distance from the head mounted display 100 to the home position will be described with reference to Figs. 7(a) and 7(b). Fig. 7 is a diagram for explaining a method for calculating the distance from the head mounted display 100 to the home position. Fig. 7(a) is a top view of the head mounted display 100 in a state where the user 1 is wearing the head mounted display 100, and is a diagram showing the horizontal angle. Fig. 7(b) is a side view of the head mounted display 100, and is a diagram showing the elevation and depression angles.

Specifically, the detection unit 75 calculates the horizontal angle α of the reference position of the home position AR relative to the pointing direction of the head mounted display 100 shown in FIG. 7(a) based on the position of the finger in the captured image. The detection unit 75 also calculates the elevation/depression angle β of the home position reference position relative to the pointing direction of the head mounted display 100 shown in FIG. 7(b) based on the position of the finger in the captured image.

The detection unit 75 calculates the distance L to the reference position of the home position AR based on the distance to the center of the imaging range, the horizontal angle α, and the elevation/depression angle β.

The detection unit 75 sends the distance from the head mounted display 100 to the reference position of the home position AR, the width and height of the home position AR, and the reference position of the home position AR to the setting unit 76. In response to this, the setting unit 76 stores the distance L from the head mounted display 100 to the reference position of the home position AR, the width of the home position, the height of the home position, and the reference position P of the home position in the storage unit 71 as information related to the home position. It goes without saying that the absolute coordinates of the reference position P of the home position can be calculated from the position of the head mounted display 100, the pointing direction, the horizontal angle α, the elevation and depression angle β, and the distance L acquired by the position identification unit 73.

Now, the procedure for setting the home position AR will be described with reference to FIG. 8. FIG. 8 is a flowchart showing the processing procedure for setting the home position AR. First, when the setting reception unit 72 receives an operation area setting request in response to an operation indicating a setting request by the user 1, the setting reception unit 72 switches to the home position setting mode (step S1: Yes). Note that if the setting reception unit 72 does not receive an operation area setting request (step S1: No), the processing ends.

When the setting reception unit 72 receives the operation area setting request, it notifies the position identification unit 73, the imaging unit 74, and the detection unit 75 that the operation area setting request has been received. In response to this, the imaging unit 74 starts (enables) the imaging operation and sends the imaging result to the detection unit 75 (step S2).

The position identification unit 73 acquires the current position of the head mounted display 100 by the GPS receiving unit 151 or the like (step S3). The position identification unit 73 also acquires the orientation direction (the direction in which the head mounted display 100 is facing) by the geomagnetic sensor 153 or the like (step S4). The detection unit 75 also acquires the imaging result from the imaging unit 74, identifies the movement of the user 1 based on the imaging result, and calculates the distance to the finger of the user 1 based on the imaging result (step S5). The detection unit 75 also identifies the height and width of the home position based on the imaging result (step S6). The detection unit 75 also calculates the reference position P based on the distance to the finger of the user 1. The detection unit 75 sends the distance L from the head mounted display 100 to the reference position of the home position AR, the width of the home position, the height of the home position, and the reference position P of the home position to the setting unit 76 (step S7).

When the setting unit 76 acquires the height and width of the reference position P and the home position AR, it displays a confirmation message for the home position AR on the transparent display unit 131 to confirm the setting (step S8).

When user 1 performs an operation input indicating approval of the setting (step S8: Yes), the setting unit 76 sets the distance L, the height and width of the home position AR, and the reference position P in the storage unit 71. The setting unit 76 also sets the current position and pointing direction identified by the position identification unit 73 as information related to the home position AR in the storage unit 71 (step S9).

The setting unit 76 also notifies the imaging unit 74 of the completion of the setting, and the imaging unit 74 turns off (disables) the imaging state (step S10). The setting unit 76 also cancels the home position setting mode. Note that in step S8, if the user 1 performs an operation input indicating that the setting is not to be accepted (step S8: No), the process proceeds to step S3.

Next, a display example of the home position AR when the user 1 moves will be described with reference to Figs. 9 and 10. First, with reference to Fig. 9, a process up to setting the home position AR on the head mounted display 100 worn by the user 1 will be described. Fig. 9 is a diagram for explaining the state up to setting the home position AR on the head mounted display 100 worn by the user 1.

Figure 9(a) is a floor plan of user 1's room. As shown in Figure 9(a), user 1's room includes a bed 600, a desk 601, a chair 602, a bookshelf 603, a TV stand 604, and a TV 605.

Figure 9(b) is a diagram showing a state in which a user 1 is sitting in a chair 602. The user 1 wearing the head mounted display 100 operates the operation input unit 120 while seated in the chair 602, and the head mounted display 100 sets information about the home position AR in response to this.

Next, the display control of the home position will be described with reference to FIG. 10. FIG. 10 is a diagram for explaining the display control of the home position AR. FIG. 10(a) is a diagram showing an example in which the user 1 is looking at a location different from the home position (the head mounted display 100 is facing in a direction different from the home position). As shown in FIG. 10(a), if the reference position P is not in the direction (visual recognition area) of the head mounted display 100, the head mounted display 100 does not display the home position AR.

Also, FIG. 10(b) shows an example in which the head mounted display 100 faces the direction of the home position AR (i.e., the reference position P is in the visual recognition area). As shown in FIG. 10(b), when the head mounted display 100 faces the direction of the home position AR, the visual recognition area includes the reference position P, so the head mounted display 100 displays the home position AR. The display mode of the home position at this time may be kept in the orientation at the time of setting, or may be changed to face the user 1.

Next, the process procedure for controlling the display of the home position AR will be described with reference to FIG. 11. FIG. 11 is a flowchart showing the process for controlling the display of the home position AR. When the home position use mode is designated by the operation input of the user 1 (step S11: Yes), the application control unit 77 acquires information about the home position AR stored in the memory unit 71 (current position, pointing direction, distance L, width and height of the home position, and reference position P) (step S12).

Next, the application control unit 77 acquires current location information from the position identification unit 73 (step S13). The application control unit 77 also acquires the orientation direction from the position identification unit 73 (step S14).

The application control unit 77 compares the information about the home position AR stored in the memory unit 71 with the current position information and orientation direction identified by the position identification unit 73. If the comparison result indicates that the reference position P of the home position is within the visual recognition area (step S15: Yes) and the distance from the head mounted display 100 to the reference position P of the home position is within a predetermined value (step S16: Yes), the application control unit 77 displays the home position on the transparent display unit 131 (step S17). Note that if the condition of step S15 is not met (step S15: No) or the condition of step S16 is not met (step S16: No), the application control unit 77 proceeds to step S18.

<Move application to home position>
Next, a method for moving an application displayed on the head mounted display 100 to the home position will be described with reference to FIGS.

First, a display example of the home position when the home position is set will be described with reference to FIG. 12. FIG. 12 is a diagram for explaining a display example of the home position when the home position is set. FIG. 12(a) is a diagram showing a state in which the user 1 is sitting on the chair 602. In this state, the head mounted display 100 sets the home position in response to an operation request from the user 1.

Figure 12 (b) is an example of an image displayed on the transparent display section 131 of the head mounted display 100 in this state. As shown in Figure 12 (b), the head mounted display 100 displays app AP1 (app A) and app AP2 (app B), and also displays the home position on the transparent display section 131.

Next, FIG. 13 shows an example in which the head mounted display 100 faces in a direction different from the home position. FIG. 13(a) shows an example in which the head mounted display 100 faces in a direction different from the home position. FIG. 13(b) shows an example of an image displayed on the transmissive display section 131 of the head mounted display 100 in the state shown in FIG. 13(a).

As shown in FIG. 13(b), the head mounted display 100 does not display the home position, but displays app AP3 (app C), app AP4 (app D), app AP5 (app E), and app AP6 (app F) assigned to the direction in the transparent display section 131.

Next, FIG. 14 shows an example in which the head mounted display 100 moves an application to the home position. As shown in FIG. 14, when the application AP5 is selected by an operation input by the user 1 in the state shown in FIG. 13(b), a menu is displayed. It is assumed that "Call" is selected in this menu. This "Call" is a menu that indicates calling to the home position.

The state after "call" is selected in the menu will be described with reference to FIG. 15. As shown in FIG. 15(a), the direction in which the user 1 is facing has not changed, as in FIG. 13(a). FIG. 15(b) is a diagram showing the display state of the transparent display unit 131 after the menu is selected. As shown in FIG. 15(b), the app AP5 selected and designated with "call" has moved to the home position, and is therefore no longer a display target. In this case, the app control unit 77 stores the position of the app AP5 before and after the movement. Note that after the app AP5 is called, the head mounted display 100 may display the location of the app AP5 before the movement in a lighter color than before the movement. This allows the user 1 to visually confirm the position of the app AP5 before the movement, and can use this as a reference when adjusting the display position of the app in the area of the app AP5 before the movement.

Next, a case where the user 1 faces the direction in which the home position is located will be described with reference to FIG. 16. FIG. 16(a) shows a state in which the user 1 has changed his/her posture and the head mounted display 100 faces in a direction in which the home position is included in the visual recognition area.

Fig. 16(b) is a diagram showing an example of an image displayed on the transparent display section 131 of the head mounted display 100 in the above state. As shown in Fig. 16(b), the application control section 77 positions and displays the application AP5 at the position of the home position AR. Furthermore, when a menu display request for the application AP5 is received by an input operation of the user 1, the application control section 77 displays the menu, and when "Back" is selected, removes the application AP5 from the display targets.

Next, a case where the user 1 faces left again will be described with reference to FIG. 17. FIG. 17(a) shows a state in which the user 1 has changed his/her posture and the head mounted display 100 faces left. FIG. 17(b) shows an example of an image displayed on the transmissive display section 131 of the head mounted display 100 in the above state.

As shown in FIG. 17(b), when the "Back" menu for app AP5 is selected, the app control unit 77 returns app AP5 to its original position and displays it.

Next, the processing procedure for application display control will be described with reference to FIG. 18. FIG. 18 is a flowchart showing the processing procedure for application display control. First, in the case of the home position use mode (step S21: Yes), the position identification unit 73 acquires the current position of the head mounted display 100 (step S22). The position identification unit 73 also acquires the pointing direction (step S23). The application control unit 77 acquires the current position and pointing direction from the position identification unit 73, and displays an application according to the current position and pointing direction based on the stored display positions of the applications (step S24).

When the application control unit 77 detects that any application has been selected (step S25: Yes), it displays a menu (step S26). The application control unit 77 identifies the selected command (step S27). If the identified command is "return", the application control unit 77 moves the selected application to its original position (step S29) and proceeds to step S30. If the identified command is "call", the application control unit 77 moves the selected application to the home position (step S28) and proceeds to step S30. If a menu other than "call" or "return" is selected, the application control unit 77 proceeds to step S30 without doing anything. If an instruction to end the home position use mode has not been given in step S30 (step S30: No), it proceeds to step S22. If an instruction to end the home position has been given in step S30 (step S30: Yes), it ends the process.

<Monitor link processing>
Next, an example of linking a monitor to set the home position AR when the monitor is located at the home position AR will be described. First, an example of setting the home position AR will be described. FIG. 19(a) is a diagram showing a state in which a user 1 sits on a chair 602. The user 1 wearing the head mounted display 100 operates the operation input unit 120 while seated on the chair 602, and sets the home position AR accordingly. FIG. 19(b) is a diagram showing an example of an image displayed on the transparent display unit 131 of the head mounted display 100 in the above state. As shown in FIG. 19(b), the application control unit 77 displays the home position AR on the transparent display unit 131.

Next, an example of resetting the monitor 700 as the home position AR when the monitor 700 is placed at the home position AR will be described with reference to FIG. 20.

Figure 20 (a) shows an example in which a monitor is placed in front of user 1 while user 1 is sitting in chair 602.

In this state, the application control unit 77 determines whether or not a monitor is located within the range of the home position. For example, the application control unit 77 compares the image capture result by the image capture unit 74 with a pre-stored monitor template to determine whether or not a monitor is present.

When the application control unit 77 determines that a monitor exists, it displays a message on the transparent display unit 131 inquiring whether or not to connect to the monitor. When the user 1 inputs an operation to connect to the monitor, the application control unit 77 establishes a state in which it is possible to send and receive information with the monitor by short-range wireless communication with the monitor, acquires identification information of the monitor, and adds the identification information of the monitor as information related to the home position AR. In this way, the application control unit 77 functions as a setting correction unit.

Then, the application control unit 77 displays the monitor as the home position AR, as shown in FIG. 20(b). As a modified example, if the monitor 700 is visible in the visible area, the screen of the monitor 700 may be set as the new home position. In this case, it may be possible to return to the original home position setting that is not linked to the monitor, if necessary.

Next, the process for linking the home position AR to the monitor will be described with reference to FIG. 21. FIG. 21 is a flowchart showing the process for linking the home position AR to the monitor 700. Steps S31 to S35 shown in FIG. 21 are the same as steps S11 to S15 shown in FIG. 11, respectively, and therefore will not be described.

In step S36, the application control unit 77 compares the imaging result by the imaging unit 74 with a pre-stored template of the monitor 700 to determine whether the monitor 700 is present within the range of the home position (step S36). If the monitor 700 is present within the range of the home position AR (step S36: Yes), the application control unit 77 outputs a message inquiring whether to link with the monitor 700 (step S37). If a command indicating unlinking is given by a user operation, the link with the monitor 700 is unlinked (step S40) and the process proceeds to step S41. Note that if the monitor 700 is not present within the range of the home position AR (step S36: No), the process proceeds to step S41.

In step S37, if a user operation is performed to link with a monitor (step S37: Yes), and if there are multiple linkable monitors 700, a list of the monitors 700 is displayed and the selection of the monitor 700 is accepted (step S38). Note that if there is only one monitor 700, the processing of step S38 may be omitted. The application control unit 77 links with the selected monitor 700 (resets the home position to that monitor) (step S39), and proceeds to step S41.

If an instruction to end the home position use mode is given in step S41 (step S41: Yes), the process ends. If an instruction to end the home position use mode is not given in step S41 (step S41: No), the process proceeds to step S33.

Next, the procedure for application display control when the monitor 700 is in the home position will be described with reference to FIG. 22. FIG. 22 is a flowchart showing the procedure for application display control when the monitor 700 is in the home position AR. Note that steps S51 to S57 and S62 shown in FIG. 22 are the same as steps S21 to S27 and S30 shown in FIG. 18, respectively, and therefore will not be described.

If the identified command in step S57 is "return," there is nothing to display on the monitor 700, so the application control unit 77 ends the monitor output of the selected application (step S60), moves the application to its original position (step S61), and proceeds to step S62.

If the command identified in step S57 is "call," the application control unit 77 ends the display of the selected application (step S58), displays the application on the monitor 700 in the home position (step S59), and proceeds to step S62. If a menu other than "call" or "return" is selected, the application control unit 77 proceeds to step S62 without doing anything.

As described above, in the head mounted display 100, the storage unit 71 stores applications. The imaging unit 74 captures images of the user's body parts, and the position identification unit 73 identifies the position and orientation of the head mounted display 100. The detection unit 75 detects the state of the user 1 based on the image captured by the imaging unit 74, and the setting unit 76 sets a position indicating the home position based on the result of detection by the detection unit 75 and the position and orientation identified by the position identification unit 73.

The head mounted display 100 sets the area in which the application is operated based on the user's input designation, allowing the user to operate the application in a position that is easy for the user to operate.

Specifically, the imaging unit 74 captures an image of a portion including the user 1's finger, the detection unit 75 detects a gesture of the user 1 from the imaging result, and the setting unit 76 sets a position related to the home position using the position determined by the gesture. In this case, the head mounted display 100 sets a position related to the home position based on the result of the finger movement (gesture) of the user 1, so that the home position can be set by an intuitive and simple operation rather than a complicated operation input.

The imaging unit 74 also captures an image of the area around the eyes of the user 1, the detection unit 75 detects the user's line of sight using the imaging result by the imaging unit 74 and detects a position based on the line of sight, and the setting unit 76 sets a position related to the home position using the position based on the line of sight. In this case as well, the head mounted display 100 can set the home position through intuitive and simple operations rather than through complicated operational inputs.

When the monitor 700 is located at a position based on the position detected by the detection unit 75 and the position identified by the position identification unit 73, the setting unit 76 sets the position of the monitor 700 to a position related to the home position AR. In this case, the head mounted display 100 can set the monitor 700 as the home position AR by an intuitive operation such as a gesture by the user 1.

In addition, when the setting unit 76 sets a position related to the home position AR, if the application control unit 77 identifies a new position for the monitor 700 corresponding to that position, the application control unit 77 resets the position of the monitor as a new home position. This allows the head mounted display 100 to set the position of the monitor as the home position after setting the home position without requiring the user 1 to perform an operation indicating a new position designation.

In addition, the application control unit 77 controls the display of an application stored in the memory unit 71 in the home position in response to a specific input operation, thereby enabling the application to be displayed in a position that is easy for the user to operate.

Although not specifically mentioned in the above embodiment, multiple home positions may be set. In this case, the target home position may be determined based on the current position of the head mounted display 100, or a home position specified by a user operation may be given priority.

Although the position identification unit 73 has been described as identifying a position based on a current position such as latitude and longitude, the position may be identified by various other methods. For example, a relative position based on a reference object (a landmark object) may be identified.

Although not specifically mentioned in the above embodiment, command specification (e.g., "call") may be specified by voice input.

In the above embodiment, a case was described in which distance L, width and height of the home position, and reference position P are set as information related to the home position in addition to the current position and direction of orientation. However, it is also possible to hold predetermined values for distance L, width and height of the home position, and set the current position, direction of orientation, and reference position P.

The functions of the present invention described above may be realized in part or in whole by hardware, for example by designing them as integrated circuits. They may also be realized by software, with a microprocessor unit or the like interpreting and executing an operating program that realizes each function. Hardware and software may also be used in combination.

The control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all control lines and information lines on the product. In reality, it is safe to assume that almost all components are interconnected.

The control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all control lines and information lines on the product. In reality, it is safe to assume that almost all components are interconnected.

The above embodiment includes the following aspects.
(Appendix 1)
A head mounted display that stores an application,
An imaging unit that images a part of a user wearing the head mounted display;
A detection unit that detects a position indicated by the user based on an image captured by the imaging unit;
a position identification unit that identifies a position and a direction of the head mounted display;
a setting unit that sets information regarding an operation area, which is an area in which the application is operated, based on the result of detection by the detection unit and the position and direction specified by the position specifying unit;
Equipped with
Head-mounted display.

(Appendix 2)
A setting method executed on a head mounted display storing an application, comprising:
An imaging step of imaging a part of a user;
a detection step of detecting a position indicated by a user wearing the head mounted display based on the image captured in the imaging step;
A position determination step of determining a position and an orientation of the head mounted display;
A setting step of setting information regarding an operation area, which is an area for operating the application, based on the result detected in the detection step and the position and direction identified in the position identification step.
Setting method.

1...user, 71...storage unit, 72...settings reception unit, 73...position identification unit, 74...imaging unit, 75...detection unit, 76...setting unit, 77...application control unit, 100...head mounted display, 101...main control unit, 103...RAM, 110...storage, 120...operation input unit, 121...operation keys, 122...touch sensor, 131...transparent display unit, 132...image signal processing unit, 133...first imaging unit, 134...second imaging unit, 141...audio output unit, 142...audio signal processing unit, 143...audio input unit, 150...sensor unit, 151...GPS receiving unit, 152...gyro sensor, 153...geomagnetic sensor, 154...acceleration sensor, 160...communication unit, 170...extension I/F unit.

Claims (6)

A virtual object operation method executed on a head mounted display in which a virtual object placed at an arbitrary position in a virtual space can be viewed, comprising:
an operation receiving step of receiving an operation input for operating the virtual space or the virtual object;
an external display linking step of setting an external display device in a real space as an operation area for operating the virtual object when an external display linking operation is accepted in the operation accepting step;
a virtual object selection step of selecting a predetermined virtual object from virtual objects arranged at any position in the virtual space when a virtual object selection operation is accepted in the operation acceptance step;
a virtual object output step of outputting, when a virtual object editing request operation is accepted in the operation accepting step, information for displaying the predetermined virtual object selected in the virtual object selecting step on the external display device to the external display device linked in the external display linking step;
including,
How to manipulate virtual objects. The virtual object operation method according to claim 1 ,
an external display confirmation step of confirming, when an operation area setting operation is accepted in the operation acceptance step, whether or not the external display device is located at a position in a real space corresponding to a predetermined position in a virtual space designated by the operation area setting operation;
In the external display linking step, when it is confirmed that the external display device is present in the external display confirmation step, the external display device is set as an operation area for operating the virtual object.
How to manipulate virtual objects. The virtual object operation method according to claim 1 ,
a virtual object output terminating step of terminating output of information for displaying the predetermined virtual object selected in the virtual object selection step on the external display device linked in the external display linking step, when a virtual object editing terminating operation is accepted in the operation accepting step;
Further comprising:
How to manipulate virtual objects. A head mounted display capable of visually recognizing a virtual object placed at an arbitrary position in a virtual space,
an operation receiving unit that receives an operation input for operating the virtual space or the virtual object;
A communication unit that transmits and receives data to and from the outside;
A control unit;
Equipped with
The control unit is
When the operation reception unit receives an external display linked operation, the external display device in a real space is set as an operation area for operating the virtual object;
When the operation reception unit receives a virtual object selection operation, the operation reception unit selects a predetermined virtual object from virtual objects arranged at any position in the virtual space;
When the operation reception unit receives a virtual object editing request operation, information for displaying the selected predetermined virtual object on the external display device is output to the linked external display device via the communication unit.
Head-mounted display. The head mounted display according to claim 4,
The control unit further
when the operation reception unit receives an operation area setting operation, it is confirmed whether or not an external display device is present at a position in a real space corresponding to a predetermined position in a virtual space designated by the operation area setting operation;
when it is confirmed that the external display device is present, setting the external display device as an operation area for operating the virtual object;
Head-mounted display. The head mounted display according to claim 4,
The control unit further terminates, when the operation reception unit receives a virtual object editing end operation, outputting, via the communication unit , information for displaying the selected predetermined virtual object on the external display device to the linked external display device .
Head-mounted display.