JP2021068296A - Information processing device, head-mounted display, and user operation processing method - Google Patents

Abstract

To provide a user interface that is easy to operate even when a head-mounted display is attached.SOLUTION: In an information processing device 70 of a head-mounted display 100, an illuminance information acquisition unit 72 acquires illuminance information at a plurality of locations on the outer surface of the head-mounted display 100. A user operation determination unit 74 determines that a user operation has been performed when a predetermined change occurs in the luminance information. An operation content acquisition unit 76 acquires operation content associated with the change in the luminance information as a user operation, and an operation processing unit 78 executes processing corresponding to the acquired operation content.SELECTED DRAWING: Figure 5

Description

The present invention relates to an information processing device that performs processing corresponding to the operation of a user wearing a head-mounted display, a head-mounted display, and a user operation processing method.

A system that can express a realistic image world by detecting the movement of the head of a user wearing a head-mounted display and expressing the space to be displayed in the corresponding field of view has become widespread. In addition, a walk-through system has also been developed that allows a user wearing a head-mounted display to virtually walk around in the space displayed as an image by physically moving.

A head-mounted display that shields the light of the outside world enhances the immersive feeling in the display world, but it is difficult for the user to hold or operate surrounding objects because the real world cannot be seen while wearing it. For example, even if a user who enjoys content wants to adjust the volume of audio and the brightness of the display panel, adjustment with general switches and buttons is not easy because it is a fumbling operation. In order to facilitate the operation and increase the number of adjustable objects, it is conceivable to increase the size or number of switches, etc., but especially when such an operation means is provided on the head-mounted display itself. It is not preferable because it increases design restrictions.

The present invention has been made in view of these problems, and an object of the present invention is to provide a user interface that is easy to operate even when a head-mounted display is attached. Another object of the present invention is to provide a technique that enables various operations without increasing the design restrictions of the head-mounted display.

One aspect of the present invention relates to an information processing device. In this information processing device, a user operation is performed when a predetermined change occurs in at least one of the luminance information acquisition unit that acquires the luminance information of light at a plurality of locations on the outer surface of the head mount display and the luminance information at the plurality of locations. Corresponds to the user operation determination unit that determines that, the operation content acquisition unit that acquires the operation content associated with the change in the luminance information as the user operation when it is determined that the user operation has been performed, and the operation content. It is characterized by including an operation processing unit for performing processing.

Another aspect of the invention relates to a head-mounted display. This head-mounted display includes the above-mentioned information processing device, a plurality of cameras arranged at a plurality of locations and capturing images as luminance information, and a display unit and an audio output unit adjusted by an operation processing unit. It is characterized by.

Yet another aspect of the present invention relates to a user operation processing method. In this user operation processing method, when a predetermined change occurs in at least one of a step of acquiring light luminance information at a plurality of locations on the outer surface of the head-mounted display at a predetermined rate and at least one of the luminance information at the plurality of locations, the user operation is performed. A step of determining that the operation has been performed, a step of acquiring the operation content associated with the change in the luminance information as a user operation when it is determined that the user operation has been performed, and a process corresponding to the operation content are executed. It is characterized by including steps.

It should be noted that any combination of the above components and the conversion of the expression of the present invention between a method, a device, a system, a computer program, a recording medium on which a computer program is recorded, and the like are also effective as aspects of the present invention. ..

According to the present invention, the user operation with the head-mounted display attached is facilitated without increasing design restrictions.

It is a figure which shows the appearance example of the head-mounted display of this embodiment. It is a figure which shows the configuration example of the content processing system to which this embodiment can be applied. It is a figure for demonstrating the example of the operation method in this Embodiment. It is a figure which shows the internal circuit structure of the head mounted display in this embodiment. It is a figure which shows the structure of the functional block of the head-mounted display in this embodiment. In this embodiment, it is a flowchart which shows the procedure which an information processing apparatus processes a content while corresponding to a user operation. It is a figure which illustrates the information which corresponded with the change of the luminance information and the operation content, which is held internally by the operation content acquisition part in this embodiment. In this embodiment, it is a figure for demonstrating the mode of determining a user operation based on the change of the brightness information of light from a plurality of places on the outer surface of a head-mounted display.

FIG. 1 shows an example of the appearance of the head-mounted display 100. In this example, the head-mounted display 100 is composed of an output mechanism unit 102 and a mounting mechanism unit 104. The mounting mechanism unit 104 includes a mounting band 106 that goes around the head and realizes fixing of the device when the user wears it. The output mechanism 102 includes a housing 108 having a shape that covers the left and right eyes when the head-mounted display 100 is worn by the user, and includes a display panel inside so as to face the eyes when the head-mounted display 100 is worn.

Further, the inside of the housing 108 is provided with an eyepiece located between the display panel and the user's eyes when the head-mounted display 100 is attached to magnify the image. Further, the head-mounted display 100 may further include a speaker or an earphone at a position corresponding to the user's ear when worn. The head-mounted display 100 also incorporates motion sensors such as an acceleration sensor, a gyro sensor, and a geomagnetic sensor, and detects the translational motion and rotational motion of the head of the user wearing the head-mounted display 100, as well as the position and posture at each time. You may.

The head-mounted display 100 further includes cameras 110a, 110b, 110c, and 110d on the front surface of the housing 108, and captures a moving image of the surrounding real space with a field of view corresponding to the orientation of the user's face. The number and arrangement of the cameras 110a, 110b, 110c, and 110d are not particularly limited, but in the illustrated example, they are provided at the four corners of the front surface of the housing 108 of the output mechanism 102. Hereinafter, the cameras 110a, 110b, 110c, and 110d may be collectively referred to as the camera 110. In a certain aspect, the head-mounted display 100 provides a see-through mode in which a moving image taken by the camera 110 is immediately displayed to show the state of the real space in the direction in which the user is facing.

The head-mounted display 100 further analyzes the image captured by the camera 110 to determine the position and posture of the head-mounted display 100 itself, the position and posture of the user's head, and the existence, shape, and position of surrounding objects. Acquire at least one of the information at a predetermined rate. Hereinafter, this information is collectively referred to as "real space information". SLAM (Simultaneous Localization and Mapping) is a technique for obtaining real space information by analyzing sensing results such as captured images.

SLAM using captured images is a process of estimating the three-dimensional position of the real object from the positional relationship of the images of the same real object appearing in the captured images from multiple viewpoints, and the image of the real object whose position has been estimated. This is a method of acquiring the position and orientation of the camera while creating an environmental map by repeating the process of estimating the position and orientation of the camera based on the position on the captured image. The real space information obtained from the captured image in this way can be used to determine the field of view of the image to be displayed on the head-mounted display 100, or can be used to draw a virtual object that interacts with a surrounding real object. .. In addition, it is understood by those skilled in the art that various types of information processing using real space information can be considered.

FIG. 2 shows a configuration example of a content processing system to which the present embodiment can be applied. The head-mounted display 100 is connected to the content processing device 200 by wireless communication or an interface for connecting peripheral devices such as USB Type-C. The content processing device 200 may be further connected to the server via a network. In that case, the server may provide the content processing device 200 with an online application such as a game in which a plurality of users can participate via a network.

The content processing device 200 basically processes a content program, generates display image and audio data, and transmits the data to the head-mounted display 100. The head-mounted display 100 receives the display image or sound data and outputs the content as an image or sound. At this time, the content processing device 200 acquires information on the position and posture of the head-mounted display 100 from the head-mounted display 100 at a predetermined rate, specifies the position of the user's viewpoint and the direction of the line of sight based on the information, and then responds. A display image may be generated and transmitted in the field of view.

Alternatively, the content processing device 200 may acquire information such as an object around the user, its position, and its shape from the head-mounted display 100, and generate and transmit an image in which a virtual object is drawn at the corresponding position. For example, augmented reality or mixed reality may be realized by combining and displaying the transmitted virtual object with the captured image of the camera 110 on the head-mounted display 100. However, the purpose of using the real space information acquired by the head-mounted display 100 is not particularly limited.

For example, the content processing device 200 may acquire the movement of the head-mounted display 100 as a gesture of the user, determine the content of the information processing to be performed, or advance the electronic game accordingly. In any case, by acquiring the real space information inside the head-mounted display 100, it is possible to significantly reduce the size of the data to be transmitted to the content processing device 200 rather than transmitting the captured image. However, the present embodiment is not limited to this, and for example, a part or all of the functions of the content processing device 200 described above may be mounted on the head-mounted display 100. Alternatively, the content processing device 200 may be provided with a function of acquiring real space information.

In the present embodiment, under such a premise, the user operation for the function provided by the head-mounted display 100 or the function provided by the content output to the head-mounted display 100 is realized by using the camera 110. FIG. 3 is a diagram for explaining an example of the operation method in the present embodiment. In (a), the head-mounted display 100 is provided with four cameras 110a, 110b, 110c, and 110d as shown in FIG.

Therefore, as shown in the upper part of (b), four image frames 122 are acquired at a predetermined shooting rate. In the illustrated example, the four image frames 122 show a table in front of the user with parallax according to the position of the camera. Here, as shown in (a), when the user holds the hand 120 in front of a certain camera 110d, as shown in the lower part of (b), only the image frame 124 captured by the camera 110d changes. That is, since the light incident on the camera 110d is blocked by the hand 120, the image is not captured or the brightness is lowered as a whole.

Here, the "change" may be a change in a time-series frame of a moving image taken by one camera 110d, or may be a difference from an image frame taken by another cameras 110a, 110b, 110c at the same timing. Alternatively, a combination thereof may be used. When those parameters meet a predetermined condition for which the change is considered to have occurred, the head-mounted display 100 detects that the hand 120 is held over the camera 110d that is capturing the image frame in which the change has occurred. Then, the head-mounted display 100 identifies the operation content associated with the position of the camera 110d in advance, and performs the corresponding processing.

For example, if the camera 110d is associated with volume adjustment, the head-mounted display 100 adjusts the volume of audio. At this time, the head-mounted display 100 may change the adjustment amount depending on the number of repetitions in which the hand 120 is held and the duration of holding the hand 120. Further, in the figure, the hand is held over only one camera 110d, but when it is detected that the hand is held over a plurality of cameras 110 at the same time, the head-mounted display 100 determines the operation content according to the combination of the positions of the cameras 110. You may discriminate.

Further, the head-mounted display 100 may determine the operation content by the movement of the hand 120 in front of the camera 110. Further, the object to be held is not limited to the hand 120, and may be a predetermined object or an object of a predetermined color, and variations in the operation contents may be increased depending on what is held over, what color is held over, and the like. ..

The user can roughly grasp the position and the field of view of each camera 110 even when the head-mounted display 100 is attached. Further, if the detection conditions are appropriately set, the head-mounted display 100 can detect that a hand or the like is held up even if the camera 110 is not strictly covered. Therefore, the user operation can be made much easier than finding and distinguishing a small button or switch. Further, when the camera 110 is provided on the head-mounted display 100 for the purpose of acquiring the position and orientation and the see-through mode, the camera 110 can be diverted, so that there are few design restrictions and the manufacturing cost is low as compared with providing a separate operating means. It can be suppressed.

FIG. 4 shows the internal circuit configuration of the head-mounted display 100. The head-mounted display 100 includes a CPU 50, a main memory 52, a display unit 54, and an audio output unit 56. Each of these parts is connected to each other via a bus 58. An input / output interface 60 is further connected to the bus 58. A communication unit 62, a motion sensor 64, and an imaging unit 66, which are wired or wireless LAN network interfaces, are connected to the input / output interface 60.

The CPU 50 processes the information acquired from each unit of the HMD 18 via the bus 58, and supplies the output data to the display unit 54 and the audio output unit 56. The main memory 52 stores programs and data required for processing in the CPU 50. However, depending on the design of the application or the device to be executed, the content processing device 200 may perform almost all the processing, and the head-mounted display 100 may be sufficient to output the data transmitted from the content processing device 200. In this case, the CPU 50 and the main memory 52 can be replaced with a simpler device.

The display unit 54 is composed of a display panel such as a liquid crystal panel or an organic EL panel and a control mechanism thereof, and displays an image in front of the eyes of a user wearing the head-mounted display 100. Stereoscopic vision may be realized by displaying a pair of parallax images in the regions corresponding to the left and right eyes. The voice output unit 56 is composed of a speaker or earphone provided at a position corresponding to the user's ear when the head-mounted display 100 is attached, and a control mechanism thereof, and allows the user to hear the voice.

The communication unit 62 establishes communication with the content processing device 200 and transmits / receives necessary data. The motion sensor 64 is composed of at least one of an acceleration sensor, a gyro sensor, a geomagnetic sensor, and the like, and detects the posture and movement of the head-mounted display 100 in a three-dimensional space. The imaging unit 66 is composed of a plurality of cameras 110, captures a real space at a predetermined rate, performs appropriate correction processing, and outputs a captured image.

FIG. 5 shows the configuration of the functional block of the head-mounted display 100. Each functional block shown in FIG. 5 can be realized by the configuration of CPU, GPU, various memories, display panel, speaker, camera, motion sensor, etc. in terms of hardware, and is loaded into memory from a recording medium or the like in terms of software. It is realized by a program that demonstrates various functions such as data input function, data retention function, image processing function, and communication function. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any of them.

The head-mounted display 100 includes an image pickup unit 66, a display unit 54, and an audio output unit 56 shown in FIG. 4, as well as an information processing device 70 realized by a CPU 50, a main memory 52, and the like. The information processing device 70 includes a brightness information acquisition unit 72 that acquires data of brightness information captured by the imaging unit 66, a user operation determination unit 74 that determines that a user operation has been performed based on the brightness information, and a user operation. At that time, the operation content acquisition unit 76 for acquiring the operation content, the operation processing unit 78 for performing the processing corresponding to the user operation, the real space information acquisition unit 80 for acquiring the real space information based on the brightness information, the real space information, etc. The content processing unit 82 that processes the content based on the content is included.

The brightness information acquisition unit 72 acquires light brightness information at a plurality of locations on the outer surface of the head-mounted display 100 at a predetermined rate. Here, the "luminance information" is typically a two-dimensional image taken by the camera 110, but the number of pixels is not limited as long as it is information representing the brightness of light at a plurality of locations. Further, when the camera 110 is used as described above, the luminance information acquisition unit 72 acquires the luminance information (hereinafter mainly referred to as a captured image) of the light incident on a plurality of locations on the outer surface of the head-mounted display 100. ..

On the other hand, as a modification, the same user operation can be performed even if the luminance information acquisition unit 72 detects the luminance of the light reflected or emitted from a plurality of locations on the outer surface of the head-mounted display 100. In this case, an external camera for detecting the emitted light is separately provided. This aspect will be described later. The user operation determination unit 74 determines that the user operation has been performed when a predetermined change occurs in at least one of the luminance information at a plurality of locations on the outer surface of the head-mounted display 100.

Here, the user operation determination unit 74 detects, for example, the following event as a predetermined change.
1. 1. 2. Satisfying the predetermined criteria that the light incident on the camera 110 is considered to be blocked. 2. Satisfying a predetermined criterion that a predetermined object is considered to be held in front of the camera 110. 2. Satisfying a predetermined criterion that an object of a predetermined color is considered to be held in front of the camera 110. An object held in front of the camera 110 meets a predetermined criterion that is considered to have made a predetermined movement.

As the above 1, the user operation determination unit 74 detects, for example, that the luminance or the average value of the luminance has decreased by a predetermined value or more. As described in 2, the user operation determination unit 74 detects, for example, that a user's hand, a predetermined controller, a mobile terminal, a high-performance mobile phone, or the like is captured by a general image recognition process. As described in 3, the user operation determination unit 74 detects by a general image recognition process that an object of a predetermined color, for example, a card of a predetermined color, a mobile terminal, a high-performance mobile phone, or the like is captured. As described in 4, the user operation determination unit 74 detects, for example, the horizontal direction, the vertical direction, the oblique direction, and the movement of drawing a specific figure of the imaging surface based on the optical flow obtained by general image analysis.

Here, the object for detecting the movement may be any of the object and the color detected in 2 or 3 above. When such an object is captured in images taken by two or more cameras 110, the user operation determination unit 74 obtains the movement in the three-dimensional space by stereo matching, and collates it with the above-mentioned predetermined standard. You may. For example, the user operation determination unit 74 detects a movement of moving a hand closer to or further away from the image pickup surface. For these detection processes, the user operation determination unit 74 holds the registration information of the predetermined criteria such as the threshold value of the amount of change in brightness, the object or color to be detected, and the movement to be detected in the internal memory.

As described above, the user operation determination unit 74 may detect a change based on a time-series comparison of image frames captured by each camera 110, or may compare a plurality of images captured by a plurality of cameras 110 at the same time. Changes may be detected based on this. Alternatively, the user operation determination unit 74 may detect a change based on both. Further, the user operation determination unit 74 may detect an event in which the above 1 to 4 are appropriately combined, such as a predetermined object being held over the camera and the predetermined object moving.

Further, the user operation determination unit 74 may detect the above 2 to 4 based on the images captured by the plurality of cameras 110. For example, when the user operation determination unit 74 recognizes that a predetermined mobile terminal is captured in a plurality of captured images, there is no contradiction between the positional relationship of the captured camera 110 and the positional relationship of the image of the mobile terminal on the captured image. After confirming that, it is determined that one mobile terminal is held over. As a result, it is possible to reduce the possibility of erroneous recognition such as detecting a surrounding object that originally existed as an object held over, as compared with a determination using only one captured image.

When it is determined by the user operation determination unit 74 that the user operation has been performed, the operation content acquisition unit 76 acquires the operation content associated with the detected change as the user operation. That is, the operation content acquisition unit 76 specifies the operation content intended by the user based on the position or a combination thereof that satisfies the criteria such as 1 to 4 above among the plurality of locations on the outer surface of the head-mounted display 100. Therefore, in the operation content acquisition unit 76, the position where the change has occurred, the combination thereof, and the operation content are registered in advance in association with each other. The registration information held by the operation content acquisition unit 76 may be further associated with the operation content by combining with at least one of the content of the change such as an object satisfying the above-mentioned predetermined criteria, its color, and its movement.

The operation processing unit 78 executes a process corresponding to the operation content acquired by the operation content acquisition unit 76. The content of the operation to be performed is not particularly limited, and may be output adjustment or output control of the head-mounted display 100 itself, or may be an operation for the content. However, the operation processing unit 78 can achieve the effect of the present embodiment by performing adjustments and controls (independent of the content) that are different from the processing of the content itself, which has been conventionally realized via switches and buttons. It will be higher.

For example, the operation processing unit 78 turns off the power of the head-mounted display 100, increases / decreases the brightness of the display panel, increases / decreases the volume of audio, starts playing, pauses, fast-forwards, rewinds, and contents the moving image or music which is the output target content. Switching, image zooming up and zooming down, etc. The operation processing unit 78 adjusts the output state of the display unit 54, the voice output unit 56, and the like according to the processing to be performed. The operation processing unit 78 may adjust the length of the wearing band 106 of the head-mounted display 100, adjust the distance between the left and right eyepieces, adjust the distance between the left-eye / right-eye display images, and the like. In any case, the method for realizing these adjustments and controls may be the same as the general method using switches and buttons.

The real space information acquisition unit 80 acquires information related to the real space necessary for generating output data of contents such as a display image by using at least one of the images captured by the camera 110. As described above, the real space information is information that affects the data generation of the content image and sound itself, such as the position and posture of the head-mounted display and the positional relationship with objects around the user, and is the user operation judgment unit. It is obtained from a captured image in a state in which a "predetermined change" has not occurred, which is detected by 74.

Therefore, the user operation determination unit 74 determines whether or not a predetermined change has occurred in the captured image, and sequentially supplies the data of the captured image that has not occurred to the real space information acquisition unit 80. By providing a plurality of cameras 110 and obtaining a plurality of captured images at the same time, it is possible to continue to acquire real space information using the other captured images even if a change occurs due to a user operation in a part of them. For example, the real space information acquisition unit 80 acquires real space information by analyzing a photographed image other than the photographed image in which a user's hand is held over and the surrounding objects are not captured. The real space information acquisition unit 80 may further improve the accuracy of the real space information by using the output value of the motion sensor 64.

The content processing unit 82 performs the processing defined as the content by using the real space information, and acquires the display image and the output sound. For example, the content processing unit 82 generates a display image in a field of view corresponding to the posture of the head-mounted display 100 acquired by the real space information acquisition unit 80. Alternatively, a display image is generated in which the surrounding objects reflected in the captured image are replaced with virtual objects that reflect the position and orientation in the three-dimensional space. In this case, the content processing unit 82 acquires the captured image from the real space information acquisition unit 80, and superimposes and draws the virtual object.

In addition, those skilled in the art will understand that there are various modes in which real space information is reflected in the displayed image and sound of the content. The real space information acquisition unit 80 may transmit the acquired real space information to the content processing device 200 to generate display image and audio data in the content processing device 200. In this case, the content processing unit 82 may only perform necessary correction processing on the display image and audio data acquired from the content processing device 200.

For example, the content processing unit 82 may perform processing such as giving a distortion corresponding to the eyepiece included in the head-mounted display 100 to the display image. The content processing unit 82 supplies the display image data and the output audio data to the display unit 54 and the audio output unit 56, respectively, to display the content image and output the audio. As described above, by using the camera 110, which is normally used for generating content data, for user operations, it is possible to realize easy and diverse user operations without increasing the manufacturing cost.

However, the purpose of the camera 110 is not limited to those, and in some cases, a plurality of cameras 110 or a brightness detection device may be provided for user operation. Further, when the real space information is not used for generating the content, the function of the real space information acquisition unit 80 can be omitted. In these cases as well, the user can easily operate the content while enjoying the content without having to fumble for the switch or remove and adjust the attached head-mounted display 100. Note that some or all the functions of the information processing device 70 may be provided in the content processing device 200.

Next, the operation of the information processing apparatus 70 that can be realized by the configuration described so far will be described. FIG. 6 is a flowchart showing a procedure in which the information processing device 70 processes contents while responding to user operations. This flowchart starts with the user wearing the head-mounted display 100 in a power-on state. First, the luminance information acquisition unit 72 acquires the luminance information at a plurality of locations of the head-mounted display 100 (S10). When two-dimensional captured images are acquired by a plurality of cameras 110, the luminance information acquisition unit 72 acquires the data of the first frame of the moving image captured by each in this step.

Then, the user operation determination unit 74 confirms whether or not a predetermined change as described above has occurred in at least one of the luminance information (S12). If a predetermined change has not occurred (N in S12), the real space information acquisition unit 80 analyzes the luminance information (captured image) to acquire the real space information, and the content processing unit 82 uses the information to obtain the content. Process and acquire output data (S18). The acquired display image and audio data are appropriately output from the display unit 54 and the audio output unit 56. If a predetermined change has occurred in at least one of the luminance information (Y in S12), the operation content acquisition unit 76 has set at least one of the position of the camera in which the change has occurred, the combination of positions, and the content of the change. Acquire the associated operation content (S14).

Then, the operation processing unit 78 executes the process corresponding to the operation content acquired in S14 (S16). In this case as well, the real space information acquisition unit 80 analyzes the captured image in which no change has occurred to acquire the real space information, and the content processing unit 82 uses the captured image to process the content (S18). If it is not necessary to end the content processing by ending the content or performing an operation to stop the processing by the user (N in S20), the processing from S12 to S18 is performed next based on the acquired luminance information. repeat.

However, once it is determined that a change has occurred in S12, the processing of S12 and S14 is skipped and the processing corresponding to the operation of S16 is continued while the state continues. .. By these processes, the image and sound of the content are continuously output regardless of the presence or absence of the user operation, and on the other hand, the adjustment and the control corresponding to the user operation are performed. Then, when it becomes necessary to end the processing of the content, all the processing is ended (Y in S20).

FIG. 7 exemplifies the information that the operation content acquisition unit 76 internally holds in which the change in the luminance information and the operation content are associated with each other. In this example, as in FIG. 1, a head-mounted display 100 including four cameras A, B, C, and D is assumed. Further, as described in 1 above, an example of a case where it is determined that the light incident on the camera is blocked by the user's hand as a user operation is shown. In the illustrated user operation correspondence information 90, the camera in which the incident light is blocked is shown by filling. However, it does not mean to limit the format of information. Further, as described above, if it is found that the hand is held by setting a threshold value for the amount of reduction in brightness, the incident light does not need to be completely blocked.

For example, blocking the incident light on the camera A is associated with increasing the volume of the sound, and blocking the incident light on the camera B is associated with decreasing the volume of the sound. Further, the fact that the incident light to the camera C is blocked corresponds to the increase in the brightness of the display panel, and the fact that the incident light to the camera D is blocked corresponds to the decrease in the brightness of the display panel. In these cases, the volume and brightness may be gradually changed according to the duration of the blocked state.

Further, the interruption of the incident light to the cameras A and B is interpreted as the start of playback of the content, and the interruption of the incident light to the cameras B and C is interpreted as the pause. Further, blocking the incident light on the cameras C and D is interpreted as fast-forwarding the content, and blocking the incident light on the cameras D and A is interpreted as rewinding the content. In this case as well, the amount of fast forward and rewind may be gradually increased according to the duration of the blocked state. In the illustrated example, the maximum number of cameras in which the incident light is blocked at the same time is two, but the purpose is not limited to this.

Even with such a simple operation of holding a hand over the camera 110, various operations can be performed by using the plurality of cameras 110. Moreover, since it can be determined that the operation has been performed even if the incident light is not completely blocked, there is no need for an accurate operation such as finding a switch. As described above, by associating the held object, color, movement, etc. with the operation content, the operation content can be further subdivided by a simple operation.

FIG. 8 is a diagram for explaining a mode in which a user operation is determined based on changes in brightness information of light from a plurality of locations on the outer surface of the head-mounted display 100. The head-mounted display 100a in this case includes light emitters 130a, 130b, 130c, 130d instead of the cameras 110a, 110b, 110c, 110d of the head-mounted display 100 described with reference to FIG. However, even in this case, the number and arrangement of the light emitters are not limited. In this case, a camera 12 that captures the head-mounted display 100a from the outside is incorporated into the system, and the brightness information acquisition unit 72 of the information processing device 70 sequentially acquires images of the head-mounted display 100a that the camera 12 captures at a predetermined rate. To do.

Although the camera 12 is a stereo camera in the illustrated example, it may be a monocular camera if there is no other purpose such as specifying the position of the head-mounted display 100 in the real space. Then, the user operation determination unit 74 analyzes the captured image and detects that at least one of the light emitters 130a, 130b, 130c, and 130d is hidden by the user's hand based on the change in the image (luminance information). For example, the user operation determination unit 74 tracks the images of the light emitters 130a, 130b, 130c, and 130d by a general visual tracking technique, and if there is a light emitter that disappears and cannot be tracked, it is hidden by the user's hand. It is determined that it has been done.

In this case as well, the operation content acquisition unit 76 can determine the operation content based on the position of the light emitting body hidden by the hand and the combination thereof with reference to the same information as shown in FIG. Instead of the light emitting bodies 130a, 130b, 130c, and 130d that emit light by themselves, reflective members that reflect incident light may be provided at a plurality of locations on the head-mounted display 100. The reflecting member may be one for reflecting ambient light with high efficiency, or may be one that irradiates light in a predetermined wavelength band such as infrared rays from the outside and reflects the light.

In this case as well, the user operation can be determined based on the change in brightness in the image captured by the camera 12, as in the case of the light emitting body. Since it is premised that all the light emitting bodies and the reflecting members are shown in the captured image during the period when there is no user operation, the user wearing the head-mounted display 100 continues to face the camera 12. It is effective for such content.

According to the present embodiment described above, changes in the luminance information at a plurality of locations on the outer surface of the head-mounted display are detected, and the content of the user operation is acquired based on the positions where the predetermined changes occur and their combinations. As a result, various operations can be performed with simple operations without the trouble of finding and distinguishing switches and buttons. As a result, it is possible to realize an easy-to-use head-mounted display with few design restrictions.

In addition, in a system that acquires information necessary for determining the field of view of an image to be displayed and generating an image of contents from an image captured by a camera provided in a head-mounted display, the captured image can be used as it is. Therefore, the manufacturing cost can be suppressed. For such purposes, a plurality of cameras are usually provided, so that even if some of them are used for user operations, information necessary for content processing can be acquired in parallel.

At this time, since the user can operate the head-mounted display without contacting it, the effect on accuracy such as posture tracking can be minimized compared to the case where vibration occurs due to switch operation, etc., which can interfere with the normal operation of content output. Sex can be suppressed. Further, if the images taken by a plurality of cameras are used, there is less erroneous recognition of the object held by the cameras, and the user operation can be determined with high accuracy.

The present invention has been described above based on the embodiments. The above-described embodiment is an example, and it is understood by those skilled in the art that various modifications are possible for each of these components and combinations of each processing process, and that such modifications are also within the scope of the present invention. is there.

50 CPU, 52 main memory, 54 display unit, 56 audio output unit, 60 input / output interface, 62 communication unit, 64 motion sensor, 66 imaging unit, 70 information processing device, 72 brightness information acquisition unit, 74 user operation judgment unit, 76 Operation content acquisition unit, 78 Operation processing unit, 80 Real space information acquisition unit, 82 Content processing unit, 100 Head mount display, 110 camera, 130a light emitter, 200 Content processing device.

Claims (15)

A brightness information acquisition unit that acquires light brightness information at multiple locations on the outer surface of the head-mounted display,
A user operation determination unit that determines that a user operation has been performed when a predetermined change occurs in at least one of the luminance information at the plurality of locations.
When it is determined that a user operation has been performed, an operation content acquisition unit that acquires the operation content associated with the change in the luminance information as a user operation, and an operation content acquisition unit.
An operation processing unit that executes processing corresponding to the operation content, and
An information processing device characterized by being equipped with. The information processing device according to claim 1, wherein the luminance information acquisition unit acquires images taken by cameras arranged at each of the plurality of locations. The information processing apparatus according to claim 2, further comprising a real space information acquisition unit that acquires information related to the real space necessary for generating a display image by using at least one of the captured images. The user operation determination unit determines that the user operation has been performed when at least one of the captured images satisfies a predetermined criterion in which the light incident on the camera is considered to be blocked.
The information processing apparatus according to claim 2 or 3, wherein the operation content acquisition unit acquires operation content based on a position or a combination thereof in which the captured image satisfies the predetermined criteria among the plurality of locations. .. The user operation determination unit determines that the user operation has been performed when at least one of the captured images satisfies a predetermined criterion in which a predetermined object is considered to be held in front of the camera.
Claim 2 or 3 characterized in that the operation content acquisition unit acquires operation content based on a position or a combination thereof in which the captured image satisfies the predetermined criteria among the plurality of locations, and an object held over. The information processing device described in. The user operation determination unit determines that the user operation has been performed when at least one of the captured images satisfies a predetermined criterion in which an object of a predetermined color is considered to be held in front of the camera.
2. The operation content acquisition unit is characterized in that the operation content acquisition unit acquires the operation content based on the position or combination thereof in which the photographed image satisfies the predetermined criteria among the plurality of locations and the color of the object held over. Or the information processing apparatus according to 3. The user operation determination unit determines that the user operation has been performed when at least one of the captured images satisfies a predetermined criterion in which an object held in front of the camera is considered to have made a predetermined movement. ,
The information processing device according to claim 5 or 6, wherein the operation content acquisition unit further acquires operation content based on the movement. The user operation determination unit obtains the movement of the object held over in the three-dimensional space by using the images captured by the plurality of cameras, and collates it with the predetermined reference to perform the user operation. The information processing apparatus according to claim 7, wherein the information processing apparatus is determined. After confirming that there is no contradiction between the positional relationship of the images on the images captured by the plurality of cameras and the positional relationship of the plurality of cameras, the user operation determination unit holds an object in front of the cameras. The information processing apparatus according to any one of claims 5 to 8, wherein the information processing apparatus is characterized in that the above is determined. The luminance information acquisition unit acquires an image of the head-mounted display provided with light emitters or reflective members at a plurality of locations from an external camera.
The user operation determination unit is characterized in that it determines that a user operation has been performed when at least one of the luminance information of the image of the light emitting body or the reflective member in the image has changed. The information processing apparatus according to 1. The information processing according to any one of claims 1 to 10, wherein the operation processing unit adjusts an output state of a display unit that displays an image on the head-mounted display or an audio output unit that outputs audio. apparatus. The operation processing unit turns off the power of the head-mounted display, increases / decreases the brightness of the display panel, increases / decreases the volume of audio, starts playback of the content to be output, pauses, fast-forwards, rewinds, switches the content, and changes the image. A claim comprising zooming up, zooming down, adjusting the length of the wearing band of the head-mounted display, adjusting the distance between the left and right eyepieces, and adjusting the distance between the display images for the left eye and the right eye. The information processing apparatus according to any one of 1 to 10. The information processing device according to any one of claims 1 to 12 and
A camera that is arranged at each of the plurality of locations and captures an image as the luminance information,
An output device controlled by the operation processing unit and
A head-mounted display featuring. A step of acquiring light brightness information at multiple locations on the outer surface of a head-mounted display at a predetermined rate,
A step of determining that a user operation has been performed when a predetermined change occurs in at least one of the luminance information at the plurality of locations, and a step of determining that the user operation has been performed.
When it is determined that a user operation has been performed, a step of acquiring the operation content associated with the change in the luminance information as a user operation, and
The step of executing the process corresponding to the operation content and
A user operation processing method by an information processing device, which comprises. A function to acquire light brightness information at multiple locations on the outer surface of the head-mounted display at a predetermined rate,
A function for determining that a user operation has been performed when a predetermined change occurs in at least one of the luminance information at the plurality of locations, and a function for determining that a user operation has been performed.
When it is determined that a user operation has been performed, a function to acquire the operation content associated with the change in the luminance information as a user operation, and
A function to perform processing corresponding to the above operation contents and
A computer program characterized by realizing a computer.

Images