JP7085578B2 - Information processing device, user guide presentation method, and head-mounted display - Google Patents

Description

The present invention relates to an information processing apparatus that performs information processing based on a captured image, a user guide presentation method that is performed by the information processing apparatus, and a head-mounted display that displays an image.

A head-mounted display (hereinafter referred to as "HMD") connected to a game machine is attached to the head, and a game is played while looking at the displayed screen (see, for example, Patent Document 1). For example, if the position and posture of the user's head are acquired and the image of the virtual world is displayed so as to change the field of view according to the direction of the face, it is possible to produce a situation as if the user has entered the virtual world. The position and posture of the user are generally acquired based on the analysis result of the visible light or infrared image taken by the user, the measured value of the motion sensor built in the HMD, and the like.

Japanese Patent No. 5580855

Techniques for performing some kind of information processing based on captured images are based on the premise that an object such as a user is within the angle of view of the camera. Further, depending on the content of information processing, a suitable area in which the user should be may be further limited. However, when the HMD is attached, the user cannot see the outside world, so that he / she may lose his / her sense of direction or move to an unexpected position in the real space so much that he / she is absorbed in the game. This causes inconveniences such as deterioration of information processing accuracy, failure, and collision with other objects, but it is difficult to find the cause unless the HMD is removed. If such a situation occurs frequently, the world view of the virtual space constructed by using the HMD is hindered.

The present invention has been made in view of these problems, and an object of the present invention is to provide a technique for continuously enjoying the world expressed by an HMD with a small burden.

One aspect of the present invention relates to an information processing apparatus. This information processing device generates and outputs image data to be displayed as a result of information processing, a position information acquisition unit that acquires the position information of the head mount display, and an information processing unit that performs information processing using the information processing. The user guide generation unit includes an output data generation unit for generating information and a user guide generation unit for generating and outputting data of a user guide image representing the user's position information in the real space using the position information. It is characterized in that whether or not to display the guide is determined according to the situation specified by the position information.

Another aspect of the present invention relates to a user guide presentation method. In this user guide presentation method, the information processing device generates and outputs image data to be displayed as a result of information processing, a step of acquiring the position information of the head mount display and a step of performing information processing using the position information. The step of generating and outputting the data of the image of the user guide including the step of generating and outputting the data of the image of the user guide representing the position information of the user in the real space using the position information. It is characterized in that whether or not to display the user guide is determined according to the situation specified by the position information.

Yet another aspect of the invention relates to a head-mounted display. This head-mounted display is a head-mounted display that displays an image generated by the information processing device, and the communication unit that sends and receives data to and from the information processing device and the information processing device display the position information of the head-mounted display. The user guide is provided with a display unit that displays an image generated as a result of information processing performed using the user and a user guide image showing the user's position information in the real space using the position information. It is characterized by switching between display and non-display according to the situation specified by.

It should be noted that any combination of the above components and the conversion of the expression of the present invention between a method, an apparatus, a system, a computer program, a recording medium on which a computer program is recorded, and the like are also effective as aspects of the present invention. ..

According to the present invention, a user wearing an HMD can continuously enjoy the world of expression with a small burden.

It is a figure which shows the structural example of the information processing system to which this embodiment is applied. It is a figure which shows the example of the appearance shape of the HMD in this embodiment. It is a figure which shows the internal circuit structure of the information processing apparatus in this embodiment. It is a figure which shows the internal circuit structure of the HMD in this embodiment. It is a figure which shows the structure of the functional block of the information processing apparatus in this embodiment. It is a figure for demonstrating the information which can be acquired from a photographed image in this embodiment. It is a figure which illustrates the image at the time of displaying the bird's-eye view as a user guide in this embodiment. It is a figure which illustrates the display screen which displayed the image of the user guide in this embodiment. It is a figure which illustrates the image at the time of displaying the side view as a user guide in this embodiment. It is a figure which shows another image example in the case of displaying a bird's-eye view as a user guide in this embodiment. It is a figure which shows another image example in the case of displaying a bird's-eye view as a user guide in this embodiment. It is a figure which illustrates the image at the time of displaying the point cloud as a user guide in this embodiment. It is a figure which shows another image example in the case of displaying a bird's-eye view as a user guide in this embodiment. It is a flowchart which shows the processing procedure which the information processing apparatus generates the output data according to the movement of a user in this embodiment.

FIG. 1 shows a configuration example of an information processing system to which this embodiment can be applied. The information processing system 8 includes an image pickup device 12 that captures an object, an information processing device 10 that performs information processing based on the captured image, a flat plate display 16 and an HMD 18 that display images obtained as a result of information processing, and a user. Includes an input device 14 to operate.

The information processing device 10 and the image pickup device 12, the input device 14, the flat plate display 16, and the HMD 18 may be connected by a wired cable or by a known wireless communication technique such as Bluetooth (registered trademark). Further, depending on the information processing performed by the information processing apparatus 10, the flat plate type display 16 may not be provided. Further, the external shape of these devices is not limited to those shown in the figure. Further, a device may be integrally provided with two or more of these devices. For example, the information processing device 10, the input device 14, and the flat plate display 16 may be realized by a mobile terminal provided with them.

The image pickup device 12 generates output data of a photographed image by performing general processing such as demosaic processing on a camera that photographs an object such as a user at a predetermined frame rate and the output signal thereof, and causes the information processing apparatus 10 to generate output data. It has a mechanism to send out. The camera is equipped with a visible light sensor used in general digital cameras and digital video cameras such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image pickup apparatus 12 may have only one camera, or may be a so-called stereo camera in which two cameras are arranged on the left and right at known intervals as shown in the figure.

Alternatively, the image pickup device 12 may be configured by combining a monocular camera and a device that irradiates an object with reference light such as infrared rays and measures the reflected light. When a stereo camera or a reflected light measurement mechanism is introduced, the position of the object can be obtained in a three-dimensional real space, and the information processing by the information processing device 10 and the image display by the display device can be further diversified. can. A method of identifying the distance of the subject from the camera using the stereo image taken by the stereo camera from the left and right viewpoints, and the TOF (Time of Flight) and pattern irradiation by measuring the reflection of the reference light. All the methods for specifying the distance of the subject from the camera are widely known.

Hereinafter, a mode in which the image pickup device 12 captures a stereo image will be mainly described, but as described above, the present embodiment is not limited to this, and the image pickup device 12 may include at least one camera. .. The information processing device 10 performs necessary information processing using the data transmitted from the image pickup device 12, and generates output data such as an image and a voice. Here, the content of the processing performed by the information processing apparatus 10 is not particularly limited, and may be appropriately determined depending on the function, application, electronic content, and the like required by the user.

For example, the information processing apparatus 10 performs general face detection and tracking processing on a captured image to advance a game in which a character reflecting the movement of the user who is an object appears, or commands the movement of the user. It converts to input and processes information. At this time, the movement of the input device 14 may be acquired by using the marker provided on the input device 14. Further, by tracking a plurality of markers provided on the outer surface of the HMD 18, the position and posture of the head of the user wearing the HMD 18 can be specified, and the virtual world seen from a moving viewpoint corresponding to the position can be displayed on the HMD 18. good. The output data generated by the information processing apparatus 10 is transmitted to at least the HMD 18.

The HMD 18 is a display device that displays an image on a display panel such as an organic EL panel located in front of the user when the user wears it on his / her head. For example, the image may be viewed stereoscopically by generating a parallax image viewed from the left and right viewpoints and displaying the display screen in the left and right regions divided into two. However, the present embodiment is not limited to this, and one image may be displayed on the entire display screen. The HMD 18 may further include a speaker or earphone that outputs sound at a position corresponding to the user's ear.

The flat plate type display 16 may be a television having a display for outputting a two-dimensional image and a speaker for outputting audio, and may be, for example, a liquid crystal television, an organic EL television, a plasma television, a PC display, or the like. Alternatively, it may be a display and a speaker of a tablet terminal or a mobile terminal. The input device 14 receives requests such as start, end, function selection, and input of various commands by the user, and supplies the information processing device 10 as an electric signal.

The input device 14 may be realized by any or a combination of general input devices such as a game controller, a keyboard, a mouse, a joystick, and a touch pad provided on the display screen of the flat plate display 16. The input device 14 may further include a light emitting marker composed of an element or a set thereof that emits light in a predetermined color. In this case, the information processing device 10 tracks the movement of the marker using the captured image, so that the movement of the input device 14 itself can be controlled by the user. The input device 14 may be composed of only a light emitting marker and a mechanism for gripping the light emitting marker.

FIG. 2 shows an example of the appearance shape of the HMD 18. In this example, the HMD 18 is composed of an output mechanism unit 102 and a mounting mechanism unit 104. The mounting mechanism unit 104 includes a mounting band 106 that goes around the head and realizes fixing of the device when the user wears it. The wearing band 106 is made of a material or structure whose length can be adjusted according to the head circumference of each user. For example, it may be an elastic body such as rubber, or a buckle or a gear may be used.

The output mechanism 102 includes a housing 108 having a shape that covers the left and right eyes when the HMD 18 is worn by the user, and includes a display panel inside so as to face the eyes when the HMD 18 is worn. The outer surface of the housing 108 is provided with light emitting markers 110a, 110b, 110c, 110d, and the like. The number and arrangement of the light emitting markers are not particularly limited, but in the present embodiment, they are provided at the four corners of the front surface of the housing of the output mechanism unit 102.

Further, light emitting markers 110e and 110f are also provided on both side surfaces behind the mounting band 106. By arranging the light emitting markers in this way, even if the user turns sideways or backwards with respect to the image pickup device 12, the situation can be specified based on the number and positional relationship of the light emitting markers in the captured image. .. The light emitting markers 110c and 110d are on the lower side of the output mechanism unit 102, and the light emitting markers 110e and 110f are on the outside of the mounting band 106. Since they are not originally visible from the viewpoint of FIG. 2, the outer periphery is represented by a dotted line.

FIG. 3 shows the internal circuit configuration of the information processing apparatus 10. The information processing device 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. Each of these parts is connected to each other via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 includes peripheral device interfaces such as USB and IEEE1394, a communication unit 32 composed of a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive and a non-volatile memory, and data to a flat plate display 16 and an HMD 18. An output unit 36 that outputs data, an image pickup device 12, an input device 14, an input unit 38 that inputs data from the HMD 18, and a recording medium drive unit 40 that drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory are connected. ..

The CPU 22 controls the entire information processing apparatus 10 by executing the operating system stored in the storage unit 34. The CPU 22 also executes various programs read from the removable recording medium, loaded into the main memory 26, or downloaded via the communication unit 32. The GPU 24 has a geometry engine function and a rendering processor function, performs drawing processing according to a drawing command from the CPU 22, and stores a display image in a frame buffer (not shown). Then, the display image stored in the frame buffer is converted into a video signal and output to the output unit 36. The main memory 26 is composed of a RAM (Random Access Memory) and stores programs and data necessary for processing.

FIG. 4 shows the internal circuit configuration of the HMD 18. The HMD 18 includes a CPU 50, a main memory 52, a display unit 54, and an audio output unit 56. Each of these parts is connected to each other via a bus 58. An input / output interface 60 is further connected to the bus 58. A communication unit 62, an acceleration sensor 64, and a light emitting unit 66, which are wired or wireless LAN network interfaces, are connected to the input / output interface 60.

The CPU 50 processes the information acquired from each unit of the HMD 18 via the bus 58, and supplies the output data to the display unit 54 and the audio output unit 56. The main memory 52 stores programs and data necessary for processing in the CPU 50. However, depending on the design of the application or device to be executed, the information processing device 10 may perform almost all processing, and the HMD 18 may be sufficient to output the data transmitted from the information processing device 10. In this case, the CPU 50 and the main memory 52 can be replaced with a simpler device.

The display unit 54 is composed of a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of a user wearing the HMD 18. As described above, stereoscopic vision may be realized by displaying a pair of parallax images in the regions corresponding to the left and right eyes. The display unit 54 may further include a pair of lenses that are located between the display panel and the user's eyes when the HMD 18 is attached and that expand the user's viewing angle.

The voice output unit 56 is composed of a speaker or earphone provided at a position corresponding to the user's ear when the HMD 18 is attached, and allows the user to hear the voice. The number of audio channels to be output is not particularly limited, and may be monaural, stereo, or surround. The communication unit 62 is an interface for transmitting and receiving data to and from the information processing device 10 and the flat plate display 16, and can be realized by using a known wireless communication technique such as Bluetooth (registered trademark).

The acceleration sensor 64 detects the inclination of the HMD 18 by measuring the gravitational acceleration in a predetermined axial direction. The HMD 18 may be further provided with various other sensors such as a gyro sensor. The measured value by the sensor is transmitted to the information processing apparatus 10 via the communication unit 62. The light emitting unit 66 is an element or a set thereof that emits light in a predetermined color, and is provided at a plurality of locations on the outer surface of the HMD 18 as shown in FIG. By tracking this as a marker, the position of the HMD 18 is acquired, and the posture of the HMD 18 is acquired from the number and positional relationship of the images in the captured image.

The information processing apparatus 10 can acquire the position and posture of the user's head with higher accuracy by integrating the information obtained by a plurality of means such as the acceleration sensor 64 and the light emitting unit 66. On the other hand, in the present embodiment, the accelerometer 64 may be omitted in some cases.

FIG. 5 shows the configuration of the functional block of the information processing apparatus 10. Each functional block shown in FIG. 5 can be realized by the configuration of the CPU, GPU, various memories, data buses, etc. shown in FIG. 3 in terms of hardware, and is loaded into the memory from a recording medium or the like in terms of software. It is realized by a program that exerts various functions such as data input function, data retention function, image processing function, and communication function. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and the present invention is not limited to any of them.

The information processing device 10 performs information processing according to contents such as an input information acquisition unit 72 that acquires input information from the input device 14 and the HMD 18, a captured image acquisition unit 74 that acquires captured image data from the image pickup device 12, and a game. It includes an information processing unit 76 that performs information processing, an output data generation unit 78 that generates data to be output, and a content data storage unit 84 that stores data necessary for information processing and image generation. Further, the information processing apparatus 10 further includes a position / attitude information acquisition unit 80 that acquires information on the user's position and attitude based on a captured image and the like, and a user guide generation unit 82 that generates information related to the user's situation in real space as a user guide. , A guide data storage unit 85 that stores user guide presentation rules and data necessary for output, and an output data transmission unit 86 that transmits data to be output to the HMD 18.

The input information acquisition unit 72 acquires the content of the user operation from the input device 14. Here, the user operation is performed by a general information processing device such as selection of an application or content to be executed, start / end of processing, command input, etc., as well as display / non-display of a user guide described later, as a user guide. Operations on the user guide, such as switching the orientation of the image to be displayed, are also included. The input information acquisition unit 72 supplies the information acquired from the input device 14 to the captured image acquisition unit 74, the information processing unit 76, or the user guide generation unit 82 according to the content thereof. The input information acquisition unit 72 further receives the measured value from the acceleration sensor 64 of the HMD 18 at a predetermined rate and supplies it to the position / attitude information acquisition unit 80.

The captured image acquisition unit 74 acquires captured image data such as a stereo image obtained by imaging a moving image by the image pickup apparatus 12 at a predetermined rate. The captured image acquisition unit 74 further controls the start / end of imaging in the image pickup apparatus 12 according to the processing start / end request from the user acquired by the input information acquisition unit 72, and responds to the processing result in the information processing unit 76. The type of data acquired from the image pickup apparatus 12 may be controlled.

The position / posture information acquisition unit 80 acquires the position and posture information of the user at a predetermined rate by detecting an image of a predetermined object from the captured image. For example, the positions of the user's head and hands in the real space are acquired based on the images of the light emitting markers provided on the HMD 18 and the input device 14. Alternatively, an image analysis technique such as tracking a part of the user's body using contour lines or detecting a face or an object having a specific pattern by pattern matching may be appropriately combined. Further, depending on the configuration of the image pickup apparatus 12, the user's distance may be specified by measuring the reflection of infrared rays as described above. The position / posture information acquisition unit 80 may further integrate the measurement results of the acceleration sensor of the HMD 18 to specify the posture of the user's head in more detail. Further, as will be described later, depending on the content of the presented user guide, the position / posture information acquisition unit 80 may also detect the position of a subject other than the user.

The information processing unit 76 processes electronic contents such as games specified by the user. This process includes the use of the user's position and posture information acquired by the position / posture information acquisition unit 80. As described above, the content of the information processing in the subsequent stage performed by the information processing unit 76 is not particularly limited according to the user operation or the movement of the user via the input device 14.

The output data generation unit 78 generates image and audio data to be output as a result of information processing in accordance with the request from the information processing unit 76. For example, a virtual world viewed from a viewpoint corresponding to the position and posture of the user's head is generated as a left-right parallax image. By displaying this parallax image in front of the left and right eyes on the HMD 18 or outputting the sound in the virtual world, the user can feel as if he / she has entered the virtual world. The program and image / audio data required for information processing in the information processing unit 76 and data generation processing in the output data generation unit 78 are stored in the content data storage unit 84.

The user guide generation unit 82 confirms the surrounding situation in the real world based on the position and posture information of the user and other subjects acquired by the position / posture information acquisition unit 80, and suggests actions to be taken thereafter. Performs the process related to the presentation of the user guide for giving. For example, an easy-to-understand format such as illustrating information such as a relative position with the image pickup device 12, a suitable range of the user's position (hereinafter referred to as "play area"), and a moving direction for returning to the center of the play area. To be presented at.

By superimposing the user guide representing such information on the running game screen or the like, it is possible to grasp the situation in the real world without removing the HMD even in the middle of the game. The user guide generation unit 82 causes the user guide to be displayed at any time according to the user guide display operation by the user. Alternatively, depending on the situation such as warning of danger, the user guide may be displayed regardless of the user operation.

The guide data storage unit 85 contains information such as the angle of view of the camera of the image pickup apparatus 12, the setting of the play area for the camera, the setting information such as the conditions for outputting the user guide and the content to be presented, and the image data necessary for generating the user guide. Store in and read as needed. Information other than the angle of view of the camera may be set in association with the content of information processing performed by the information processing unit 76. The user guide generation unit 82 may present the user guide by voice in addition to displaying the image. In this case, the voice data to be output according to the situation is also stored in the guide data storage unit 85. Specific examples of the user guide will be described later.

The user guide generation unit 82 supplies the generated user guide data to the output data generation unit 78. The user guide should always reflect changes in the situation, such as the position of the user. Therefore, the user guide generation unit 82 continues to generate and supply data at a predetermined rate for a period in which the condition for displaying the user guide is satisfied. The output data generation unit 78 superimposes the supplied user guide image and sound on the image and sound generated as a result of the information processing of the information processing unit 76. The output data transmission unit 86 sequentially acquires the output data generated by the output data generation unit 78, forms it as necessary, and transmits it to the HMD 18.

FIG. 6 is a diagram for explaining information that can be acquired from a captured image in the present embodiment. In the figure, the user 120 holds the input device 14 and wears the HMD 18. The input device 14 includes a light emitting marker 122 at a position facing the image pickup device 12 when gripped by a holding method suitable for operation. The emission marker of HMD18 is as shown in FIG. When the image pickup device 12 is a stereo camera, the distance Z from the image pickup surface of the image pickup device 12 to each light emitting marker is obtained based on the parallax of the image in the stereo image. Further, the position of the image of the light emitting marker on the image plane (XY plane) of either photographed image represents the apparent position from the image pickup apparatus 12.

By integrating these information, specifically, by back-projecting the position in the XY plane using the distance Z from the image pickup apparatus 12, the position of each light emitting marker in the three-dimensional space in the real world can be obtained. Further, the attitude (vector va) of the HMD 18 in the real space is determined from the number and positional relationship of the image of the light emitting marker of the HMD 18, and the attitude (vector vb) of the input device 14 in the real space is determined from the shape of the image of the light emitting marker 122 of the input device 14. Desired.

The information processing unit 76 of the information processing device 10 has an image in a virtual world in which the field of view changes according to the orientation of the face of the user 120 or an image according to the movement of the input device 14 based on the position and posture of these devices in the real space. It is possible to express how the objects inside move. When the distance from the image pickup device 12 is estimated from the apparent size of the marker, the image pickup device 12 does not have to be a stereo camera. The same applies when introducing a distance measurement technique using reference light. Also, it is not always necessary to track both the HMD 18 and the input device 14.

In such a system, the user guide generation unit 82 recognizes the position and posture of the HMD 18 as the position of the user and the posture of the head, and illustrates the surrounding situation so that it can be grasped from the user's point of view. FIG. 7 illustrates an image when a bird's-eye view is displayed as a user guide. Both the user guides in FIGS. (A) and (b) represent a schematic view of the real space including the image pickup device and the user, and (a) is an image based on the position of the image pickup device, (a). b) is an image based on the orientation of the user. In both figures, the rectangle 130 representing the image pickup apparatus and the circle 132 representing the user are represented so as to reflect the actual positional relationship and orientation.

A bar indicating the direction in which the user is actually facing is added to the circle 132 representing the user. Both images are further shown with an area 134 representing a horizontal play area determined by the angle of view of the camera and the like, and an arrow 136 indicating the direction in which the user should move. The play area basically matches the angle of view of the camera in the horizontal or vertical direction, or sets a narrower range of positions suitable for information processing. Further, in the depth direction from the camera, a range in which the acquisition accuracy of the distance can be sufficiently maintained, or a range in which the position is narrower and suitable for information processing is set. The shape and size of the play area may be switched or selected by the user depending on the content of information processing such as the game to be performed and the size of the room.

The image of (a) is expressed so that the rectangle 130 representing the image pickup apparatus is always placed in the center of the upper side. From this image, the user can confirm that he / she is relatively behind the image pickup device 12 at the left end of the play area. If you move further to the left or backward, you can recognize that you are deviating from the play area and the direction of movement to return to the center of the play area. When the user actually moves while displaying such a user guide, the circle 132 representing the user is also moved so as to be interlocked with the user. By expressing the bird's-eye view that reflects the movement of the user in this way, the relative position between the image pickup device or the play area and the user is clarified, and the stride length and direction including sensation and ambiguity are visually and objectively. It is expressed as. As a result, the "field" recognized by the information processing apparatus 10 can be matched with the user's own sense of movement.

In the case of the expression (a), when the user does not face the image pickup device 12, the front / rear / left / right of the user does not match the top / bottom / left / right of the image. Therefore, even if you try to move in the direction of arrow 136, it is possible that you cannot move in that way. On the other hand, the image of (b) is expressed so that the direction in which the user is facing is always vertically upward in the image. In this case, when the direction of the user changes, the field itself including the rectangle 130 and the area 134 rotates in the relatively opposite direction. By doing so, since the front / back / left / right of the user himself / herself always matches the top / bottom / left / right of the image, the direction of the arrow 136 matches his / her sense of direction, and it becomes easy to move in that direction accurately.

It should be noted that the figure of the image pickup device or the like represented by the illustrated image is not limited to this. Further, the setting information such as the shape of the play area to be set, whether or not to guide by an arrow or the like, and the direction thereof when guiding may vary depending on the content and situation of information processing by the information processing unit 76. Further, depending on the content of the guidance, the arrow may be a curve such as a U shape. The length of the arrow may indicate the distance to be moved. Whether the reference for the orientation of the bird's-eye view is the image pickup device or the user's orientation may be switched depending on the content of the information processing, or may be switched by the user via the input device 14 or the like. The same applies to the user guides exemplified below.

FIG. 8 illustrates a display screen on which an image of a user guide is displayed. The display screen 200 has a configuration in which a user guide 202 as illustrated in FIG. 7 is superimposed and displayed on an image of contents such as a game screen. The user guide generation unit 82 displays the user guide 202 when the user performs an operation to display the user guide, and hides the user guide 202 when the user performs an operation to end the display. Or, depending on the situation, it is displayed regardless of the user operation.

In any case, by superimposing the user guide on the image of the original content, the information shown can be confirmed without significantly moving the line of sight. On the other hand, especially when the image of the content is a virtual world with a sense of depth, it is conceivable that the user may be surprised or uncomfortable by the appearance of a component image without a sense of depth in front of him.

Therefore, by blurring the periphery of the user guide 202 by blurring or α-blending as shown in the figure, the affinity with the virtual world expressed in the background while being in the front is enhanced. The entire user guide 202 may be translucent. Further, unless the information is displayed with urgency, it is desirable that the user guide 202 is displayed at a position avoiding the gazing point. For example, in a display mode in which the field of view is changed according to the line of sight of the user, the viewpoint is inevitably fixed at the center of the screen, so that the display is performed at a position separated from the center of the screen by a predetermined distance or more.

Also, since it is easier for people to lower their eyes than to raise them, by displaying the user guide 202 in the lower half area of the screen (the area below the center line C in the vertical direction), it is impossible to shift the gaze point from the image of the content. It can be moved without. Alternatively, the game operation or the like can be continued as it is by checking the user guide 202 with the eyes while keeping the gazing point on the content image. By these measures, it is possible to prevent the eyes from being out of focus immediately on the displayed user guide and to be intoxicated by small eye movements.

FIG. 9 illustrates an image when a side view is displayed as a user guide. In this example, a schematic view of a real space including an image pickup device and a user as viewed from the left side surface of the user is shown. That is, the rectangle 140 representing the image pickup device and the object 142 representing the user are represented so as to reflect the actual distance and orientation. Further, the image also shows an area 144 indicating a vertical play area, which is determined by the angle of view of the camera and the like, and an arrow 146 indicating the direction of movement that the user should take.

Even with such a display, it is possible to grasp the situation as described with reference to FIG. 7. For example, the user can grasp that the head is likely to come out of the angle of view due to being too close to the image pickup device, or the processing accuracy cannot be maintained because the head is too far away. Furthermore, when the posture is defined by the content of information processing, it is possible to encourage the user who has other postures to have the correct posture. For example, in a game on the premise that the user is seated, when the user suddenly stands up, the arrow 146 can be displayed to encourage the user to sit down.

This example is compared with the image shown in FIG. 7, and the figure of the user is represented as a more detailed object. As a result, the actual posture of the user can be reflected in detail, but depending on the content to be displayed, a simple figure as illustrated in FIG. 7 may be used. Animations that change such objects and figures may also be used to encourage them to take the correct posture. For example, an animation in which the figure expands may be used to encourage the figure to stand up, or an animation in which the figure is reduced may be used to encourage the figure to sit down.

FIG. 10 shows another example of an image when displaying a bird's-eye view as a user guide. In this example, as in the image of FIG. 7, the rectangle 150 representing the image pickup apparatus and the circle 152 representing the user are shown so as to reflect the actual positional relationship and orientation. On the other hand, in the figure, the information related to the angle of view of the camera is represented by the line 154 instead of the play area. Specifically, it represents a radial straight line representing the direction that divides the angle of view of the camera into four equal parts, and a concentric arc that represents the distance from the image pickup device at predetermined intervals.

When the entire area within the field of view of the camera is set as the play area, the line 154 also indicates the play area. For example, before executing information processing such as a game, such a user guide is displayed with the HMD 18 attached, and the user actually moves while looking at it, so that the amount and direction of his / her movement with respect to the field of view of the camera And can understand the correspondence with the senses such as stride length and direction recognition. By superimposing such a user guide on the virtual world displayed on the HMD, not only before information processing, the scenery of the virtual world is associated with your position in the field of view of the camera, with that in mind. Can work.

Although the illustrated example is a bird's-eye view in which the image pickup apparatus is placed on the upper side, it may be a bird's-eye view in which the user's direction is vertically upward as shown in FIG. 7B with the same configuration. Further, in a side view as shown in FIG. 9, a line that equally divides the angle of view in the vertical direction and a line that represents the distance from the image pickup apparatus may be shown. Further, a three-dimensional structure including a surface that equally divides the angle of view in the horizontal direction and the vertical direction and an equidistant surface from the image pickup device may be defined, and a state in which the structure is viewed from a desired viewpoint may be drawn. By superimposing the state of such a structure viewed from the user side on the image of the virtual world displayed on the HMD 18, the correspondence between the angle of view and the virtual world can be understood at a glance. good.

FIG. 11 shows another image example when the bird's-eye view is displayed as a user guide. In this example, the direction in which the user is facing is always vertically upward. Therefore, the circle 162 representing the user is represented in the center of the lower end of the image, and the rectangle 160 representing the image pickup device is represented at a position reflecting the relative position with the user. Further, in the figure, instead of the information related to the angle of view of the camera of FIG. 10, the information related to the user's field of view is represented by the line 164.

Specifically, it represents a radial straight line representing an orientation that divides the effective viewing angle of the user into four equal parts, and a concentric arc that represents the distance from the user at predetermined intervals. Here, the effective viewing angle is a general numerical value representing an angle at which an object can be accurately recognized by the brain. With such a user guide, even if the real world cannot be seen by the HMD, it is possible to easily understand what is in the original field of view and the position of the image pickup device in the field of view. This makes it possible to intuitively determine in which direction and how much to move to face the image pickup device. Although the illustrated example is a bird's-eye view based on the orientation of the user, it may be a bird's-eye view in which the image pickup apparatus is placed on the upper side as in FIG. Further, the images shown in FIGS. 10 and 11 may be further displayed with arrows for guiding the user.

FIG. 12 illustrates an image when the point cloud is displayed as a user guide. The examples shown in FIGS. 7, 9, 10 and 11 are a two-dimensional view of the real world from a bird's-eye view or a side view, but by showing a state in which the real world is viewed from an angle, a three-dimensional space is shown. It may be easy to grasp as. FIG. 12 is an example thereof, in which a point cloud 172 representing a subject is represented in a three-dimensional space corresponding to the real world together with a graphic 170 representing an image pickup device, and the point cloud 172 is drawn as an image. The point cloud 172 represented here is a set of points representing the surface of the subject, which is obtained by back-projecting the depth map of the subject including the user into a three-dimensional space.

As described above, the depth map is an image in which the distance in the depth direction is acquired for each pixel based on the parallax of the stereo image and the value is expressed as a pixel value, and is generated by the position / attitude information acquisition unit 80. The user guide generation unit 82 arranges the points constituting the point cloud 172 in the global coordinate system by plotting each pixel of the depth map in a virtual three-dimensional space based on the distance represented by the pixel value. Objects of the image pickup device and a plane representing the floor surface are also arranged in the three-dimensional space. Then, by projecting them onto a predetermined screen coordinate system, an image as shown in the figure can be generated.

The depth map changes according to the change in the position and posture of the user and other subjects, and the point cloud 172 also changes. As a result, it is possible to objectively grasp one's position and movement with respect to the image pickup device as in the examples described so far. Further, by making it possible to accept a user operation that changes the position and posture of the screen surface, the user can confirm the relative position from an angle that is easy for the user to understand.

Furthermore, when creating a depth map by extracting an image from the entire stereo image, subjects other than the user are also drawn at the same time, so there are obstacles around, children, pets, etc. have not invaded the play area. You can also confirm that. The example shown is composed of only a figure 170 representing an image pickup device and a point cloud 172, but as in the previous examples, it represents a play area or an angle of view of a camera, and an arrow indicates the direction in which the user should move. It may be indicated by.

FIG. 13 shows another image example when the bird's-eye view is displayed as a user guide. In this example, as in the image of FIG. 7, the rectangle 180 representing the image pickup apparatus, the circle 182 representing the user, and the area 184 representing the play area are represented so as to reflect the actual positional relationship and orientation. Further, in the figure, the existence of objects other than the user in the play area is represented by figures associated with each. Specifically, a circle mark 186 representing an input device and a cross mark 188 representing an obstacle are shown.

The position of the input device or obstacle is obtained by extracting an image other than the user's image by the above-mentioned depth map or background subtraction method. Among them, the input device can be distinguished from other objects by the external light emitting marker, the shape of the input device itself, and the like. Also, objects that are moving other than the user can be extracted by taking the motion difference from the previous frame. With such a user guide, even if the user misplaces the input device or needs the input device in the middle of information processing, the HMD can be moved to the location of the input device and gripped without removing the HMD. In addition, even if there is an obstacle in the surroundings or a child or pet enters the play area, the danger of collision can be avoided by taking precautions such as recognizing it and dismissing it.

In the example of the figure, the figure representing the input device and the figure representing the obstacle are shown at the same time, but of course, either one may be shown depending on the situation. In particular, in a dangerous situation such as when an obstacle is detected in the play area, the user guide showing the position in a graphic shape may be displayed at a position easily noticed by the user, such as the center of the screen, regardless of the user operation. It is desirable that the figures representing the user, the input device, and the obstacle be in a form that can be distinguished at first glance. In addition to making the shapes significantly different as shown, the colors may be distinguished and various different processes may be applied.

Further, depending on the situation, it may be expressed by an animation in which the figure blinks or slightly vibrates. For example, when the user is about to leave the play area or when an obstacle invades the play area, such an animation display can be used to call the user's attention. If the image pickup device 12 itself falls or changes its direction due to some external force, the user may be notified by blinking the rectangle 180 representing the image pickup device. Such a change in the image pickup apparatus 12 is caused by the position / attitude information acquisition unit 80 or the user guide generation unit 82 by separately acquiring the measured value by the acceleration sensor built in the image pickup apparatus 12 by the captured image acquisition unit 74. You just have to detect it.

Next, the operation of the information processing apparatus 10 that can be realized by the configuration described so far will be described. FIG. 14 is a flowchart showing a processing procedure in which the information processing apparatus generates output data according to the movement of the user in the present embodiment. This flowchart is started when the user requests the information processing apparatus 10 to start processing via the input device 14.

First, the captured image acquisition unit 74 of the information processing device 10 requests the image pickup device 12 to start shooting, and accordingly, starts acquiring data of the image captured and output by the image pickup device 12 (S10). On the other hand, the user guide generation unit 82, from the guide data storage unit 85, sets information such as the angle of view of the camera of the image pickup device 12, the play area for the angle of view, the timing at which the user guide should be displayed, and the content to be displayed accordingly. Is read (S12). At this time, the setting information associated with the content to be executed, such as the user's selection, may be selected and read.

Next, the position / posture information acquisition unit 80 acquires the user's position / posture information based on the captured image (S14). Then, the information processing unit 76 performs information processing using the information, and the output data generation unit 78 draws a display image of the content as a result (S16). On the other hand, the user guide generation unit 82 confirms whether or not the situation at that time satisfies the condition for displaying the user guide (S18). Here, the conditions for displaying the user guide include when the user operates to display the user guide, when there is a danger such as an obstacle in the play area, or when the orientation of the image pickup apparatus 12 is abnormal. There are times when a factor that hinders the normal progress of information processing is detected.

The user guide generation unit 82 sets a flag to be held in an internal register or the like when such a state occurs, lowers the flag when such a state is resolved, and based on the value of the flag. Conditionally branch the process. If the condition for displaying the user guide is not satisfied (N in S18), the output data transmission unit 86 outputs the image of the content drawn in S16 to the HMD 18 as it is (S20). When the condition for displaying the user guide is satisfied (Y in S18), the user guide generation unit 82 generates an image of the user guide based on the position and posture information acquired by the position / posture information acquisition unit 80.

Then, by supplying the data to the output data generation unit 78, the output data generation unit 78 superimposes the image of the user guide on the image of the content (S20). Then, the output data transmission unit 86 outputs the image on which the user guide is superimposed to the HMD 18 (S22). At the same time in the process of S20, the voice data of the user guide may be superimposed on the voice data of the content. For example, specific expressions or instructions such as "please crouch" or "move to the right" may be used, or when the user approaches the boundary line of the play area, a predetermined warning sound is emitted on the boundary line. The sound image localization technique may be used, for example, the sound image may be localized to the sound image.

The data to be output as the user guide may be a combination of images and sounds, or either one may be used. Further, depending on the setting, the image taken by the image pickup apparatus 12 may be output to the HMD 18 as it is as a user guide. As a result, the user can judge from the actual image whether there is an obstacle in the field of view of the camera or whether there is an abnormality in the installation of the camera. In this case, the user guide generation unit 82 superimposes and displays text information such as "Are there any dangerous objects?" And a GUI (Graphical User Interface) for inputting that the user has confirmed it on the captured image, and confirms it to the user. You may be prompted for input.

During the period when it is not necessary to end the process such as when the user requests to stop the process, the image of the content is drawn based on the location information, and during the period when the conditions are met, the user guide is superimposed and output to the HMD18. , Repeat at a predetermined rate (N of S24, S14 to S22). Then, when it becomes necessary to end the processing, all the processing is terminated (Y in S24).

According to the present embodiment described above, in a system that acquires user's position information based on an image in a captured image and processes information using the result, information related to the surrounding situation of the user in the real world. Is presented as a user guide. As a result, even if the user wears the HMD and cannot see the outside world, he / she can recognize his / her position and moving direction in the real world, so that the angle of view of the camera and the expected play area are deviated. It is possible to suppress the possibility of problems in information processing or collision with objects outside the angle of view. In addition, by simultaneously presenting the state of an object other than the user as a user guide, it is possible to avoid a collision in the play area.

Since the above confirmation can be performed without removing the HMD, the burden on the user for recognizing the current situation and avoiding danger is small. Furthermore, by superimposing the content on the screen at a suitable position when necessary, the content can be confirmed while looking at the image of the content expressed by the HMD, and the world view is not easily impaired. Furthermore, things that had to rely on the senses such as stride length and movement direction are visualized in correspondence with the position of the image pickup device and the range of the play area in the real world, so it becomes easier to learn appropriate movements. In addition, since the position and posture information used in the original information processing is used, it can be realized with a small processing load.

In addition, by allowing the user to switch the orientation standard when displaying the bird's-eye view and the viewpoint when displaying the three-dimensional space, the user can intuitively understand the direction in which he / she should move by displaying it in an easy-to-understand manner. Can be judged. Furthermore, by giving the user a movement guideline with an arrow or voice, it is possible to respond regardless of age or comprehension.

The present invention has been described above based on the embodiments. It is understood by those skilled in the art that the above-described embodiment is an example, and that various modifications are possible for each of these components and combinations of each processing process, and that such modifications are also within the scope of the present invention. be.

For example, in the embodiment, an image is presented as a user guide so that the positional relationship of the user with respect to the image pickup apparatus can be understood. On the other hand, when the position of the user is specified by a means other than the image pickup device for photographing the user, for example, a camera or a motion sensor mounted on the HMD, the positional relationship with an object other than the image pickup device may be expressed. For example, if the play area is set by some rule, it may only indicate the position of the user in the play area. Alternatively, when playing equipment or the like is installed in the real space or there are other users and the positional relationship with them is reflected in the information processing, the positions of those objects and people are shown in the user guide instead of the image pickup device. May be good.

8 Information processing system, 10 Information processing device, 12 Imaging device, 14 Input device, 16 Flat plate display, 18 HMD, 22 CPU, 24 GPU, 26 Main memory, 72 Input information acquisition unit, 74 Captured image acquisition unit, 76 Information Processing unit, 78 output data generation unit, 80 position / attitude information acquisition unit, 82 user guide generation unit, 85 guide data storage unit, 86 output data transmission unit.

Claims (16)

The location information acquisition unit that acquires the location information of the head-mounted display,
An information processing unit that performs information processing using the location information,
An output data generation unit that generates and outputs data of a display image of a virtual world to be displayed as a result of the information processing.
A user guide generator that uses the location information to generate and output user guide image data that represents the user 's location information in a bird's-eye view of real space.
Equipped with
The user guide generation unit is characterized in that the user guide is displayed when there is a user operation for displaying the user guide, or when a predetermined situation occurs in a real space regardless of the user operation. Information processing device. The location information acquisition unit that acquires the location information of the head-mounted display,
An information processing unit that performs information processing using the location information,
An output data generation unit that generates and outputs data of a display image of a virtual world to be displayed as a result of the information processing.
Generates user guide image data that shows the relationship between the user's position in real space based on the location information and the play area that is set as the range that the user should be and has the shape and size selected by the user. User guide generator to output and
Equipped with
The user guide generation unit is characterized in that the user guide is displayed when there is a user operation for displaying the user guide, or when a predetermined situation occurs in a real space regardless of the user operation. Information processing device. The user guide generation unit according to claim 1 or 2 , wherein the user guide generation unit detects the occurrence of the predetermined situation based on the positional relationship between the play area set as the range in which the user should be and the real object. Information processing device. The information processing apparatus according to claim 3 , wherein the user guide generation unit detects a situation in which a real object other than the user exists in the play area as the predetermined situation. The position information acquisition unit generates a depth map in which the distance from the image pickup device for photographing the real space to the subject is used as a pixel value.
The information processing apparatus according to claim 3 , wherein the user guide generation unit acquires the position of the real object by using the depth map. The information processing apparatus according to claim 3 or 4 , wherein the user guide generation unit acquires the position of the real object based on an image extracted by the background subtraction method from an image obtained by taking a picture of the real space. The user guide generation unit according to any one of claims 1 to 6 , wherein the user guide is displayed in the center of the display screen in response to the occurrence of the predetermined situation regardless of the user operation. Information processing device. The information processing apparatus according to claim 1 , wherein the user guide generation unit represents a direction in which the user is actually facing in the user guide. The information processing apparatus according to any one of claims 1 to 8 , wherein the user guide generation unit blurs the peripheral edge of the user guide. The information processing device according to any one of claims 1 to 9 , wherein the position information acquisition unit acquires the position information by an image pickup device mounted on the head-mounted display. Steps to get the location information of the head-mounted display,
The step of performing information processing using the position information and
The step of generating and outputting the data of the display image of the virtual world to be displayed as a result of the information processing, and
Using the above position information, the step of generating and outputting the data of the image of the user guide showing the user 's position information in a bird's-eye view of the real space, and
Including
The step of generating and outputting the image data of the user guide is the user when there is a user operation for displaying the user guide, or when the occurrence of a predetermined situation in the real space is detected regardless of the user operation. A method of presenting a user guide by an information processing device, which comprises displaying a guide. Steps to get the location information of the head-mounted display,
The step of performing information processing using the position information and
The step of generating and outputting the data of the display image of the virtual world to be displayed as a result of the information processing, and
Generates user guide image data that shows the relationship between the user's position in real space based on the location information and the play area that is set as the range that the user should be and has the shape and size selected by the user. Steps to output and
Including
The step of generating and outputting the image data of the user guide is the user when there is a user operation for displaying the user guide, or when the occurrence of a predetermined situation in the real space is detected regardless of the user operation. A method of presenting a user guide by an information processing device, which comprises displaying a guide. The function to acquire the position information of the head-mounted display and
A function that processes information using the location information and
The function to generate and output the data of the display image of the virtual world to be displayed as a result of the information processing, and
A function to generate and output user guide image data that represents the user 's position information in a bird's-eye view of the real space using the above position information.
To the computer,
The function of generating and outputting image data of the user guide is a function of generating and outputting the image data of the user guide when there is a user operation for displaying the user guide, or when a occurrence of a predetermined situation in the real space is detected regardless of the user operation. A computer program characterized by displaying a guide. The function to acquire the position information of the head-mounted display and
A function that processes information using the location information and
The function to generate and output the data of the display image of the virtual world to be displayed as a result of the information processing, and
Generates user guide image data that shows the relationship between the user's position in real space based on the location information and the play area that is set as the range that the user should be and has the shape and size selected by the user. Output function and
To the computer,
The function of generating and outputting image data of the user guide is a function of generating and outputting the image data of the user guide when there is a user operation for displaying the user guide, or when a occurrence of a predetermined situation in the real space is detected regardless of the user operation. A computer program characterized by displaying a guide. A head-mounted display that displays images generated by an information processing device.
A communication unit that sends and receives data to and from the information processing device,
A display image of a virtual world generated as a result of information processing performed by the information processing device using the position information of the head-mounted display, and a user 's position information using the position information are shown in a bird's-eye view of the real space. Equipped with an image of the user guide and a display unit that displays
A head-mounted display characterized in that the user guide is displayed when there is a user operation for displaying the user guide, or when the occurrence of a predetermined situation in a real space is detected regardless of the user operation. A head-mounted display that displays images generated by an information processing device.
A communication unit that sends and receives data to and from the information processing device,
The display image of the virtual world generated as a result of the information processing performed by the information processing device using the position information of the head-mounted display, the position in the real space of the user based on the position information, and the user should be. It is provided with a display unit that displays a user guide image that is set as a range and shows the relationship with a play area having a shape and size selected by the user.
A head-mounted display characterized in that the user guide is displayed when there is a user operation for displaying the user guide, or when the occurrence of a predetermined situation in a real space is detected regardless of the user operation.

Images