JP2021114310A - Information processor and warning presentation method - Google Patents

Abstract

To provide an information processor and a warning presentation method capable of presenting images free from restriction by an HMD (head mounted display) due to actual restriction conditions.SOLUTION: In an actual space including a user 120, the information processor sets a boundary of a play area 184 using boundaries 182a and 182b in a field of vision through an imaging apparatus 12 according to a predetermined rule. When the user gets out of the play area 184, a warning situation determination part determines that warning situation has occurred and superimpose a warning image over a display image. The information processor determines whether to set a margin area between the boundaries 182a and 182b in the field of vision and the boundary of the play area 184, or whether to change the width of the margin area depending on the direction and a situation in the actual space.SELECTED DRAWING: Figure 7

Description

The present invention relates to an information processing apparatus that performs information processing based on captured images and a warning presenting method that is performed by the information processing apparatus.

A head-mounted display (hereinafter referred to as “HMD”) connected to a game machine is attached to the head, and the game is played while looking at the displayed screen (see, for example, Patent Document 1). For example, if the position and posture of the user's head are acquired and the image of the virtual world is displayed so as to change the field of view according to the orientation of the face, it is possible to produce a situation as if the user has entered the virtual world. The position and posture of the user are generally acquired based on the analysis results of visible light or infrared images taken by the user, the measured values of the motion sensor built in the HMD, and the like.

Japanese Patent No. 5580855

Technology that performs some kind of information processing based on captured images is based on the premise that an object such as a user is within the angle of view of the camera. However, when the HMD is worn, the user cannot see the outside world, so that he / she may lose his / her sense of direction or move to an unexpected position in the real space so much that he / she is absorbed in the game. If the angle of view of the camera deviates from this, the information processing may be disrupted or the accuracy may be deteriorated, and the user may not be aware of the cause. The accuracy of information processing can be maintained so that the user is aware of the situation in the real world such as the positional relationship with the camera, but at the same time, a dilemma arises in which the world view of the virtual space constructed by using the HMD is easily obstructed.

The present invention has been made in view of these problems, and an object of the present invention is to provide a technique capable of optimizing the balance between entertainment and information processing accuracy realized by an HMD.

One aspect of the present invention relates to an information processing device. This information processing device has an information processing unit that detects the position of an object in real space and performs information processing based on the object, and an image generation unit that generates image data to be displayed as a result of the information processing. A warning status determination unit that determines the necessity of warning to the user based on a preset play area and superimposes a warning image on the image generation unit, and an image data generated by the image generation unit are output to the display device. The output unit includes an output unit, and the output unit outputs a confirmation screen prompting the user to confirm whether or not there is an obstacle to the display device, and after the user inputs that the confirmation has been made, the warning status determination unit Is characterized in that the determination process of the necessity of the warning is started.

Another aspect of the present invention also relates to an information processing device. This information processing device has an information processing unit that detects the position of an object in real space and performs information processing based on the object, and an image generation unit that generates image data to be displayed as a result of information processing. A warning status determination unit that determines the necessity of warning to the user based on a preset play area and superimposes a warning image on the image generation unit, and an image data generated by the image generation unit are output to the display device. The information processing unit is provided with an output unit, and the output unit outputs a confirmation screen prompting the user to confirm whether or not there is an obstacle to the display device. It is characterized in that the position detection process is started.

Yet another aspect of the present invention relates to a warning presentation method. This warning presentation method includes a step in which an information processing device detects the position of an object in real space and performs information processing based on the object, and a step of generating image data to be displayed as a result of the information processing. A step of determining the necessity of warning to the user based on a preset play area, a step of superimposing a warning image on an image to be displayed according to the determination, and outputting the generated image data to a display device. The step of including and the step of outputting outputs a confirmation screen to the display device prompting the user to confirm whether there is an obstacle, and the step of determining is after the user inputs that the confirmation has been made. It is characterized in that the determination process of the necessity of a warning is started.

Yet another aspect of the present invention also relates to a warning presentation method. This warning presentation method includes a step in which an information processing device detects the position of an object in real space and performs information processing based on the object, and a step of generating image data to be displayed as a result of the information processing. A step of determining the necessity of warning to the user based on a preset play area, a step of superimposing a warning image on an image to be displayed according to the determination, and outputting the generated image data to the display device. In the step of including and outputting, a confirmation screen prompting the user to confirm whether or not there is an obstacle is output to the display device, and in the step of performing information processing, the user inputs that the confirmation has been performed. After that, the position detection process is started.

It should be noted that any combination of the above components and the conversion of the expression of the present invention between a method, a device, a system, a computer program, a recording medium on which a computer program is recorded, and the like are also effective as aspects of the present invention. ..

According to the present invention, the user wearing the HMD can enjoy the expression world without being conscious of the constraint conditions in the real world as much as possible.

It is a figure which shows the configuration example of the information processing system to which this embodiment can be applied. It is a figure which shows the example of the appearance shape of the HMD in this embodiment. It is a figure which shows the internal circuit structure of the information processing apparatus in this embodiment. It is a figure which shows the internal circuit structure of the HMD in this embodiment. It is a figure which shows the structure of the functional block of the information processing apparatus in this embodiment. It is a figure for demonstrating the information which can be acquired from the photographed image in this embodiment. It is a figure which shows typically the setting example of the play area which is referred to when the warning situation determination part in this Embodiment determines the necessity of a warning. It is a figure which shows another example of the setting of the play area in this embodiment schematically. It is a figure for demonstrating the influence on the processing by the presence or absence of a margin area in this embodiment. It is a figure for demonstrating the setting method of the play area in the image plane in this embodiment. It is a figure which illustrates the display screen generated when it is determined that the warning is necessary in this embodiment. It is a flowchart which shows the processing procedure which the information processing apparatus generates the output data according to the movement of a user in this embodiment. It is a figure which shows typically the state transition of the display / non-display of a warning image in this embodiment.

FIG. 1 shows a configuration example of an information processing system to which this embodiment can be applied. The information processing system 8 includes an image pickup device 12 that captures an object, an information processing device 10 that performs information processing based on the captured image, a flat plate display 16 and an HMD 18 that display images obtained as a result of information processing, and a user. Includes an input device 14 to operate.

The information processing device 10, the image pickup device 12, the input device 14, the flat plate display 16, and the HMD 18 may be connected by a wired cable or by a known wireless communication technique such as Bluetooth (registered trademark). Further, depending on the information processing performed by the information processing apparatus 10, the flat plate display 16 may not be provided. Further, the external shape of these devices is not limited to those shown in the figure. Further, it may be a device in which two or more of these devices are integrally provided. For example, the information processing device 10, the input device 14, and the flat plate display 16 may be realized by a mobile terminal provided with them.

The image pickup device 12 generates output data of a captured image by performing a camera that shoots an object such as a user at a predetermined frame rate and general processing such as demosaic processing on the output signal, and causes the information processing device 10 to generate output data. It has a mechanism to send out. The camera is equipped with a visible light sensor used in general digital cameras and digital video cameras such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image pickup apparatus 12 may have only one camera, or may be a so-called stereo camera in which two cameras are arranged on the left and right at known intervals as shown in the figure.

Alternatively, the imaging device 12 may be configured by combining a monocular camera and a device that irradiates an object with reference light such as infrared rays and measures the reflected light. When a stereo camera or a reflected light measurement mechanism is introduced, the position of the object can be obtained in a three-dimensional real space, and the information processing by the information processing device 10 and the image display by the display device can be further diversified. can. Using stereo images taken by the stereo camera from the left and right viewpoints, the distance from the camera of the subject is specified by the principle of triangulation, and the TOF (Time of Flight) or pattern irradiation method is used to measure the reflected light of the subject. All methods for determining the distance from the camera are widely known.

Hereinafter, a mode in which the image pickup apparatus 12 captures a stereo image will be mainly described, but as described above, the present embodiment is not limited thereto, and the image pickup apparatus 12 may include at least one camera. .. The information processing device 10 performs necessary information processing using the data transmitted from the image pickup device 12, and generates output data such as an image and a sound. Here, the content of the processing performed by the information processing device 10 is not particularly limited, and may be appropriately determined depending on the function required by the user, the content of the application, and the like.

For example, the information processing device 10 performs general face detection and tracking processing on a captured image to advance a game in which a character reflecting the movement of the user, which is an object, appears, or commands the movement of the user. It converts to input and processes information. At this time, the movement of the input device 14 may be acquired by using the marker provided on the input device 14. Further, by tracking a plurality of markers provided on the outer surface of the HMD18, the position and posture of the head of the user wearing the HMD18 can be specified, and the virtual world seen from the viewpoint of moving correspondingly can be displayed on the HMD18. good. The output data generated by the information processing device 10 is transmitted to at least the HMD 18.

The HMD 18 is a display device that displays an image on a display panel such as an organic EL panel located in front of the user when the user wears it on his / her head. For example, the image may be viewed stereoscopically by generating a parallax image viewed from the left and right viewpoints and displaying the display screen in the left and right regions divided into two. However, the present embodiment is not limited to this, and one image may be displayed on the entire display screen. The HMD 18 may further include a speaker or earphone that outputs sound at a position corresponding to the user's ear.

The flat-panel display 16 may be a television having a display that outputs a two-dimensional image and a speaker that outputs audio, and is, for example, a liquid crystal television, an organic EL television, a plasma television, a PC display, or the like. Alternatively, it may be a display and a speaker of a tablet terminal or a mobile terminal. The input device 14 receives requests such as start and end of processing, selection of functions, and input of various commands by the user's operation, and supplies the information processing device 10 as an electric signal.

The input device 14 may be realized by any or a combination of general input devices such as a game controller, a keyboard, a mouse, a joystick, and a touch pad provided on the display screen of the flat plate display 16. The input device 14 may further include a light emitting marker composed of an element or a set thereof that emits light in a predetermined color. In this case, the information processing device 10 tracks the movement of the marker using the captured image, so that the movement of the input device 14 itself can be controlled by the user. The input device 14 may be composed of only a light emitting marker and a mechanism for gripping the light emitting marker.

FIG. 2 shows an example of the external shape of the HMD 18. In this example, the HMD 18 is composed of an output mechanism unit 102 and a mounting mechanism unit 104. The mounting mechanism unit 104 includes a mounting band 106 that goes around the head and realizes fixing of the device when the user wears it. The wearing band 106 is made of a material or structure whose length can be adjusted according to the head circumference of each user. For example, it may be an elastic body such as rubber, or a buckle, a gear, or the like may be used.

The output mechanism 102 includes a housing 108 having a shape that covers the left and right eyes when the HMD 18 is worn by the user, and includes a display panel inside so as to face the eyes when the HMD 18 is worn. Light emitting markers 110a, 110b, 110c, 110d, etc. are provided on the outer surface of the housing 108. The number and arrangement of the light emitting markers are not particularly limited, but in the present embodiment, they are provided at the four corners of the front surface of the housing of the output mechanism 102.

Further, light emitting markers 110e and 110f are also provided on both side surfaces behind the mounting band 106. By arranging the light emitting markers in this way, even if the user turns sideways or backwards with respect to the imaging device 12, the situation can be specified based on the number and position of the light emitting markers in the captured image. The light emitting markers 110c and 110d are on the lower side of the output mechanism 102, and the light emitting markers 110e and 110f are on the outside of the mounting band 106. Since they are not originally visible from the viewpoint of FIG. 2, the outer periphery is represented by a dotted line.

FIG. 3 shows the internal circuit configuration of the information processing device 10. The information processing device 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. Each of these parts is connected to each other via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 includes data to a peripheral device interface such as USB or IEEE1394, a communication unit 32 composed of a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive or a non-volatile memory, a flat plate display 16 or an HMD 18. The output unit 36 that outputs the data, the image pickup device 12, the input device 14, the input unit 38 that inputs data from the HMD 18, and the recording medium drive unit 40 that drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory are connected. ..

The CPU 22 controls the entire information processing apparatus 10 by executing the operating system stored in the storage unit 34. The CPU 22 also executes various programs read from the removable recording medium, loaded into the main memory 26, or downloaded via the communication unit 32. The GPU 24 has a geometry engine function and a rendering processor function, performs drawing processing according to a drawing command from the CPU 22, and stores a display image in a frame buffer (not shown). Then, the display image stored in the frame buffer is converted into a video signal and output to the output unit 36. The main memory 26 is composed of a RAM (Random Access Memory) and stores programs and data required for processing.

FIG. 4 shows the internal circuit configuration of the HMD 18. The HMD 18 includes a CPU 50, a main memory 52, a display unit 54, and an audio output unit 56. Each of these parts is connected to each other via a bus 58. An input / output interface 60 is further connected to the bus 58. A communication unit 62, an acceleration sensor 64, and a light emitting unit 66, which are network interfaces of a wired or wireless LAN, are connected to the input / output interface 60.

The CPU 50 processes the information acquired from each unit of the HMD 18 via the bus 58, and supplies the output data to the display unit 54 and the audio output unit 56. The main memory 52 stores programs and data required for processing in the CPU 50. However, depending on the design of the application or device to be executed, the information processing device 10 may perform almost all processing, and the HMD 18 may suffice to output the data transmitted from the information processing device 10. In this case, the CPU 50 and the main memory 52 can be replaced with a simpler device.

The display unit 54 is composed of a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of a user wearing the HMD 18. As described above, stereoscopic vision may be realized by displaying a pair of parallax images in the regions corresponding to the left and right eyes. The display unit 54 may further include a pair of lenses that are located between the display panel and the user's eyes when the HMD 18 is attached and that expand the user's viewing angle.

The voice output unit 56 is composed of speakers and earphones provided at positions corresponding to the user's ears when the HMD 18 is worn, and allows the user to hear the voice. The number of audio channels to be output is not particularly limited, and may be monaural, stereo, or surround. The communication unit 62 is an interface for transmitting and receiving data to and from the information processing device 10 and the flat plate display 16, and can be realized by using a known wireless communication technology such as Bluetooth (registered trademark).

The acceleration sensor 64 detects the inclination of the HMD 18 by measuring the gravitational acceleration in a predetermined axial direction. The HMD 18 may be further provided with various other sensors such as a gyro sensor. The measured value by the sensor is transmitted to the information processing device 10 via the communication unit 62. The light emitting unit 66 is an element or a set thereof that emits light in a predetermined color, and is provided at a plurality of locations on the outer surface of the HMD 18 as shown in FIG. In addition to acquiring the position of the HMD 18 by tracking this as a marker, the posture of the HMD 18 is acquired from the number and positional relationship of the images in the captured image.

The information processing device 10 can acquire the position and posture of the user's head with higher accuracy by integrating the information obtained by a plurality of means such as the acceleration sensor 64 and the light emitting unit 66. On the other hand, in the present embodiment, the acceleration sensor 64 may be omitted in some cases.

FIG. 5 shows the configuration of the functional block of the information processing device 10. Each functional block shown in FIG. 5 can be realized by the configuration of the CPU, GPU, various memories, data bus, etc. shown in FIG. 3 in terms of hardware, and is loaded into the memory from a recording medium or the like in terms of software. It is realized by a program that exerts various functions such as data input function, data retention function, image processing function, and communication function. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any of them.

The information processing device 10 performs information processing according to contents such as an input information acquisition unit 72 that acquires input information from the input device 14 and the HMD 18, a captured image acquisition unit 74 that acquires captured image data from the imaging device 12, and a game. It includes an information processing unit 76 that performs information processing, an output data generation unit 78 that generates data to be output, and a content data storage unit 84 that stores data necessary for information processing and image generation. The information processing device 10 further includes a position information acquisition unit 80 that acquires the user's position information based on the captured image, a warning status determination unit 82 that determines a situation in which a warning is required based on the user's position, and a real space required for determination. The area setting information storage unit 85 that stores the setting information related to the area of No. 1 and the output data transmission unit 86 that transmits the data to be output to the HMD 18 are included.

The input information acquisition unit 72 acquires the content of the user operation from the input device 14. Here, the user operation may be performed by a general information processing device such as selection of an application or content to be executed, start / end of processing, command input, and the like. The input information acquisition unit 72 supplies the information acquired from the input device 14 to the captured image acquisition unit 74 or the information processing unit 76 according to the content thereof. The input information acquisition unit 72 further receives the measured value from the acceleration sensor 64 of the HMD 18 and supplies it to the information processing unit 76.

The captured image acquisition unit 74 acquires data of a captured image, such as a stereo image, obtained by capturing a moving image by the imaging device 12 at a predetermined rate. The captured image acquisition unit 74 further controls the start / end of shooting in the imaging device 12 according to the processing start / end request from the user acquired by the input information acquisition unit 72, and responds to the processing result in the information processing unit 76. Alternatively, the type of data acquired from the image pickup apparatus 12 may be controlled.

The position information acquisition unit 80 acquires the user's position information at a predetermined rate by detecting an image of a predetermined object from the captured image. For example, the positions of the user's head and hands in the real space are acquired based on the images of the light emitting markers provided on the HMD 18 and the input device 14. Alternatively, an image analysis technique that tracks a part of the user's body using contour lines or recognizes a face or an object having a specific pattern by pattern matching may be combined. Further, depending on the configuration of the image pickup apparatus 12, the user's distance may be specified by measuring the reflection of infrared rays as described above.

The information processing unit 76 processes electronic contents such as games specified by the user. This process includes the use of the user's position information acquired by the position information acquisition unit 80. The information processing unit 76 may specify the posture of the user by integrating the measurement results of the acceleration sensor of the HMD 18 and the like, if necessary. As described above, the content of the information processing in the subsequent stage performed by the information processing unit 76 is not particularly limited according to the user operation via the input device 14 and the movement of the user.

The output data generation unit 78 generates image and audio data to be output as a result of information processing in accordance with the request from the information processing unit 76. For example, a virtual world viewed from a viewpoint corresponding to the position and posture of the user's head is generated as a left-right parallax image. By displaying this parallax image in front of the left and right eyes on the HMD 18 or outputting the sound in the virtual world, the user can feel as if he / she has entered the virtual world. The program and image / audio data required for information processing in the information processing unit 76 and data generation processing in the output data generation unit 78 are stored in the content data storage unit 84.

The warning status determination unit 82 constantly monitors the occurrence of a situation that gives a warning based on the user's location information acquired by the location information acquisition unit 80, and determines the occurrence. Specifically, the inside / outside determination is performed for the play area set directly or indirectly in the three-dimensional space of the real world, and it is determined that a warning is required when the user leaves the play area. In the present embodiment, since the position information is acquired using the captured image, the information processing progresses on the premise that the user is within the angle of view of the camera.

However, it is difficult for the user wearing the HMD as shown in FIG. 2 to recognize the surrounding situation and, by extension, whether or not he / she is within the angle of view. In order to continue normal information processing, it is necessary to provide such information related to the real world by some means, but it may impair the world view expressed by the content. As a measure for that, it is conceivable to give a warning to the user only when the angle of view is likely to come out. However, if this timing is too early, a warning will be given frequently with a small movement, and the movable range will be substantially narrowed. On the other hand, if the timing is late, the angle of view is inevitably deviated, and there may be a period during which position information cannot be obtained.

Therefore, in the present embodiment, as described above, the timing of generating the warning is determined by determining the inside and outside of the area set as the play area in the real world, and further, the relationship between the play area and the angle of view is adjusted to obtain the movable range. Optimize the balance of processing accuracy according to the situation. The play area setting information is stored in the area setting information storage unit 85 in advance. The specific setting contents will be described later. When the warning status determination unit 82 determines that a warning is necessary, the warning status determination unit 82 notifies the output data generation unit 78 to that effect.

At this time, the output data generation unit 78 additionally draws a component image representing a warning on the display image. The warning status determination unit 82 also determines that the warning is no longer necessary, such as when the user returns to a normal position, and notifies the output data generation unit 78 to that effect. In response to this, the output data generation unit 78 hides the component image representing the warning. The warning may be accompanied by sound as well as an image. The output data transmission unit 86 sequentially acquires the output data generated by the output data generation unit 78, shapes the output data as necessary, and transmits the output data to the HMD 18.

FIG. 6 is a diagram for explaining information that can be acquired from the captured image in the present embodiment. In the figure, the user 120 holds the input device 14 and wears the HMD 18. The input device 14 includes a light emitting marker 122 at a position facing the image pickup device 12 when gripped in a way suitable for operation. The emission marker of HMD18 is as shown in FIG. When the image pickup device 12 is a stereo camera, the distance Z from the image pickup surface of the image pickup device 12 to each light emitting marker is obtained based on the parallax of the image in the stereo image. Further, the position of the image of the light emitting marker on the image plane (XY plane) of either captured image represents the apparent position from the image pickup apparatus 12.

By integrating these information, specifically, by back-projecting the position in the XY plane using the distance Z from the image pickup apparatus 12, the position of each light emitting marker in the three-dimensional space in the real world can be obtained. Further, the attitude (vector va) of the HMD 18 in the real space is determined from the number and positional relationship of the light emitting marker images of the HMD 18, and the attitude (vector vb) of the input device 14 in the real space is determined from the shape of the image of the light emitting marker 122 of the input device 14. Desired.

The information processing unit 76 of the information processing device 10 has an image in a virtual world in which the field of view changes according to the orientation of the face of the user 120 or an image in response to the movement of the input device 14 based on the position and posture of these devices in the real space. It is possible to express how the objects inside move. The image pickup device 12 does not have to be a stereo camera when the distance from the image pickup device 12 is estimated from the apparent size of the marker or when information processing that does not require movement in the depth direction is performed. The same applies when introducing a distance measurement technique using reference light. Also, it is not always necessary to track both the HMD 18 and the input device 14.

When the image of the light emitting marker is used in this way, strictly speaking, the position of the light emitting marker of the HMD 18 or the input device 14 is directly obtained. However, since the HMD 18 is worn by the user on the head, it can be regarded as the position of the user's head. Strictly speaking, even when the position is specified based on a subject other than the light emitting marker, the position of the subject is obtained, and the position of the center of gravity of the user is estimated from the information as needed.

In the following description, the "user's position" may be the position of the user himself / herself or the position of the subject on which the position acquisition is based. Further, even if the input device 14 goes out of the play area, a warning may not be required if the HMD 18 does not come out, and rules related to the determination criteria may be provided according to the nature and situation of the subject.

FIG. 7 schematically shows a play area setting example that the warning status determination unit 82 refers to when determining the necessity of a warning. The figure shows a bird's-eye view of the real space, and the user 120 exists so as to face the image pickup apparatus 12. _{The horizontal angle of view θ h} of the camera of the image pickup apparatus 12 determines the boundary surfaces 182a and 182b of the field of view shown by the dotted lines. In the case of a stereo camera, it is the intersection of the angles of view of both. The boundary surface of the play area 184 is set according to a predetermined rule with reference to the boundary surface in the vertical direction of the visual field. When the user goes out of the play area 184, the warning status determination unit 82 determines that the need for a warning has occurred.

In the illustrated example, the play area 184 is set inside the boundary surfaces 182a and 182b of the visual field in the range where the distance Z from the image pickup apparatus 12 is Z1 ≦ Z <Z3. In the range of Z1 ≦ Z <Z2 close to the image pickup device 12, the width is increased so as to be proportional to the distance Z around the optical axis of the camera, and in the range of Z2 ≦ Z <Z3, it is set to have a constant width W. is doing. The vertical planes 186a and 186b that determine the boundary of the play area in the range of Z1 ≦ Z <Z2 correspond to positions formed inward by a predetermined amount from the left and right edges on the captured image regardless of the position in the depth direction. Therefore, by setting at least the inside of the vertical surfaces 186a and 186b as the play area, it is possible to give a warning at the timing when the user has a margin before deviating from the angle of view by using the predetermined amount as the margin area on the image.

The closer to the image pickup device 12, the narrower the field of view and therefore the narrower the movable range. Therefore, by making maximum use of the condition that the vertical planes 186a and 186b are inside the range close to the image pickup apparatus 12, the range in which the warning is not issued is widened. On the other hand, since the field of view is widened at a position away from the image pickup device 12, if the play area is set inside the vertical planes 186a and 186b based on whether or not the angle of view is deviated, a warning is not issued in a wider range. I live in. However, in the illustrated example, the play area is limited by a narrower width W in consideration of the suitability of information processing and the existence of surrounding objects.

As described above, the play area 184 can simultaneously have a role of setting an appropriate movable range in addition to reducing the possibility of deviation from the angle of view. In addition, by setting the play area according to different rules according to the distance range from the camera, the movable range is prioritized in the narrow field of view, and other factors such as the content situation are prioritized in the wide field of view. Can adapt to changes in.

FIG. 8 schematically shows another example of setting the play area. The figure shows a state in which the real space is viewed from the left side surface of the user, and the user 120 exists so as to face the image pickup apparatus 12. _{The vertical angle of view θ v} of the camera of the image pickup apparatus 12 determines the boundary surfaces 192a and 192b of the field of view shown by the dotted lines. With reference to the horizontal boundary surface of the field of view, the boundary surface of the play area is set in the range where the distance Z from the image pickup apparatus 12 is Z1 ≦ Z <Z3.

In the example of the figure, the boundary surfaces 192a and 192b of the visual field are aligned with the boundary surface of the play area 194 over the range of Z1 ≦ Z <Z3. That is, as compared with the play area shown in FIG. 7, a margin area is not set between the play area and the boundary surface of the visual field. That is, the play area is a region corresponding to the upper end to the lower end of the captured image without providing a predetermined amount of margin such as the horizontal planes 196a and 196b that are imaged inward from the upper and lower edges on the captured image. If this is done, a warning will be given at the same time as the angle of view deviates, so there is a high possibility that the position acquisition will not be possible even for a short time. On the other hand, it is not necessary to issue a warning in a wider range.

Such a setting is effective for a camera having a narrow angle of view. When the angle of view is narrow, if the margin area is set as shown in FIG. 7, a warning may be issued frequently with a small amount of movement. Therefore, comfort can be prioritized by securing a wide play area even if the accuracy of information processing is slightly deteriorated. For example, when using a camera having a wide horizontal angle of view and a narrow vertical angle of view, as shown in FIGS. 7 and 8, the boundary surface of the play area with respect to the boundary surface of the field of view depending on the direction in the real space. By making the positional relationship of the above different, it is possible to strike a good balance between information processing accuracy and a comfortable operating environment.

FIG. 9 is a diagram for explaining the influence of the presence or absence of the margin region on the processing. The upper part (a) of the figure shows the movement of the user and the time chart of each process when the play area is set by providing a margin area for the field of view, and the lower part (b) shows the movement of the user when the field of view and the play area are matched. be. That is, as schematically shown on the right side, a play area shown in white is set for the image pickup device 12, but in (a), a shaded margin area is set between the image pickup device 12 and the non-field-of-view area shown in black. It is provided. The white, shaded, and black rectangles in the "position" column of the time chart represent the play area, margin area, and time out of the field of view, respectively.

For comparison, in both settings, it is assumed that the user goes out of the field of view at time t2, and as a result of displaying the warning for the same time, the user returns to the play area. In response to such a user's movement, the period during which the position information acquisition unit 80 can acquire the position information is represented by a rectangle in the "position acquisition" column, and the period during which the warning is displayed is represented by the "warning" column. In the case of (a), since the player leaves the play area at time t1 before the time t2 when he / she goes out of the field of view, a warning is displayed from that time until he / she returns to the play area at time t3. On the other hand, in the case of (b), since the margin area is not set, a warning is displayed between the time t2 and the time t4 when the player is out of the field of view.

As shown in the figure, in (a), since the position information can be acquired even after the warning is displayed, it is guaranteed that the player is out of the play area based on the information. In the case of (b), since the location information cannot be acquired at the same time as the warning, its validity is not guaranteed during the period when the warning is displayed. Therefore, for example, if the image of the user is not detected due to occlusion at the boundary of the visual field, it may be mistaken as having gone out of the visual field and a warning may be displayed.

Further, even if the player returns to the play area at the same timing after the warnings are displayed in (a) and (b), the period in which the position information cannot be acquired in (b) becomes long. In the illustrated example, the period of going out of the field of view also occurs in (a), but depending on the width of the margin area, the user may return in response to the warning display before going out of the field of view. Compared to the case of b), the period during which the position information cannot be acquired is less likely to occur. On the other hand, even if the user moves in the same manner, in the case of (a), a warning is displayed at an earlier time t1, so that the constraint condition for the user becomes strict. Based on these characteristics, it is possible to set the optimum play area according to the priority according to the situation, such as whether you want to expand the area where you do not need to issue a warning to the limit, or whether there is room in the area without doing so. desirable.

Not only whether or not the margin area is provided, but also the width of the margin area and the shape of the play area may be optimized according to the assumed situation and the direction in the real space. For example, depending on the content of the content such as a game and the user's posture expected depending on the environment, specifically, whether to play standing or sitting, facing a desk or sitting on a sofa, even with the same angle of view. The shape of the play area may be changed.

Further, when the settings of FIGS. 7 and 8 are applied at the same time, the set play area has a shape having a quadrilateral cross section with respect to the distance Z from the camera. It may be any shape such as a cylindrical shape. If the optical axis is not in the horizontal direction, the distance Z from the camera will not be horizontal to the floor surface. At this time, the shape of the play area may be defined by the distance in the optical axis direction or by the distance in the horizontal direction in the real space. Coordinate transformation can be easily realized by measuring the angle formed by the optical axis of the camera and the horizontal plane in the real space with an acceleration sensor or the like built in the image pickup apparatus 12. It will be understood by those skilled in the art that various possible data formats for setting an area in a three-dimensional space can be considered.

On the other hand, the play area may be indirectly set by defining the area corresponding to the play area on the image plane. FIG. 10 is a diagram for explaining a method of setting a play area on an image plane. The figure shows the field of view of the camera facing the subject space, and thus the plane of the captured image. The upper row (a) is a monocular camera, and the lower row (b) is a stereo camera. In the case of a monocular camera, the left and right ends having a predetermined width x with respect to the plane of the captured image 300 are defined as margin areas, and the area 302 excluding the margin areas is defined as a play area corresponding area.

The region in the three-dimensional space defined by the region 302 is a region 304 in which the width in the horizontal direction increases in proportion to the distance Z in the depth direction. That is, when focusing on the horizontal direction, by setting the area 302, the inside of the vertical surfaces 186a and 186b shown in FIG. 8 is set as the play area. When the play area corresponding area 302 is set on the image plane in this way, the cross section of the play area in the real space has a similar figure of the size corresponding to the distance Z, so that it is compared with the case where the play area corresponding area 302 is set directly in the real space. Therefore, the degree of freedom of the set shape is reduced.

On the other hand, since the necessity of the warning can be directly determined by the inside / outside determination of the image on the captured image and the play area corresponding area 302, there is less room for error than through the position information of the user. The example shown is the case where the margin area is provided in the horizontal direction, but the same applies to the case where the margin area is provided in the vertical direction. Further, for the same reason as described above, whether or not the margin region is provided and the width x thereof may be adjusted independently depending on the direction. These elements may be changed according to the content and situation of the content.

In the case of the stereo camera shown in (b), the camera that first deviates from the field of view differs depending on whether the user is at the left end or the right end of the field of view of the camera. For example, when the user is near the right end of the field of view of the camera, as shown in the figure, the image 308a in the image 306a of the left viewpoint camera is closer to the right end than the image 308b in the image 306b of the camera of the right viewpoint. Therefore, if the user moves to the right as it is, the user will be out of the field of view of the camera from the left viewpoint first. On the contrary, when the user is near the left end of the field of view of the camera, the user goes out of the field of view of the camera in the right viewpoint first.

Therefore, when setting the play area within the field of view that can be captured by both stereo cameras, the area 310a excluding the margin area of the predetermined width x at the right end on the image plane captured by the camera of the left viewpoint and the image taken by the camera of the right viewpoint. The area 310b on the image plane excluding the margin area of the predetermined width x at the left end is simultaneously set as the play area corresponding area. Then, in either image, the necessity of the warning is determined when the image of the user comes out of the play area corresponding area.

FIG. 11 illustrates a display screen generated when the warning status determination unit 82 determines that a warning is necessary. The display screen 200 has a configuration in which a warning image 202 indicating a warning is added to an image of contents such as a game screen. In the illustrated example, the warning image 202 indicates that the user has gone out of the play area by the character information "Out of play area". Furthermore, the urgency is expressed by including a mark indicating danger.

However, the form of the warning image 202 is not limited to this, and it may be represented only by a figure or a mark by devising a color or a shape, or the character information may be combined with such a figure or a mark. Further, the warning image 202 may be changed for each content or according to the content of the display at that time. Due to the nature of giving notice to the user, the warning image 202 is basically superimposed and displayed on the image of the content. On the other hand, especially when the image of the content is a virtual world with a sense of depth, it is conceivable that the user may be surprised or uncomfortable by the sudden appearance of a component image without a sense of depth. Be done.

Therefore, as shown in the figure, the peripheral edge of the warning image 202 is blurred by blurring or α-blending. The entire warning image 202 may be translucent. This enhances the affinity with the virtual world that is expressed in the background while being in the foreground. Further, it is desirable to display the warning image 202 at a position away from the viewpoint so as not to give a feeling that the line of sight is suddenly blocked. For example, in a display mode in which the field of view is changed according to the user's line of sight, the viewpoint is inevitably fixed at the center of the screen, so that the display is performed at a position separated from the center of the screen by a predetermined distance or more.

In addition, since it is easier for a person to lower the line of sight than to raise the line of sight, the warning image 202 is displayed in the lower half area of the screen (the area below the center line C in the vertical direction) regardless of the display form. Can be easily moved from the image of the content. As a result, it is possible to prevent the suddenly appearing image from being out of focus and taking a long time to be recognized, or being intoxicated by fine eye movements.

Next, the operation of the information processing apparatus 10 that can be realized by the configuration described so far will be described. FIG. 12 is a flowchart showing a processing procedure in which the information processing apparatus generates output data according to the movement of the user in the present embodiment. This flowchart is started when the user requests the information processing device 10 to start processing via the input device 14.

First, the captured image acquisition unit 74 of the information processing device 10 requests the imaging device 12 to start photographing, and accordingly, starts acquiring the captured image data captured and output by the imaging device 12 (S10). On the other hand, the warning status determination unit 82 reads the setting information related to the play area from the area setting information storage unit 85 (S12). At this time, the setting information associated with the content to be executed, such as the user's selection, may be selected. Further, it is possible to specify whether or not the user is seated by analyzing the captured image, the surrounding situation, and the like, and select the setting information based on the information.

Next, the information processing unit 76 requests the output data generation unit 78 to output the real-time captured image acquired by the captured image acquisition unit 74 to the HMD 18. By displaying the captured image on the HMD 18, the user is urged to confirm that there are no obstacles in the surroundings (S14, S16). At this time, the output data generation unit 78 superimposes and displays text information such as "Are there any dangerous objects?" And a GUI (Graphical User Interface) for inputting that the user has confirmed. As a result, the user can recognize the angle of view of the camera in advance, so that the user can move with an area equivalent to the play area in mind.

By knowing the play area, your range of movement becomes clear, and you can also judge whether objects around you can be obstacles. If the user retracts the object as necessary and then performs the subsequent processing on the condition that the confirmation is input, the deviation from the camera's field of view can be monitored and the obstacle can be detected at the same time. Collision monitoring is also being carried out. While there is no confirmation input from the user, the captured image is continuously displayed (N, S14 in S16).

When the confirmation is input from the user (Y in S16), the position information acquisition unit 80 starts acquiring the user's position information based on the captured image (S18). Then, the information processing unit 76 performs information processing using the information, and the output data generation unit 78 draws a display image of the content as a result (S20). On the other hand, the warning status determination unit 82 monitors whether or not the user has left the play area (S22). Strictly speaking, the determination that the user has left the play area is performed when the marker provided on the HMD 18 worn by the user or the input device 14 held by the user leaves the play area.

However, the criteria for the determination may vary depending on the user's location information acquisition means. For example, when the depth map is created from the user's image itself in the stereo image to be used as the position information, the center of gravity of the user may be out of the play area. The same applies when tracking the contour line. Further, when tracking is performed by pattern matching such as face detection processing, it may be when the face is no longer recognized in the play area.

If the user is not out of the play area (N in S22), the output data transmission unit 86 outputs the image of the content drawn in S20 to the HMD 18 as it is (S26). When the user leaves the play area (Y in S22), the warning status determination unit 82 notifies the output data generation unit 78 to that effect, so that the output data generation unit 78 superimposes the warning image on the content image (Y). S24). Then, the output data transmission unit 86 outputs an image on which the warning image is superimposed to the HMD 18 (S26). In the process of S26, audio data may be output at the same time.

During the period when it is not necessary to end the process such as when the user requests to stop the process, the process of drawing the image of the content based on the position information, superimposing the warning image as necessary, and outputting it to the HMD 18 is performed at a predetermined rate. Repeat with (N of S28, S20 to S26). Then, when it becomes necessary to end the processing, all the processing is terminated (Y in S28).

In the flowchart of the figure, the period during which the user is outside the play area and the period during which the warning image is superimposed are the same. That is, the warning image is hidden when the user returns to the play area. On the other hand, the condition for hiding the warning image may be provided independently of the condition for displaying the warning image. FIG. 13 schematically shows a state transition in which a warning image is displayed / hidden. First, as described above, the criterion for changing from the state where the warning image is not displayed to the state where the warning image is displayed is that the user has left the play area (arrow to the right).

On the other hand, the criterion for changing from the state in which the warning image is displayed to the state in which the warning image is not displayed is not that the player has entered the play area but that the condition that has been set separately is satisfied (arrow to the left). For example, a warning non-display area is set as a range narrower than the play area, and the warning image is hidden when entering the area. Alternatively, it is assumed that a predetermined time has passed since entering the play area. By doing so, the warning once displayed is less likely to be hidden. As a result, when the user is near the boundary of the play area, the inside / outside determination result for the play area is switched with a slight movement, and chattering in which the warning is displayed and disappears can be prevented.

According to the present embodiment described above, in a system that acquires user's position information based on an image in a captured image and processes information using the result, a play area is set in a three-dimensional space in the real world. , When the user leaves the play area, a warning to that effect is displayed to the user. As a result, it is possible to prevent the user from wearing the HMD and not being able to see the outside world or being immersed in the displayed virtual world, which unknowingly goes out of the field of view of the camera and causes a failure in information processing.

Here, the play area is set with reference to the field of view of the camera. At this time, the elements such as whether the play area is set inside the field of view in anticipation of the margin area, the same as the field of view, and the width of the margin area when the margin area is expected are optimized. Adjusting the margin area in this way defines the range in which the user can move without warning, and at the same time, adjusts the frequency with which the user goes out of the field of view and cannot acquire the position information, and by extension, the accuracy of the position information acquisition. It is nothing but being. Therefore, it is possible to make settings that match the priority order, such as whether to allow movement in a wide range as much as possible at the expense of some deterioration in position information acquisition accuracy, or to maintain position information acquisition accuracy even if the range is limited. Become.

Since such priority changes depending on the horizontal / vertical angle of view of the camera, the content of information processing, the user and the surrounding conditions, etc., the position information can be acquired by optimizing the settings for each. It is possible to achieve both accuracy and user comfort. In addition, the shape of the play area is not particularly limited, so if the shape and size are set appropriately, in addition to preventing deviation from the field of view of the camera, observing the appropriate value of the amount of movement expected from content such as games and surroundings It can play a role at the same time, such as avoiding collisions with obstacles.

When displaying a warning, a component image representing the warning is superimposed and displayed on the original display image such as a game screen. At this time, by blurring the periphery or making it translucent, the affinity with the original display image is enhanced, and the discomfort caused by the sudden appearance of an unrelated image is reduced. In addition, by setting a predetermined distance from the viewpoint or displaying it at the position of the lower half of the screen, the occurrence of discomfort and sickness due to the line of sight being blocked is suppressed, and the line of sight is aligned without burden to recognize the situation. be able to.

The present invention has been described above based on the embodiments. The above-described embodiment is an example, and it is understood by those skilled in the art that various modifications are possible for each of these components and combinations of each processing process, and that such modifications are also within the scope of the present invention. be.

8 Information processing system, 10 Information processing device, 12 Imaging device, 14 Input device, 16 Flat plate display, 18 HMD, 22 CPU, 24 GPU, 26 Main memory, 72 Input information acquisition unit, 74 Captured image acquisition unit, 76 Information Processing unit, 78 output data generation unit, 80 position information acquisition unit, 82 warning status determination unit, 85 area setting information storage unit, 86 output data transmission unit.

Claims (16)

An information processing unit that detects the position of an object in real space and processes information based on the object.
An image generation unit that generates image data to be displayed as a result of the information processing,
A warning status determination unit that determines the necessity of warning to the user based on a preset play area and superimposes a warning image on the image generation unit.
An output unit that outputs image data generated by the image generation unit to a display device, and an output unit.
With
The output unit outputs a confirmation screen prompting the user to confirm that there are no obstacles to the display device.
An information processing device characterized in that the warning status determination unit starts a determination process of the necessity of the warning after the user inputs that the confirmation has been made. An information processing unit that detects the position of an object in real space and processes information based on the object.
An image generation unit that generates image data to be displayed as a result of the information processing,
A warning status determination unit that determines the necessity of warning to the user based on a preset play area and superimposes a warning image on the image generation unit.
An output unit that outputs image data generated by the image generation unit to a display device, and an output unit.
With
The output unit outputs a confirmation screen prompting the user to confirm that there are no obstacles to the display device.
An information processing device characterized in that the information processing unit starts a position detection process after an input to the effect that the confirmation has been made is made by the user. The information processing device according to claim 1 or 2, wherein the warning status determination unit determines that a warning to the user is necessary when the object comes out of the play area. The information processing device according to any one of claims 1 to 3, wherein the output unit includes a real-time captured image in the confirmation screen. The information processing apparatus according to claim 2, wherein the warning status determination unit starts a determination process for determining the necessity of the warning after the user inputs that the confirmation has been made. A claim, wherein the warning status determination unit acquires a distance of the object in the optical axis direction based on a stereo image taken by a stereo camera constituting an imaging device for detecting the object. The information processing apparatus according to any one of 1 to 5. The information processing apparatus according to claim 6, wherein the direction of the optical axis of the stereo camera is not in the horizontal direction of the real space. When the optical axis of the imaging device is not in the horizontal direction of the real space, the warning status determination unit defines the shape of the play area with respect to the distance in the optical axis direction, thereby causing the play area and the object. The information processing apparatus according to claim 6 or 7, wherein the positional relationship of the above is acquired. The warning status determination unit defines the shape of the play area with respect to the horizontal distance in the real space, and when the optical axis of the image pickup device is not in the horizontal direction, the angle formed by the optical axis and the horizontal direction. The information processing apparatus according to claim 6 or 7, wherein the positional relationship between the play area and the object is acquired by coordinate conversion based on the measured values of the above. The warning status determination unit defines a region corresponding to the play area in the image plane captured by the imaging device for detecting the object, and based on the position of the image of the object on the captured image. The information processing apparatus according to any one of claims 1 to 5, wherein the necessity of warning is determined. The warning status determination unit sets the area corresponding to the space captured by both of the stereo cameras constituting the image pickup apparatus to each of the stereo images captured by the stereo camera, whereby the target in either image. The information processing apparatus according to claim 10, wherein it is determined that the warning is necessary when an image of an object comes out of the setting area. The claim situation determination unit sets the boundary surface of the play area with reference to the boundary surface of the product set of the angles of view of the stereo camera constituting the imaging device for detecting the object. Item 4. The information processing apparatus according to any one of Items 1 to 8. Steps to detect the position of an object in real space and perform information processing based on the object,
The step of generating image data to be displayed as a result of the information processing, and
Steps to determine the need for warning to the user based on a preset play area,
A step of superimposing a warning image on the image to be displayed according to the determination,
Steps to output the generated image data to the display device,
Including
In the output step, a confirmation screen prompting the user to confirm whether there is an obstacle is output to the display device.
The determination step is a method of presenting a warning by an information processing device, which comprises starting a determination process of the necessity of the warning after the user inputs that the confirmation has been made. Steps to detect the position of an object in real space and perform information processing based on the object,
The step of generating image data to be displayed as a result of the information processing, and
Steps to determine the need for warning to the user based on a preset play area,
A step of superimposing a warning image on the image to be displayed according to the determination,
Steps to output the generated image data to the display device,
Including
In the output step, a confirmation screen prompting the user to confirm whether there is an obstacle is output to the display device.
The step of performing information processing is a method of presenting a warning by an information processing apparatus, which comprises starting a detection process of the position after an input to the effect that the confirmation has been made is made by the user. A function that detects the position of an object in real space and processes information based on the object,
A function to generate image data to be displayed as a result of the information processing, and
A function of determining the necessity of a warning to the user based on a preset play area and notifying the function of generating the image data to that effect, and a function of superimposing the warning image.
A function to output the generated image data to the display device,
To the computer,
The output function outputs a confirmation screen prompting the user to confirm that there are no obstacles to the display device.
The function of superimposing the warning image is a computer program characterized in that the process of determining the necessity of the warning is started after the user inputs that the confirmation has been made. A function that detects the position of an object in real space and processes information based on the object,
A function to generate image data to be displayed as a result of the information processing, and
A function of determining the necessity of a warning to the user based on a preset play area and notifying the function of generating the image data to that effect, and a function of superimposing the warning image.
A function to output the generated image data to the display device,
To the computer,
The output function outputs a confirmation screen prompting the user to confirm that there are no obstacles to the display device.
The function of performing information processing is a computer program characterized in that the position detection process is started after the user inputs that the confirmation has been made.

Images