JP2022111143A - Information processor and warning presentation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor and a warning presentation method capable of presenting images free from restriction by an HMD (head mounted display) due to actual restriction conditions.

SOLUTION: In an actual space including a user 120, the information processor sets a boundary of a play area 184 using boundaries 182a and 182b in a field of vision through an imaging apparatus 12 according to a predetermined rule. When the user gets out of the play area 184, a warning situation determination part determines that warning situation has occurred and superimpose a warning image over a display image. The information processor determines whether to set a margin area between the boundaries 182a and 182b in the field of vision and the boundary of the play area 184, or whether to change the width of the margin area depending on the direction and a situation in the actual space.

SELECTED DRAWING: Figure 7

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to an information processing device that performs information processing based on a captured image and a warning presentation method performed by the device.

2. Description of the Related Art A head-mounted display (hereafter referred to as "HMD") connected to a game machine is worn on the head, and a game is played while looking at the displayed screen (see Patent Document 1, for example). For example, by acquiring the position and posture of the user's head and displaying an image of the virtual world so as to change the field of view according to the orientation of the user's face, it is possible to produce a situation as if the user had entered the virtual world. The position and posture of the user are generally obtained based on the analysis results of visible light and infrared images of the user, the measurement values of the motion sensor incorporated in the HMD, and the like.

Patent No. 5580855

A technique for performing some kind of information processing based on a captured image assumes that an object such as a user is within the angle of view of the camera. However, since the user cannot see the outside world while wearing the HMD, the user may lose a sense of direction or move to an unexpected position in the real space due to being absorbed in the game. As a result, if the angle of view is out of the camera's field of view, the information processing may fail or the accuracy may deteriorate, and the user himself/herself may be unaware of the cause. The more the user is aware of the real world situation such as the positional relationship with the camera, the more accurate information processing can be maintained, but at the same time, a dilemma arises that the world view of the virtual space constructed using the HMD is likely to be disturbed.

The present invention has been made in view of these problems, and its object is to provide a technique capable of optimizing the balance between entertainment and information processing accuracy realized by an HMD.

One aspect of the present invention relates to an information processing apparatus. This information processing device includes an information processing unit that detects the position of an object in real space and performs information processing based on the object, an image generation unit that generates image data to be displayed as a result of the information processing, A warning situation determination unit that determines the necessity of a warning to the user based on the play area whose shape is changed according to the situation, superimposes a warning image on the image generation unit, and an image data generated by the image generation unit. and an output unit for outputting to a display device.

Yet another aspect of the present invention relates to a warning presenting method. This warning presenting method comprises steps of an information processing device detecting the position of an object in a real space and performing information processing based on the object; generating image data to be displayed as a result of the information processing; A step of determining the necessity of warning the user based on the play area whose shape is changed according to the situation, a step of superimposing the warning image on the image to be displayed according to the determination, and data of the generated image. and outputting to a display device.

Any combination of the above constituent elements, and any conversion of the expression of the present invention between a method, an apparatus, a system, a computer program, a recording medium recording a computer program, and the like are also effective as aspects of the present invention. .

According to the present invention, a user wearing an HMD can enjoy the world of expression without being conscious of constraints in the real world as much as possible.

It is a figure which shows the structural example of the information processing system which can apply this Embodiment. It is a figure which shows the example of the external appearance shape of HMD in this Embodiment. 1 is a diagram showing an internal circuit configuration of an information processing device according to this embodiment; FIG. It is a figure which shows the internal circuit structure of HMD in this Embodiment. 1 is a diagram showing a configuration of functional blocks of an information processing device according to an embodiment; FIG. FIG. 4 is a diagram for explaining information that can be acquired from a captured image in the present embodiment; FIG. FIG. 4 is a diagram schematically showing a setting example of a play area referred to by a warning situation determination unit in the present embodiment when determining necessity of warning. FIG. 10 is a diagram schematically showing another example of play area setting in the present embodiment; FIG. 10 is a diagram for explaining the influence of presence/absence of a margin area on processing in the present embodiment; FIG. 4 is a diagram for explaining a method of setting a play area on an image plane in this embodiment; In the present embodiment, it is a diagram illustrating a display screen generated when it is determined that a warning is necessary. 4 is a flow chart showing a processing procedure for generating output data according to a user's movement by the information processing device in the present embodiment. FIG. 4 is a diagram schematically showing state transition between display/non-display of a warning image according to the present embodiment;

FIG. 1 shows a configuration example of an information processing system to which this embodiment can be applied. The information processing system 8 includes an imaging device 12 that captures an image of an object, an information processing device 10 that performs information processing based on the captured image, a flat panel display 16 and an HMD 18 that display an image obtained as a result of the information processing, and a user. It includes an input device 14 to operate.

The information processing device 10, the imaging device 12, the input device 14, the flat panel display 16, and the HMD 18 may be connected by a wired cable or by a known wireless communication technology such as Bluetooth (registered trademark). Further, depending on the information processing performed by the information processing apparatus 10, the flat display 16 may be omitted. Further, the external shape of these devices is not limited to that shown in the drawings. Further, the device may be a device integrally including two or more of these devices. For example, the information processing device 10, the input device 14, and the flat-panel display 16 may be realized by a mobile terminal or the like having them.

The imaging device 12 generates output data of a captured image by performing general processing such as demosaicing on the output signal of a camera that captures an object such as a user at a predetermined frame rate. and a delivery mechanism. The camera includes a visible light sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, which is used in general digital cameras and digital video cameras. The imaging device 12 may have only one camera, or may be a so-called stereo camera in which two cameras are arranged left and right at a known interval as illustrated.

Alternatively, the imaging device 12 may be configured by a combination of a monocular camera and a device that irradiates an object with reference light such as infrared rays and measures the reflected light. When a stereo camera or a mechanism for measuring reflected light is introduced, the position of an object can be obtained in a three-dimensional real space, and information processing by the information processing device 10 and image display by the display device can be diversified. can. Using stereo images taken from left and right viewpoints by a stereo camera, the distance from the subject to the camera is specified by the principle of triangulation. All techniques for determining the distance from the camera are well known.

Hereinafter, a mode in which the imaging device 12 captures a stereo image will be mainly described. . The information processing device 10 performs necessary information processing using the data transmitted from the imaging device 12 and generates output data such as images and sounds. Here, the content of the processing performed by the information processing apparatus 10 is not particularly limited, and may be appropriately determined according to the functions requested by the user, the content of the application, and the like.

For example, the information processing apparatus 10 performs general face detection and tracking processing on a captured image to advance a game in which a character reflecting the movement of a user, who is an object, appears, or to command the movement of the user. It converts it into an input and performs information processing. At this time, the movement of the input device 14 may be acquired using a marker provided on the input device 14 . Further, by tracking a plurality of markers provided on the outer surface of the HMD 18, the position and posture of the user's head wearing the HMD 18 can be identified, and the virtual world viewed from a viewpoint that moves correspondingly can be displayed on the HMD 18. good. The output data generated by the information processing device 10 is transmitted to at least the HMD 18 .

The HMD 18 is a display device that displays an image on a display panel, such as an organic EL panel, positioned in front of the user's head when worn by the user. For example, an image may be stereoscopically viewed by generating parallax images viewed from left and right viewpoints and displaying them in left and right regions obtained by dividing the display screen into two. However, the present embodiment is not intended to be limited to this, and one image may be displayed on the entire display screen. The HMD 18 may further incorporate speakers or earphones that output sound at positions corresponding to the ears of the user.

The flat-panel display 16 may be a television having a display for outputting two-dimensional images and speakers for outputting audio, such as a liquid crystal television, an organic EL television, a plasma television, or a PC display. Alternatively, it may be the display and speaker of a tablet terminal or mobile terminal. The input device 14 accepts requests such as the start and end of processing, selection of functions, input of various commands, etc., and supplies them to the information processing apparatus 10 as electrical signals.

The input device 14 may be implemented by any of general input devices such as a game controller, keyboard, mouse, joystick, touch pad provided on the display screen of the flat panel display 16, or a combination thereof. The input device 14 may further include luminous markers, which are elements or collections of elements that emit light in a predetermined color. In this case, the information processing apparatus 10 can track the movement of the marker using the captured image, so that the movement of the input device 14 itself can be the user's operation. It should be noted that the input device 14 may be composed only of a luminous marker and a mechanism for gripping it.

FIG. 2 shows an example of the external shape of the HMD 18. As shown in FIG. In this example, the HMD 18 is composed of an output mechanism section 102 and a mounting mechanism section 104 . The mounting mechanism section 104 includes a mounting band 106 that is worn by the user so as to go around the head and fix the device. The mounting band 106 is made of a material or structure whose length can be adjusted according to the head circumference of each user. For example, an elastic body such as rubber may be used, or a buckle or a gear may be used.

The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the HMD 18, and has a display panel inside so as to face the eyes when the HMD 18 is worn. Light-emitting markers 110 a, 110 b, 110 c, 110 d, etc. are provided on the outer surface of the housing 108 . Although the number and arrangement of the luminous markers are not particularly limited, they are provided at the four corners of the front surface of the housing of the output mechanism section 102 in the present embodiment.

Further, luminous markers 110e and 110f are also provided on both side surfaces behind the mounting band 106. As shown in FIG. By arranging the light-emitting markers in this way, even if the user turns sideways or turns back to the imaging device 12, it is possible to identify the situation based on the number and positions of the light-emitting marker images in the captured image. The luminous markers 110c and 110d are located below the output mechanism section 102, and the luminous markers 110e and 110f are located outside the mounting band 106. Since they are not visible from the viewpoint of FIG.

FIG. 3 shows the internal circuit configuration of the information processing apparatus 10. As shown in FIG. The information processing device 10 includes a CPU (Central Processing Unit) 22 , a GPU (Graphics Processing Unit) 24 and a main memory 26 . These units are interconnected via a bus 30 . An input/output interface 28 is also connected to the bus 30 . The input/output interface 28 includes a peripheral device interface such as USB and IEEE 1394, a communication unit 32 including a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive and a nonvolatile memory, and data to the flat panel display 16 and HMD 18. , an input unit 38 for inputting data from the imaging device 12, the input device 14, and the HMD 18, and a recording medium driving unit 40 for driving a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory. .

The CPU 22 controls the entire information processing apparatus 10 by executing an operating system stored in the storage section 34 . The CPU 22 also executes various programs read from a removable recording medium and loaded into the main memory 26 or downloaded via the communication section 32 . The GPU 24 has a function of a geometry engine and a function of a rendering processor, performs drawing processing according to a drawing command from the CPU 22, and stores a display image in a frame buffer (not shown). Then, the display image stored in the frame buffer is converted into a video signal and output to the output section 36 . The main memory 26 is composed of a RAM (Random Access Memory) and stores programs and data necessary for processing.

FIG. 4 shows the internal circuit configuration of the HMD 18. As shown in FIG. The HMD 18 includes a CPU 50 , a main memory 52 , a display section 54 and an audio output section 56 . These units are interconnected via a bus 58 . An input/output interface 60 is also connected to the bus 58 . Connected to the input/output interface 60 are a communication unit 62 , an acceleration sensor 64 , and a light emitting unit 66 , each of which is a wired or wireless LAN network interface.

The CPU 50 processes information acquired from each section of the HMD 18 via the bus 58 and supplies output data to the display section 54 and the audio output section 56 . A main memory 52 stores programs and data necessary for processing in the CPU 50 . However, depending on the application to be executed or the design of the device, it may be sufficient for the information processing device 10 to perform almost all the processing and for the HMD 18 to simply output the data transmitted from the information processing device 10 . In this case, the CPU 50 and main memory 52 can be replaced with simpler devices.

The display unit 54 is composed of a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the user wearing the HMD 18 . Stereoscopic vision may be realized by displaying a pair of parallax images in regions corresponding to the left and right eyes as described above. The display unit 54 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the HMD 18 is worn, and that expand the user's viewing angle.

The audio output unit 56 includes speakers and earphones provided at positions corresponding to the ears of the user when the HMD 18 is worn, and allows the user to hear audio. The number of audio channels to be output is not particularly limited, and may be monaural, stereo, or surround. The communication unit 62 is an interface for transmitting and receiving data between the information processing device 10 and the flat panel display 16, and can be realized using a known wireless communication technology such as Bluetooth (registered trademark).

The acceleration sensor 64 detects the tilt of the HMD 18 by measuring gravitational acceleration in a predetermined axial direction. The HMD 18 may be further provided with various other sensors such as a gyro sensor. The sensor-measured values are transmitted to the information processing device 10 via the communication unit 62 . The light emitting units 66 are elements or groups thereof that emit light in a predetermined color, and are provided at a plurality of locations on the outer surface of the HMD 18 as shown in FIG. By tracking this as a marker, the position of the HMD 18 is obtained, and the orientation of the HMD 18 is obtained from the number of images in the captured image and the positional relationship.

The information processing apparatus 10 can acquire the position and posture of the user's head with higher accuracy by integrating information obtained by a plurality of means such as the acceleration sensor 64 and the light emitting unit 66 . On the other hand, in this embodiment, the acceleration sensor 64 can be omitted in some cases.

FIG. 5 shows the configuration of functional blocks of the information processing apparatus 10. As shown in FIG. Each functional block shown in FIG. 5 can be realized by the configuration of CPU, GPU, various memories, data buses, etc. shown in FIG. It is realized by a program that exhibits various functions such as a data input function, a data holding function, an image processing function, and a communication function. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and are not limited to either one.

The information processing apparatus 10 includes an input information acquisition unit 72 that acquires input information from the input device 14 and the HMD 18, a captured image acquisition unit 74 that acquires captured image data from the imaging device 12, and information processing according to content such as games. an information processing section 76 for processing, an output data generation section 78 for generating data to be output, and a content data storage section 84 for storing data necessary for information processing and image generation. The information processing apparatus 10 further includes a position information acquisition unit 80 that acquires user position information based on the captured image, a warning situation determination unit 82 that determines a situation requiring a warning based on the user position, and a real space and an output data transmission unit 86 for transmitting data to be output to the HMD 18 .

The input information acquisition unit 72 acquires the content of the user's operation from the input device 14 . Here, the user operation may be performed by a general information processing apparatus, such as selection of an application or content to be executed, start/end of processing, command input, and the like. The input information acquisition section 72 supplies the information acquired from the input device 14 to the captured image acquisition section 74 or the information processing section 76 depending on the content of the information. The input information acquisition unit 72 further receives the measured value from the acceleration sensor 64 of the HMD 18 and supplies it to the information processing unit 76 .

The captured image acquisition unit 74 acquires data of captured images such as stereo images obtained by capturing moving images by the imaging device 12 at a predetermined rate. The captured image acquisition unit 74 further controls the start/end of shooting in the imaging device 12 according to the processing start/end request from the user acquired by the input information acquisition unit 72, may be used to control the type of data acquired from the imaging device 12 .

The positional information acquiring unit 80 acquires user's positional information at a predetermined rate by detecting an image of a predetermined object from the captured image. For example, the positions of the user's head and hands in real space are acquired based on images of luminous markers provided on the HMD 18 or the input device 14 . Alternatively, it may be combined with an image analysis technique that tracks a part of the user's body using contour lines, or recognizes a face or an object with a specific pattern by pattern matching. Further, depending on the configuration of the imaging device 12, as described above, the user's distance may be specified by measuring the reflection of infrared rays.

The information processing unit 76 processes electronic content such as games specified by the user. This processing includes the use of the user's location information acquired by the location information acquisition unit 80 . The information processing unit 76 may specify the user's posture by integrating measurement results obtained by the acceleration sensor of the HMD 18 as necessary. Note that, as described above, the content of subsequent information processing performed by the information processing unit 76 in accordance with user operations and user movements via the input device 14 is not particularly limited.

The output data generation unit 78 generates image and audio data to be output as a result of information processing according to a request from the information processing unit 76 . For example, a virtual world viewed from a viewpoint corresponding to the position and posture of the user's head is generated as left and right parallax images. By displaying these parallax images in front of the left and right eyes on the HMD 18 and by outputting audio in the virtual world, the user can feel as if he or she has entered the virtual world. Programs and image/audio data necessary for information processing in the information processing section 76 and data generation processing in the output data generation section 78 are stored in the content data storage section 84 .

Based on the user's location information acquired by the location information acquisition unit 80, the warning situation determination unit 82 constantly monitors the occurrence of a situation in which a warning is issued, and determines the occurrence. Specifically, inside/outside determination is performed with respect to the play area set directly or indirectly in the three-dimensional space of the real world, and it is determined that a warning is necessary when the user leaves the play area. In the present embodiment, since position information is acquired using captured images, information processing proceeds on the assumption that the user is within the angle of view of the camera.

However, it is difficult for a user wearing an HMD such as that shown in FIG. 2 to recognize the surroundings and whether he or she is within the angle of view. In order to continue normal information processing, it is necessary to provide such information related to the real world by some means, but this may spoil the view of the world expressed by the content. As a measure for this, it is conceivable to give a warning to the user only when it is about to go out of the angle of view. However, if this timing is too early, small movements will give frequent warnings, effectively reducing the range of motion. On the other hand, if the timing is too late, the angle of view is inevitably deviated, and a period during which position information cannot be obtained may occur.

Therefore, in the present embodiment, as described above, the timing for issuing a warning is determined by determining the inside/outside of the area set as the play area in the real world, and furthermore, by adjusting the relationship between the play area and the angle of view, the movable range and the Optimize the balance of processing accuracy depending on the situation. Play area setting information is stored in the area setting information storage unit 85 in advance. Specific setting contents will be described later. When the warning status determination unit 82 determines that a warning is necessary, it notifies the output data generation unit 78 of that effect.

At this time, the output data generation unit 78 additionally draws a component image representing the warning on the display image. The warning status determination unit 82 also determines that the user has returned to a normal position and the warning is no longer necessary, and notifies the output data generation unit 78 to that effect. In response to this, the output data generator 78 hides the component image representing the warning. Note that the warning may be accompanied by sound as well as an image. The output data transmission unit 86 sequentially acquires the output data generated by the output data generation unit 78 , shapes the data as necessary, and transmits the data to the HMD 18 .

FIG. 6 is a diagram for explaining information that can be acquired from a captured image in this embodiment. In the figure, the user 120 is holding the input device 14 and wearing the HMD 18 . The input device 14 has a luminous marker 122 at a position directly facing the imaging device 12 when held in a manner suitable for operation. The luminescence marker of HMD18 is as shown in FIG. When the imaging device 12 is a stereo camera, the distance Z from the imaging plane of the imaging device 12 to each luminous marker is obtained based on the parallax of the images in the stereo image. Also, the position of the image of the luminescence marker on the image plane (XY plane) of either captured image represents the apparent position from the imaging device 12 .

The position of each luminous marker in the three-dimensional space of the real world is obtained by integrating these pieces of information, specifically by inverse projecting the position on the XY plane using the distance Z from the imaging device 12 . Further, the orientation (vector va) of the HMD 18 in the real space is determined from the number and positional relationship of the images of the luminous markers of the HMD 18, and the orientation (vector vb) of the input device 14 in the real space is determined from the shape of the images of the luminous markers 122 of the input device 14. Desired.

The information processing unit 76 of the information processing device 10 creates a virtual world in which the field of view changes according to the direction of the face of the user 120 and an image according to the movement of the input device 14 based on the positions and postures of these devices in the real space. It is possible to express how the objects inside move. When estimating the distance from the imaging device 12 based on the apparent size of the marker or when performing information processing that does not require movement in the depth direction, the imaging device 12 need not be a stereo camera. The same is true when introducing a distance measurement technique using reference light. Also, it is not necessary to track both the HMD 18 and the input device 14 .

Strictly speaking, when using the image of the luminous marker, the position of the luminous marker of the HMD 18 or the input device 14 is directly obtained. However, since the HMD 18 is worn on the user's head, it can also be regarded as the position of the user's head. Strictly speaking, even when the position is specified based on subjects other than the light-emitting markers, the positions of these subjects are obtained, and the position of the user's center of gravity and the like are estimated as necessary from the information.

In the following description, the "user's position" may be the position of the user himself or the position of the subject that serves as a basis for position acquisition. Further, even if the input device 14 goes out of the play area, if the HMD 18 is not displayed, there is no need to issue a warning.

FIG. 7 schematically shows a setting example of a play area referred to by the warning status determination unit 82 when determining the necessity of warning. The figure shows a bird's-eye view of the real space, and the user 120 is present facing the imaging device 12 . Boundary planes 182a and 182b of the field of view indicated by dotted lines are determined by the horizontal angle of view θ _h of the camera of the imaging device 12 . In the case of a stereo camera, it is the product set of both angles of view. The boundary plane of the play area 184 is set according to a predetermined rule based on such a boundary plane in the vertical direction of the field of view. When the user goes out of the play area 184, the warning situation determination unit 82 determines that a warning is necessary.

In the illustrated example, the play area 184 is set inside the boundary planes 182a and 182b of the field of view when the distance Z from the imaging device 12 is in the range of Z1≦Z<Z3. In the range of Z1≤Z<Z2 close to the imaging device 12, the width is increased in proportion to the distance Z centered on the optical axis of the camera, and in the range of Z2≤Z<Z3, the width is set to have a constant width W. is doing. The vertical planes 186a and 186b that determine the boundaries of the play area in the range of Z1≦Z<Z2 correspond to positions that are imaged inward from the left and right ends of the captured image by a predetermined amount regardless of the position in the depth direction. Therefore, by setting at least the inner side of the vertical surfaces 186a and 186b as the play area, it is possible to give a warning at the timing when the user sees a margin before deviating from the angle of view by using the predetermined amount as a margin area on the image.

The closer to the imaging device 12, the narrower the field of view and the narrower the movable range. Therefore, by making maximum use of the condition that the range is inside the vertical planes 186a and 186b in the range close to the imaging device 12, the range in which no warning is issued is widened. On the other hand, since the field of view is widened at a position away from the imaging device 12, if the play area is set to the inner side of the vertical planes 186a and 186b, the warning will not be issued in a wider range if the play area is set based on whether or not it is out of the angle of view. live in However, in the illustrated example, the play area is limited to a narrower width W in consideration of the suitability of information processing and the presence of surrounding objects.

In this way, the play area 184 can have the role of setting an appropriate movable range in addition to reducing the possibility of deviation from the angle of view. In addition, by setting different rules for the play area according to the distance range from the camera, priority is given to the movable range in areas with a narrow field of view, and other factors such as content conditions in areas with a wide field of view. can adapt to changes in

FIG. 8 schematically shows another example of play area setting. The figure shows the real space viewed from the left side of the user, and the user 120 is present facing the imaging device 12 . Boundary planes 192a and 192b of the field of view indicated by dotted lines are determined by the vertical angle of view θ _v of the camera of the imaging device 12 . Based on such a horizontal boundary plane of the field of view, the boundary plane of the play area is set such that the distance Z from the imaging device 12 is in the range of Z1≦Z<Z3.

In the example shown in the figure, the boundary planes 192a and 192b of the field of view are aligned with the boundary plane of the play area 194 over the range of Z1≦Z<Z3. That is, unlike the play area shown in FIG. 7, no margin area is set between the boundary plane of the field of view. In other words, the area corresponding to the upper end to the lower end of the photographed image is used as the play area without providing a margin such as the horizontal surfaces 196a and 196b imaged inward from the upper and lower ends of the photographed image. In this way, a warning is issued at the same time as the object deviates from the angle of view, so there is a high possibility that the position cannot be acquired even for a very short time. On the one hand, it allows a wider range of warnings to go unnoticed.

Such settings are effective for a camera with a narrow angle of view. When the angle of view is narrow, if the margin area is set as shown in FIG. 7, even a slight movement may result in frequent warnings. Therefore, comfort can be prioritized by securing a wide play area, even if the accuracy of information processing is slightly deteriorated. For example, when using a camera with a wide horizontal angle of view and a narrow vertical angle of view, as shown in FIGS. By changing the positional relationship of , it is possible to appropriately balance information processing accuracy and a comfortable operating environment.

FIG. 9 is a diagram for explaining the influence of the presence or absence of a margin area on processing. (a) in the upper part of the same figure is a time chart of the user's movement and each process when the play area is set by providing a margin area for the field of view, and (b) in the lower part is a time chart of the user's movement and each process when the field of view and the play area are matched. be. That is, as schematically shown on the right side, a white play area is set for the imaging device 12. In (a), a hatched margin area is set between the area outside the field of view and a black area. are provided. The white, shaded, and black rectangles in the "Position" column of the time chart represent the time the user is out of the play area, margin area, and field of view, respectively.

For comparison, both settings assume that the user leaves the field of view at time t2, and that the user returns to the play area as a result of displaying the warning for the same amount of time. The period during which the position information acquisition unit 80 can acquire position information in response to such user movement is indicated by the "position acquisition" column, and the period during which a warning is displayed is indicated by the "warning" column, indicated by rectangles. In the case of (a), the player leaves the play area at time t1 before the player leaves the field of view at time t2, so the warning is displayed from that point until the player returns to the play area at time t3. On the other hand, in the case of (b), since no margin area is set, a warning is displayed between time t2 and time t4 when the subject is out of the field of view.

As shown in the figure, in (a), the position information can be obtained even after the warning is displayed, so it is guaranteed that the player is outside the play area based on the information. In the case of (b), since the location information cannot be obtained at the same time as the warning, the validity of the warning is not guaranteed during the period when the warning is displayed. Therefore, for example, when the user's image is not detected due to occlusion at the boundary of the field of view, there is a possibility that the user may misunderstand that the user is out of the field of view and display a warning.

In addition, even if the user returns to the play area at the same timing after the warning is displayed in (a) and (b), the period in which the position information cannot be acquired becomes longer in (b). In the illustrated example, there is a period of going out of the field of view even in (a), but depending on the width of the margin area, there is a possibility that the user will return in response to the warning display before going out of the field of view. Compared with the case of b), a period in which position information cannot be acquired is less likely to occur. On the other hand, even if the user moves in the same manner, in the case of (a), the warning is displayed at an earlier time t1, so the constraint on the user becomes stricter. Based on these characteristics, it is possible to set the optimal play area based on the priority according to the situation, such as whether you want to maximize the area where you can avoid issuing a warning, or whether there is room in the area even if you do not do so. desirable.

Note that not only whether or not to provide a margin area, but also the width of the margin area and the shape of the play area may be optimized according to the assumed situation and the direction in the real space. For example, even with the same angle of view, depending on the user's posture, which is assumed depending on the content and environment, such as playing games, whether playing standing or sitting, facing a desk or sitting on a sofa, etc. The shape of the play area may vary.

When the settings of FIGS. 7 and 8 are applied at the same time, the set play area has a quadrilateral cross-section with respect to the distance Z from the camera. Any shape, such as a cylindrical shape, may be used. Also, if the optical axis is not horizontal, the distance Z from the camera is not horizontal to the floor. At this time, the shape of the play area may be defined by the distance in the optical axis direction or by the horizontal distance in real space. If the angle formed by the optical axis of the camera and the horizontal plane of the real space is measured by an acceleration sensor or the like incorporated in the imaging device 12, coordinate conversion can be easily realized. It is understood by those skilled in the art that various specific data formats for setting regions in a three-dimensional space can be considered.

On the other hand, the play area may be set indirectly by defining an area corresponding to the play area on the image plane. FIG. 10 is a diagram for explaining a play area setting method on the image plane. The figure shows the field of view of the camera facing the object space, and thus the plane of the photographed image. In the case of a monocular camera, the left and right ends having a predetermined width x with respect to the plane of the captured image 300 are defined as margin areas, and the area 302 other than the margin areas is defined as the play area corresponding area.

A region in the three-dimensional space defined by this region 302 is a region 304 whose horizontal width increases in proportion to the distance Z in the depth direction. That is, when focusing on the horizontal direction, by setting the area 302, the inside of the vertical planes 186a and 186b shown in FIG. 8 is set as the play area. When the play area corresponding area 302 is set on the image plane in this way, the cross section of the play area in the real space becomes a similar shape with a size corresponding to the distance Z. , the degree of freedom of the setting shape is reduced.

On the other hand, since it is possible to directly determine the necessity of warning by judging the inside/outside of the image on the captured image and the play area corresponding area 302, there is less room for error than through the user's position information. Although the illustrated example is the case where the margin area is provided in the horizontal direction, the same applies to the case where the margin area is provided in the vertical direction. For the same reason as described above, whether or not to provide a margin area and the adjustment of its width x may be performed independently depending on the direction. These elements may be changed according to the content or situation.

In the case of the stereo camera shown in (b), the camera that first leaves the field of view differs depending on whether the user is at the left end or the right end of the field of view of the camera. For example, when the user is near the right end of the field of view of the camera, the image 308a in the left viewpoint camera image 306a is closer to the right end than the image 308b in the right viewpoint camera image 306b. Therefore, if the user moves to the right as it is, it will first come out of the field of view of the left viewpoint camera. Conversely, when the user is near the left end of the field of view of the camera, the user first leaves the field of view of the right viewpoint camera.

Therefore, when a play area is set within a field of view that can be captured by both stereo cameras, an area 310a excluding a margin area of a predetermined width x at the right end of the photographed image plane of the camera at the left viewpoint, and the photographing of the camera at the right viewpoint. At the same time, an area 310b on the image plane, excluding the margin area of the predetermined width x at the left end, is set as the play area corresponding area. Then, when the image of the user exits the play area corresponding area in one of the images, it is determined whether the warning is necessary.

FIG. 11 exemplifies a display screen generated when the warning status determination unit 82 determines that a warning is necessary. The display screen 200 has a configuration in which a warning image 202 representing a warning is added to a content image such as a game screen. In the illustrated example, the warning image 202 indicates that the player is out of the play area by text information "Out of play area". Furthermore, the urgency is expressed by including a mark representing danger.

However, the form of the warning image 202 is not limited to this, and by devising colors and shapes, it may be represented only by graphics or marks, or such graphics or marks may be combined with character information. Also, the warning image 202 may be changed for each content or according to the contents of the display at that time. The warning image 202 is basically displayed superimposed on the image of the content because of its nature of alerting the user. On the other hand, especially when the image of the content is a virtual world with a sense of depth, the sudden appearance of a part image without a sense of depth may cause the user to be surprised or feel uncomfortable. be done.

Therefore, as shown in the figure, the periphery of the warning image 202 is blurred by blur processing, α-blending, or the like. The entire warning image 202 may be translucent. This makes it possible to enhance affinity with the virtual world that is expressed in the background while being in the foreground. In addition, it is desirable to display the warning image 202 at a position away from the viewpoint so as not to give a feeling that the line of sight is suddenly blocked. For example, in a display mode in which the field of view changes according to the user's line of sight, the viewpoint is inevitably fixed at the center of the screen.

In addition, since it is easier for people to lower their eyes rather than raise them, the warning image 202 is displayed in the lower half area of the screen (the area below the center line C in the vertical direction) regardless of the display mode. can be moved from the content image without difficulty. As a result, it is possible to prevent the eyes from taking a long time to recognize an image that suddenly appears because the eyes are out of focus, and to prevent the person from becoming intoxicated due to minute movements of the eyeballs.

Next, the operation of the information processing apparatus 10 that can be realized by the configuration described above will be described. FIG. 12 is a flow chart showing a processing procedure for the information processing apparatus to generate output data according to user's movement in the present embodiment. This flowchart is started, for example, when the user requests the information processing apparatus 10 to start processing via the input device 14 .

First, the photographed image acquisition unit 74 of the information processing device 10 requests the imaging device 12 to start photographing, and in response to the request, starts acquiring the data of the photographed image photographed and output by the imaging device 12 (S10). On the other hand, the warning condition determination unit 82 reads setting information related to the play area from the area setting information storage unit 85 (S12). At this time, the setting information associated with the content to be executed selected by the user or the like may be selected. It is also possible to identify whether or not the user is seated, the surrounding conditions, and the like by analyzing the captured image, and select the setting information based on such information.

Next, the information processing section 76 requests the output data generation section 78 to output the real-time captured image acquired by the captured image acquisition section 74 to the HMD 18 . By displaying the captured image on the HMD 18, the user is urged to check whether there are any obstacles in the surroundings (S14, S16). At this time, the output data generator 78 superimposes text information such as "Is there anything dangerous?" As a result, the user can recognize the angle of view of the camera in advance, so that the user can move while keeping in mind the area equivalent to the play area.

Knowing the play area makes it possible to clarify the range of movement of one's own, and it is possible to judge whether the surrounding objects can be obstacles. If the user moves the object away as necessary and then performs the subsequent processing on the condition that he/she inputs confirmation, it will be possible to monitor deviations out of the field of view of the camera and at the same time detect obstacles. Collision monitoring is also carried out. As long as there is no confirmation input from the user, the captured image is kept displayed (N of S16, S14).

When the user inputs confirmation (Y of S16), the positional information acquisition unit 80 starts acquiring the user's positional information based on the captured image (S18). Then, the information processing section 76 performs information processing using the information, and the output data generation section 78 draws the display image of the content as a result (S20). On the other hand, the warning status determination section 82 monitors whether or not the user has left the play area (S22). Strictly speaking, it is determined that the user has left the play area when the HMD 18 worn by the user or the marker provided on the input device 14 held by the user has left the play area.

However, the criteria for the determination may vary depending on the user's position information acquisition means. For example, when a depth map is created from the user's image itself in the stereo image and used as position information, the time when the center of gravity of the user leaves the play area may be used. The same is true when tracing contours. Also, when the face is tracked by pattern matching such as face detection processing, it may be when the face is no longer recognized in the play area.

If the user has not left the play area (N of S22), the output data transmission unit 86 outputs the image of the content drawn in S20 to the HMD 18 as it is (S26). If the user has left the play area (Y in S22), the warning situation determination unit 82 notifies the output data generation unit 78 of that fact, and the output data generation unit 78 superimposes the warning image on the image of the content ( S24). Then, the output data transmission unit 86 outputs the image superimposed with the warning image to the HMD 18 (S26). In the process of S26, voice data may be output at the same time.

During the period when there is no need to end the process, such as when there is a request to stop the process from the user, the content image is drawn based on the position information, and if necessary, the warning image is superimposed and output to the HMD 18. The process is performed at a predetermined rate. (N of S28, S20 to S26). Then, when it becomes necessary to terminate the processing, all the processing is terminated (Y of S28).

It should be noted that in the flowchart of the figure, the period during which the user is outside the play area coincides with the period during which the warning image is superimposed. That is, when the user returns to the play area, the warning image disappears. On the other hand, the condition for hiding the warning image may be provided independently of the condition for displaying the warning image. FIG. 13 schematically shows the display/non-display state transition of the warning image. First, as described above, the criterion for determining whether the warning image is displayed from a state in which the warning image is not displayed is that the user has left the play area (rightward arrow).

On the other hand, the criterion for changing the warning image from the displayed state to the non-displayed state is not that the player has entered the play area, but that a separately set condition is satisfied (leftward arrow). For example, a warning non-display area is set as a range narrower than the play area, and the warning image is hidden when the player enters the area. Alternatively, it is assumed that a predetermined time has passed after entering the play area. By doing so, the once displayed warning is less likely to be hidden. As a result, when the user is in the vicinity of the boundary of the play area, chattering in which the inside/outside determination result for the play area is switched with a slight movement, and the warning repeatedly appears and disappears can be prevented.

According to the present embodiment described above, in a system that acquires position information of a user based on an image in a photographed image and performs information processing using the result, a play area is set in a three-dimensional space in the real world. , when the user leaves the play area, a warning to that effect is presented to the user by display or the like. This prevents the user from wearing the HMD and being unable to see the outside world or being immersed in the displayed virtual world, and unintentionally leaving the field of view of the camera and causing a breakdown in information processing.

Here, the play area is set based on the field of view of the camera. At this time, factors such as whether the play area is set inside the field of view with the margin area in mind, or whether it is the same as the field of view, and when the margin area is in the area, are optimized. Adjusting the margin area in this way regulates the range in which the user can move without warning, and at the same time, adjusts the frequency at which position information cannot be obtained due to the user going out of the field of view, and thus the accuracy of position information acquisition. It's nothing but being there. Therefore, it is possible to make settings that match the priority, such as whether to allow movement in a wide range as much as possible even if it sacrifices some deterioration in the accuracy of location information acquisition, or to maintain the accuracy of location information acquisition even if the range is limited. Become.

Such priority changes depending on the horizontal/vertical angle of view of the camera, the content of information processing, the user and their surroundings, and so on. Accuracy and user comfort can be balanced. Also, since there are no particular restrictions on the shape of the play area, if the shape and size are set appropriately, in addition to preventing deviance from the camera's field of view, compliance with the appropriate amount of movement assumed from content such as games, and the surrounding It can also play a role such as collision avoidance with obstacles at the same time.

When displaying the warning, the part image representing the warning is superimposed on the original display image such as the game screen. At this time, by blurring or making the peripheral edge semi-transparent, affinity with the original display image is enhanced, and discomfort due to the sudden appearance of an irrelevant image is reduced. In addition, by setting a predetermined distance from the point of view and displaying the image in the lower half of the screen, discomfort and motion sickness due to obstruction of the line of sight can be suppressed. be able to.

The present invention has been described above based on the embodiments. It should be understood by those skilled in the art that the above embodiments are merely examples, and that various modifications can be made to combinations of each component and each treatment process, and that such modifications are within the scope of the present invention. be.

8 information processing system, 10 information processing device, 12 imaging device, 14 input device, 16 flat panel display, 18 HMD, 22 CPU, 24 GPU, 26 main memory, 72 input information acquisition unit, 74 captured image acquisition unit, 76 information Processing unit 78 Output data generation unit 80 Location information acquisition unit 82 Warning status determination unit 85 Area setting information storage unit 86 Output data transmission unit.

Claims (17)

an information processing unit that detects the position of an object in real space and performs information processing based on the object;
an image generation unit that generates image data to be displayed as a result of the information processing;
a warning situation determination unit that determines the necessity of warning the user based on the play area whose shape is changed according to the situation, and superimposes a warning image on the image generation unit;
an output unit that outputs image data generated by the image generation unit to a display device;
An information processing device comprising: 2. The information processing apparatus according to claim 1, wherein the warning situation determination section changes the shape of the play area according to content. 2. The information processing apparatus according to claim 1, wherein the warning situation determination section changes the shape of the play area according to the environment of the real space. 2. The information processing apparatus according to claim 1, wherein the warning state determination unit changes the shape of the play area according to the posture state of the user. 5. The information processing apparatus according to any one of claims 1 to 4, wherein the warning situation determination unit sets the play area including two parallel vertical planes as boundaries. 6. Any one of claims 1 to 5, wherein the warning situation determination unit sets the play area in a shape having a quadrilateral cross-section with respect to a distance from an imaging device for detecting the object. The information processing device described. 7. The information processing apparatus according to any one of claims 1 to 6, wherein the warning condition determination unit determines that a warning to the user is necessary when the object moves out of the play area. 8. The output unit according to any one of claims 1 to 7, wherein the output unit outputs to the display device a confirmation screen prompting the user to confirm whether or not there is an obstacle, and the confirmation screen includes a real-time photographed image. The information processing device described. 3. The warning situation determination unit acquires the distance of the object in the optical axis direction based on a stereo image captured by a stereo camera that constitutes an imaging device for detecting the object. 9. The information processing device according to any one of 1 to 8. 10. An information processing apparatus according to claim 9, wherein the optical axis direction of said stereo camera is not in the horizontal direction of real space. When the optical axis of the imaging device is not in the horizontal direction of the real space, the warning situation determination unit defines the shape of the play area with respect to the distance in the optical axis direction, thereby determining the distance between the play area and the object. 11. The information processing apparatus according to claim 9 or 10, wherein the positional relationship of is acquired. The warning situation determination unit defines the shape of the play area with respect to the horizontal distance in real space, and if the optical axis of the imaging device is not in the horizontal direction, the angle formed by the optical axis and the horizontal direction 11. The information processing apparatus according to claim 9, wherein the positional relationship between the play area and the object is obtained by coordinate transformation based on the measured value of . The warning situation determination unit prescribes an area corresponding to the play area on a plane of an image captured by an imaging device for detecting the object, and determines the position of the image of the object on the captured image. 2. The information processing apparatus according to claim 1, wherein the necessity of warning is determined. The warning situation determination unit sets an area corresponding to a space captured by both stereo cameras constituting the imaging device in each of the stereo images captured by the stereo cameras, so that the object is detected in either one of the images. 14. The information processing apparatus according to claim 13, wherein it is determined that said warning is necessary when the image of the object has left the set area. The warning situation determination unit sets the boundary plane of the play area with reference to a boundary plane of intersections of angles of view of stereo cameras constituting an imaging device for detecting the object. Item 1. The information processing apparatus according to item 1. detecting the position of an object in real space and performing information processing based on the object;
generating image data to be displayed as a result of the information processing;
determining the necessity of warning the user based on the play area whose shape is changed according to the situation;
superimposing a warning image on the image to be displayed according to the determination;
outputting data of the generated image to a display device;
A warning presentation method by an information processing device, comprising: A function of detecting the position of an object in real space and performing information processing based on the object;
a function of generating image data to be displayed as a result of the information processing;
A function of judging the necessity of a warning to the user based on the play area whose shape is changed according to the situation, and superimposing a warning image by notifying the function of generating data of the image to that effect;
a function of outputting data of the generated image to a display device;
A computer program characterized by causing a computer to implement

Images