JP2019045998A - Information processing device, method thereof and program - Google Patents

Abstract

To display multiple pieces of display information properly.SOLUTION: An information processing device comprises: acquisition means acquiring a position attitude of the point of view of a user; holding means holding multiple virtual objects together with their display attribute; selection means selecting multiple display elements to be shown to the user among the multiple virtual objects; and control means controlling a context of display so that the multiple display elements are displayed on a display device on the basis of the position attitude of the point of view of the user and the display attribute of each of the multiple display elements.SELECTED DRAWING: Figure 1

Description

  The present invention relates to displaying information to a user.

  BACKGROUND ART In recent years, research on mixed reality (MR) systems has been actively conducted for the purpose of seamless connection between real space and virtual space. For example, a head mounted display (HMD) can be used as an image display device that presents these systems. With the progress of MR system research, the virtual objects displayed on the MR are diverse. Not only a virtual object imitating a real object, for example, a hand object representing a user's hand or a 3D object having a role of a panel presenting information. Patent Document 1 enables display of a virtual object for notifying approach to a real object for the purpose of securing the safety of the user when using a display unit with a high sense of immersion, such as an HMD.

Patent No. 5777786 gazette

  Although the method of Patent Document 1 displays an approach alert notification object with a real object, when another virtual object is displayed in front of it, the alert notification is interrupted and may not be viewed by the user. There is sex. As described above, when a large number of pieces of display information exist, the display order required by the user is interrupted by other display information because the display order is not determined, and there is a problem that the user can not visually recognize.

  The present invention has been made in view of the above problems, and has an object of efficiently displaying a plurality of display information.

  According to one aspect of the present invention, an information processing apparatus includes an acquisition unit that acquires the position and orientation of a user's viewpoint, a holding unit that holds a plurality of virtual objects together with their respective display attributes, and the plurality of virtual objects. Based on the selection means for selecting a plurality of display elements to be displayed to the user, the position and orientation of the viewpoint of the user, and the display attribute of each of the plurality of display elements, the anteroposterior relationship of the display of the plurality of display elements is And control means for controlling and displaying on the display device.

  According to the present invention, it is possible to display information appropriate for the user.

It is a block diagram showing an example of functional composition of a system concerning an embodiment. It is a figure showing the example of the hardware constitutions of an information processor. It is a figure which shows the example of a display component. It is a flow chart of processing of an information processor. It is a figure which shows the example of a presentation screen structure to a user.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific embodiments of the configuration described in the claims.

First Embodiment
FIG. 1 is a block diagram showing an example of a functional configuration of a system according to the present embodiment. As shown in FIG. 1, the system according to the present embodiment includes an information processing device, an HMD as an example of a head-mounted display device, and an image display device 1200.

  The information processing apparatus 1000 includes a captured image acquisition unit 1010, a position and orientation measurement unit 1020, a virtual data holding unit 1030, a display component selection unit 1040, a display component setting unit 1050, and a display configuration control unit 1060. The information processing apparatus 1000 further includes a virtual space generation unit 1070 and a mixed reality image generation unit 1080.

  The HMD 1100 includes an image pickup unit 1110 and an image display unit 1120. Here, the information processing apparatus 1000 and the HMD 1100 are connected to enable data communication. The connection between the information processing apparatus 1000 and the HMD 1100 may be wired or wireless.

  First, the information processing apparatus 1000 will be described. FIG. 2 is a hardware configuration diagram of the information processing apparatus 1000 in the present embodiment. In the same figure, a CPU 2010 generally controls each device connected via a bus 2000. The CPU 2010 reads and executes processing instructions and programs stored in a read only memory (ROM) 2020. An operating system (OS), processing programs according to this embodiment, device drivers and the like according to the present embodiment are stored in the ROM 2020, temporarily stored in a random access memory (RAM) 2030, and appropriately executed by the CPU 2010. Also, the input I / F 2040 inputs an input signal in a format that can be processed by the information processing apparatus 1000 from an external device (such as a display device or an operation device). Also, the output I / F 2050 outputs an output signal to an external device (such as a display device) in a format that can be processed by the external device.

  The captured image acquisition unit 1010 acquires an image of a physical space (physical space image) captured by the image capturing unit 1110, and inputs the acquired image to the position and orientation measurement unit 1020 and the mixed reality image generation unit 1080.

  The position and orientation measurement unit 1020 performs image processing using the physical space image acquired by the captured image acquisition unit 1010, and extracts characteristic information (feature information) such as points and lines in the image. By associating the extracted feature information with the preset position of the feature information prepared in advance, the position and orientation of the user's viewpoint are measured. Furthermore, the position and orientation of another real object can also be determined from the position observed from the HMD 1100 for which the position and orientation have been determined. The image is input to the mixed reality image generation unit 1080 and the virtual data storage unit 1030. Although the position and orientation measurement by image processing is described in this embodiment, the position and orientation may be measured using infrared light, the position and orientation may be measured using an ultrasonic wave, or a magnetic sensor is used. Position and orientation may be measured. Furthermore, the position and orientation may be measured using a depth sensor, or the position and orientation may be measured mechanically.

  The virtual data holding unit 1040 holds data of the virtual space, such as data (shape information and position and orientation information) related to a virtual object constituting the virtual space and data related to a light source to be irradiated into the virtual space. These data are used by the virtual space generation unit 1070, the display component selection unit 1040, and the display configuration control unit 1060.

  The display component selection unit 1040 selects the display component to be displayed to the user from the data related to the virtual object acquired from the virtual data storage unit 1030, and sets the shape information and position and orientation information of the virtual object to display component Input to the part 1050.

  FIG. 3 shows display components assumed to be displayed in this embodiment, and each of the notification object 3000, the menu panel 3200, the hand model 3301 and the 3DCG model 3302 displayed at a fixed position in the field of view of the user is there.

  The notification object 3000 refers to, for example, an object that reports the danger of a plurality of users approaching, approaching with a real object, and collision. In the present embodiment, the notification object 3000 is described as an object notifying of the danger of collision, but the present invention is not limited to this. That is, it may be alerting when it has been used for a long time, or the user may set information to be displayed with the highest priority.

  The menu panel 3200 points to a menu that is performed on the selected virtual object, for example, including control of movement and display non-display of the virtual object. In the present embodiment, the case where the display position is displayed at a fixed position in the field of view as the menu panel 3200 and the notification object 3000 as specific objects will be described, but the present invention is not limited to this. That is, the menu may be a menu that fixes the display position of the menu in the virtual space, or a menu that allows the user to specify the display position. Furthermore, it may be a menu that can switch whether to fix the display position in the view or in the virtual space.

  The hand model 3301 is an object which can be acquired by the position and orientation measurement unit 1020 and in which the user's hand is masked by CG. The hand model in the present embodiment is not limited to this, and may be, for example, a part of the body of the user other than the hand. The technique for generating a hand model that can be seen from a predetermined position is a known fact, so detailed description thereof will be omitted.

  In the present embodiment, the 3DCG model 3302 is a 3DCG model, but the present invention is not limited thereto. That is, it may be an object representing a contact portion between 3D models in a virtual space, or may be a text indicating a model name. Furthermore, it may be a menu that can switch whether to fix the display position in the view or in the virtual space.

  The display component setting unit 1050 sets display attributes for classifying display elements. In this embodiment, the notification object 3000 is set as attribute 1, the menu panel 3200 is set as attribute 2, the hand model 3301 and the 3DCG model 3302 are set as attribute 3, and setting information is input to the display order determination unit. In the present invention, the setting of the attribute may be, for example, setting information determined in advance or a setting determined by the user.

  The display configuration control unit 1060 calculates and controls the anteroposterior relationship between the display components from the attribute information set by the display component setting unit 1050 and the position information of the plurality of objects acquired from the virtual data storage unit 1030. In this embodiment, the attribute 1 is displayed in front of the gaze direction of the user in order from the number, and the anteroposterior relationship between the hand model of the attribute and the 3DCG model is determined based on position and orientation information of each model. The configuration control information is input to the virtual space generation unit 1070.

  The virtual space generation unit 1070 generates a virtual space based on the virtual data held by the virtual data holding unit 1030 and the configuration control information acquired from the display configuration control unit 1060, and inputs the virtual space to the mixed reality image generation unit 1080.

  The mixed reality image generation unit 1080 generates a virtual space image based on the virtual space generated by the virtual space generation unit 1070 and the measurement result of the position and orientation measurement unit 1020. Furthermore, the virtual space image is superimposed on the physical space image acquired by the captured image acquisition unit 1010 to generate an image of mixed reality space (mixed reality image). In addition, since a technique for generating a virtual space visible from a predetermined position is known, the detailed description thereof is omitted. After that, the mixed reality image generation unit 1080 outputs the generated mixed reality image to the image display unit 1120 of the HMD 1100 (display control). Alternatively, the image may be displayed on the image display device 1200 simultaneously. It is not limited to the HMD, but may be a device capable of displaying, and may be a display terminal such as a tablet or a smartphone.

  These functional units are realized by the CPU 2010 developing a program stored in the ROM 2020 in the RAM 2030 and executing processing according to each flowchart to be described later. For example, when hardware is configured as a substitute for software processing using the CPU 2010, an arithmetic unit or a circuit corresponding to the processing of each functional unit described here may be configured.

  Next, the HMD 1100 will be described. The HMD 1100 is a display device configured with a liquid crystal screen or the like, and is provided for the right eye and the left eye respectively, and each of the HMD 1100 is positioned in front of the right eye and the left eye of the user wearing the head on the head. Is attached to In addition, images with parallax are respectively displayed in stereo on the left and right screens.

  The image capturing unit 1110 captures an image of a physical space displayed on the HMD 1100, and inputs the captured image to the information processing apparatus 1000 as a physical space image.

  The image display unit 1120 displays the mixed reality image generated by the information processing apparatus 1000.

  As described above, in the present embodiment, the HMD 1100 will be described as a video see-through HMD that displays a mixed reality image generated based on an image captured by an imaging device on a display device. However, the HMD 1100 may use an optical see-through HMD that superimposes and displays a virtual space image on a display medium that can be observed through the real space.

  FIG. 4 is a flowchart of processing performed by the information processing apparatus 1000 to generate a mixed reality space image and output the mixed reality space image to the HMD 1100 or the image display apparatus 1200.

  First, in step S4000, the position and orientation measurement unit 1020 acquires a physical space image from the image capturing unit 1110, and measures the position and orientation of the HMD 1100 from the image. Then, the process proceeds to step S4100.

  In step S4100, the virtual data holding unit 1030 acquires position and orientation information and shape information of a plurality of objects from the position and orientation measurement unit 1020, and the process proceeds to step S4200.

  In step S4200, the display component selection unit 1040 acquires position and orientation information of the user from the virtual data storage unit 1030, and selects a component to be displayed to the user from the position and orientation information. For example, when a danger of approach and collision between a user and a real object is detected, a danger notification is added as a component, and when positional information of the user's hand is acquired, a hand model is added to the component. Then, the process proceeds to step S4300.

  In step S4300, the display component setting unit 1050 sets the attribute of the component. In the present embodiment, it is assumed that the attribute has been set in advance, the danger notification is set as attribute 1, the menu panel is set as attribute 2, and the hand model and 3DCG model are set as attribute 3. Then, the process proceeds to step S4400.

  In step S4400, the display configuration control unit 1060 acquires attribute information of the component from the display component setting unit 1050, and acquires position information of a plurality of objects from the virtual data holding unit 1030. Furthermore, it calculates and controls the anteroposterior relationship between the display components. Then, the process proceeds to step S4500.

  In step S4500, the virtual space generation unit 1070 generates a virtual space using the data related to the virtual space held by the virtual data holding unit 1030 and the display component information controlled by the display configuration control unit 1060.

  An example in which the display order is changed by the control of the display configuration control unit 1060 in step S4500 will be described using FIG. FIG. 5 is a view showing a control example of the display configuration control unit 1060 according to the present embodiment.

  FIG. 5A is a diagram when the display component setting unit 1050 does not detect the notification object 3100 and the hand model 3301 and sets an attribute in the menu panel 3200 and the 3DCG model 3302. It shows the arrangement of multiple objects viewed from the user. Attribute 2 is set in the menu panel 3200 by the display component setting unit 1050, and attribute 3 is set in the 3DCG model 3302. At this time, the display configuration control unit 1060 controls to display in front of the user's gaze direction sequentially from 1 of the attribute. As a result, as shown in FIG. 5A, the menu panel 3200 is displayed in front of the user's gaze direction, and the 3DCG model 3302 is displayed in the back.

  FIGS. 5B and 5C are diagrams showing the case where the display component setting unit 1050 does not detect the notification object 3100 and the menu panel 3200, and an attribute is set for the hand model 3301 and the 3DCG model 3302. It is. It shows the arrangement of multiple objects viewed from the user. Attribute 3 is set in the hand model 3301 and the 3DCG model 3302 by the display component setting unit 1050. In the case of the same attribute, the display configuration control unit 1060 controls the display configuration based on depth information of the virtual space. If the depth information of the hand model 3301 is in front of the 3DCG model 3302 with respect to the user in the virtual space, the hand model 3301 is displayed in front as shown in FIG. 5B. On the other hand, in the virtual space, when the depth information of the 3DCG model 3302 is in front of the hand model 3301 with respect to the user, the 3DCG model 3302 is displayed in front as shown in FIG. 5C.

  FIG. 5D shows the arrangement of a plurality of objects viewed by the user when the display component setting unit 1050 sets attributes for the notification object 3100, the menu panel 3200, the hand model 3301 and the 3DCG model 3302. FIG. The display component setting unit 1050 sets the attribute 1 in the notification object 3100, and the attribute 2 in the attributes 2 and 3 and the DCG model 3302 in the menu panel 3200. At this time, the display configuration control unit 1060 controls the display 1 so as to be displayed in front of the user's gaze direction sequentially from the attribute 1. Therefore, as shown in FIG. 5C, the notification object 3100 is displayed in the foremost position with respect to the user's gaze direction, and the menu panel 3200, the 3DCG model 3302, and the hand model 3301 are displayed in this order from the back.

  In step S4600, the mixed reality image generation unit 1080 generates a virtual space image based on the virtual space generated by the virtual space generation unit 1070 and the position and orientation of the position and orientation measurement unit 1020. Furthermore, the virtual space image is superimposed on the physical space image acquired by the captured image acquisition unit 1010 to generate a mixed reality image.

  In step S4700, the generated mixed reality image is output to image display unit 1120 or image display device 1200 of the HMD. If the end instruction of this process has not been input, the process returns to step S4000.

(Modification)
In the above embodiment, an example has been described in which the present invention is applied to an MR system that displays an image obtained by combining a virtual space image with a captured image. However, the present invention is not limited to this. For example, the present invention can be applied to a system that displays an image in which only a real model is rendered. In that case, in the above embodiment, the virtual space generation unit 1070 may generate a virtual space configured by only the real model.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

1000 Information processing apparatus 1010 Captured image acquisition unit 1020 Position and orientation measurement unit 1030 Virtual data storage unit 1040 Display component selection unit 1050 Display component setting unit 1060 Display configuration control unit 1070 Virtual space generation unit 1080 Mixed reality image generation unit 1100 HMD
1110 image pickup unit 1120 image display unit 1200 image display apparatus

Claims (11)

Acquisition means for acquiring the position and orientation of the user's viewpoint;
Holding means for holding a plurality of virtual objects together with their respective display attributes;
Selection means for selecting a plurality of display elements to be displayed to the user among the plurality of virtual objects;
And control means for controlling the front-rear relationship of the plurality of display elements to display them on the display device based on the position and orientation of the viewpoint of the user and the display attributes of the plurality of display elements. An information processing apparatus comprising:   The information processing apparatus according to claim 1, further comprising setting means for setting display attributes for the plurality of virtual objects.   The information processing apparatus according to claim 1, wherein the display attribute is a number, and the control means controls to display in the order of the number.   The information processing apparatus according to claim 1, wherein the control unit displays a specific virtual object at a fixed position.   2. The apparatus according to claim 1, wherein the acquisition unit acquires the position and orientation based on a result measured using a magnetic sensor, a sensor using infrared light, a sensor using ultrasonic waves, or a depth sensor. Information processor as described.   The image processing apparatus further includes an imaging unit configured to capture an image of the physical space, wherein the control unit controls the virtual space image to be superimposed and displayed on the image of the physical space captured by the imaging unit. An information processing apparatus according to claim 1.   The information processing apparatus according to claim 1, wherein the display device is an optical see-through display device.   The information processing apparatus according to claim 1, wherein the display device is a video see-through display device.   The information processing apparatus according to claim 1, wherein the display device is a head-mounted display device. An acquisition step of acquiring the position and orientation of the user's viewpoint;
A selection step of selecting a plurality of display elements to be displayed to the user from the plurality of virtual objects from a holding unit which holds a plurality of virtual objects together with respective display attributes;
And displaying the plurality of display elements on the display device based on the position and orientation of the viewpoint of the user and the display attributes of the plurality of display elements. Information processing method to be.   A program which causes a computer to function as each means of the information processing apparatus according to any one of claims 1 to 9.

Images