WO2023112838A1

WO2023112838A1 - Information processing device

Info

Publication number: WO2023112838A1
Application number: PCT/JP2022/045377
Authority: WO
Inventors: 智仁山▲崎▼
Original assignee: 株式会社Ｎｔｔドコモ
Priority date: 2021-12-16
Filing date: 2022-12-08
Publication date: 2023-06-22

Abstract

This information processing device comprises: an acquisition unit that acquires visual line information indicating the visual line of a user; a display control unit that displays a first virtual object and a second virtual object at positions not overlapping each other, as seen from the user, in a virtual space; and an operation control unit that causes the second virtual object to transition from inactive to active after a transition has occurred from a first state where the visual line of the user indicated by the visual line information intersects the first virtual object and the first virtual object is active to a second state where the visual line of the user does not intersect the first virtual object.

Description

Information processing equipment

The present invention relates to an information processing device.

In MR (Mixed Reality) technology, the real environment perceived by the user is augmented by a computer. By using this technology, for example, it is possible to precisely superimpose the virtual space on the real space that the user sees through the MR glasses worn on the head.

For example, Patent Literature 1 discloses an information processing device that displays a realistic virtual reality image while ensuring the safety of a user riding in a vehicle. The information processing device includes AR goggles on which images of a plurality of virtual first objects are displayed. Also, the information processing device detects a second object that actually exists outside or inside the vehicle. Further, when it is determined that the image of the first object is superimposed on the visibility ensuring area that affects the visibility of the user with respect to the second object, the information processing device superimposes the image of the first object on the visibility ensuring area. For example, the position of the image of the first object is moved so as not to occur.

JP 2020-204984 A

However, the technique according to Patent Document 1 defines the positional relationship between the virtual first object and the real second object, and the second object is determined according to which object the user is paying attention to. It was not possible to change the state of the object.

In view of the above, the present invention provides a method in which, of two virtual objects arranged in a virtual space, when the target of the user's attention changes from a state in which the user focuses on one virtual object, the other virtual object is displayed. The problem to be solved is to change the state of

An information processing apparatus according to a preferred aspect of the present invention includes an acquisition unit that acquires line-of-sight information indicating a user's line of sight, and a first virtual object and a second virtual object that overlap each other in a virtual space as seen from the user. a display control unit for displaying a display at a position where the line of sight of the user does not match; an operation control unit that transitions the second virtual object from inactive to active after the transition to a second state that does not intersect with the first virtual object.

According to the present invention, when the target of the user's attention changes from a state in which the user focuses on one of the two virtual objects arranged in the virtual space, the other virtual object It is possible to change the state of

The figure which shows the whole structure of the information processing system 1 which concerns on 1st Embodiment. 2 is a perspective view showing the appearance of the MR glasses 20 according to the first embodiment; FIG. FIG. 4 is a schematic diagram of a virtual space VS provided to a user U1 by using the MR glasses 20 according to the first embodiment; FIG. 4 is a schematic diagram of a virtual space VS provided to a user U1 by using the MR glasses 20 according to the first embodiment; 2 is a block diagram showing a configuration example of the MR glasses 20 according to the first embodiment; FIG. 1 is a block diagram showing a configuration example of a terminal device 10 according to the first embodiment; FIG. FIG. 4 is an explanatory diagram of an operation example of the display control unit 112 according to the first embodiment; 3 is a block diagram showing a configuration example of a server 30 according to the first embodiment; FIG. 4 is a flowchart showing operations of the server 30 according to the first embodiment; Explanatory drawing about the example of an operation of the display control part 112 which concerns on a modification.

1: First Embodiment Hereinafter, the configuration of an information processing system 1 including a terminal device 10 as an information processing device according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 9. FIG.

1-1: Configuration of the first embodiment
1-1-1: Overall Configuration FIG. 1 is a diagram showing the overall configuration of an information processing system 1 according to the first embodiment of the present invention. The information processing system 1 is a system that uses MR technology to provide a virtual space to a user U1 wearing MR glasses 20, which will be described later.

The information processing system 1 includes a terminal device 10, MR glasses 20, and a server 30. The terminal device 10 is an example of an information processing device. In the information processing system 1, the terminal device 10 and the server 30 are communicably connected to each other via a communication network NET. Also, the terminal device 10 and the MR glasses 20 are connected so as to be able to communicate with each other. In FIG. 1, the terminal device 10 and the MR glasses 20 are combined as a pair of the terminal device 10-1 and the MR glasses 20-1, a pair of the terminal device 10-2 and the MR glasses 20-2, and a pair of the terminal device 10-1 and the MR glasses 20-2. A total of three pairs of devices 10-3 and MR glasses 20-3 are described. However, the number of sets is merely an example, and the information processing system 1 can include any number of sets of the terminal device 10 and the MR glasses 20 .

The server 30 provides various data and cloud services to the terminal device 10 via the communication network NET.

The terminal device 10 displays virtual objects placed in the virtual space on the MR glasses 20 that the user wears on the head. The virtual space is, for example, a celestial space. The virtual objects are, for example, virtual objects representing data such as still images, moving images, 3DCG models, HTML files, and text files, and virtual objects representing applications. Examples of text files include memos, source codes, diaries, and recipes. Examples of applications include browsers, applications for using SNS, and applications for generating document files. Note that the terminal device 10 is preferably a mobile terminal device such as a smart phone and a tablet, for example.

The MR glasses 20 are a see-through wearable display worn on the user's head. The MR glasses 20 are controlled by the terminal device 10 to display a virtual object on the display panel provided for each of the binocular lenses. Note that the MR glasses 20 are an example of a display device.

1-1-2: Configuration of MR Glasses FIG. 2 is a perspective view showing the appearance of the MR glasses 20. As shown in FIG. As shown in FIG. 2, the MR glasses 20 have

temples

91 and 92, a bridge 93,

frames

94 and 95, and

lenses

41L and 41R, like general spectacles. An imaging device 26 is provided in the bridge 93 . The imaging device 26 images the outside world. The imaging device 26 also outputs imaging information indicating the captured image.

Each of the

lenses

41L and 41R has a half mirror. The frame 94 is provided with a liquid crystal panel or an organic EL panel for the left eye and an optical member for guiding light emitted from the display panel for the left eye to the lens 41L. A liquid crystal panel or an organic EL panel is hereinafter generically referred to as a display panel. The half mirror provided in the lens 41L transmits external light and guides it to the left eye, and reflects the light guided by the optical member to enter the left eye. The frame 95 is provided with a right-eye display panel and an optical member that guides light emitted from the right-eye display panel to the lens 41R. The half mirror provided in the lens 41R transmits external light and guides it to the right eye, and reflects the light guided by the optical member to enter the right eye.

The display 28, which will be described later, includes a lens 41L, a left-eye display panel, a left-eye optical member, and a lens 41R, a right-eye display panel, and a right-eye optical member.

With the above configuration, the user can observe the image displayed by the display panel in a see-through state superimposed on the state of the outside world. Further, in the MR glasses 20, of the binocular images with parallax, the image for the left eye is displayed on the display panel for the left eye, and the image for the right eye is displayed on the display panel for the right eye, thereby allowing the user U1 to On the other hand, it is possible to perceive the displayed image as if it had depth and stereoscopic effect.

3 and 4 are schematic diagrams of the virtual space VS provided to the user U1 by using the MR glasses 20. FIG. As shown in FIG. 3, in the virtual space VS, virtual objects VO1 to VO5 representing various contents such as browsers, cloud services, images, and moving images are arranged. The user U1 walks around the public space while wearing the MR glasses 20 on which the virtual objects VO1 to VO5 arranged in the virtual space VS are displayed, thereby turning the virtual space into a private space in the public space. You can experience space VS. Ultimately, the user U1 can act in the public space while receiving benefits brought by the virtual objects VO1 to VO5 placed in the virtual space VS.

Also, as shown in FIG. 4, it is possible for a plurality of users U1 to U3 to share the virtual space VS. By sharing the virtual space VS among a plurality of users U1 to U3, the plurality of users U1 to U3 share one or a plurality of virtual objects VO, and through the shared virtual objects VO, the user U1 ~ It becomes possible to communicate between users U3.

In this embodiment, as shown in FIGS. 3 and 4, any combination of two virtual objects VO out of the plurality of virtual objects VO1 to VO5 does not overlap in the virtual space VS as seen from the user U1. is displayed. Details of the display method of the virtual object VO will be described later.

FIG. 5 is a block diagram showing a configuration example of the MR glasses 20. As shown in FIG. The MR glasses 20 include a processing device 21 , a storage device 22 , a line-of-sight detection device 23 , a GPS device 24 , a motion detection device 25 , an imaging device 26 , a communication device 27 , a display 28 and a speaker 29 . Each element of the MR glasses 20 is interconnected by one or more buses for communicating information. Note that the term "apparatus" in this specification may be replaced with another term such as a circuit, a device, or a unit.

The processing device 21 is a processor that controls the MR glasses 20 as a whole. The processing device 21 is configured using, for example, one or more chips. The processing device 21 is configured using, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic device, registers, and the like. Some or all of the functions of the processing device 21 are realized by hardware such as DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). You may The processing device 21 executes various processes in parallel or sequentially.

The storage device 22 is a recording medium that can be read and written by the processing device 21 . The storage device 22 also stores a plurality of programs including the control program PR1 executed by the processing device 21 .

The line-of-sight detection device 23 detects the line of sight of the user U1. Any method may be used to detect the line of sight by the line of sight detection device 23 . The line-of-sight detection device 23 may detect line-of-sight information based on, for example, the position of the inner corner of the eye and the position of the iris. The line-of-sight detection device 23 also outputs line-of-sight information indicating the line-of-sight direction of the user U1 to the processing device 21, which will be described later, based on the detection result. The line-of-sight information output to the processing device 21 is output to the terminal device 10 via the communication device 27 .

The GPS device 24 receives radio waves from multiple satellites. The GPS device 24 also generates position information from the received radio waves. The positional information indicates the position of the MR glasses 20 . The location information may be in any format as long as the location can be specified. The position information indicates the latitude and longitude of the MR glasses 20, for example. As an example, location information is obtained from GPS device 24 . However, the MR glasses 20 may acquire position information by any method. The acquired position information is supplied to the processing device 21 . The position information output to the processing device 21 is transmitted to the terminal device 10 via the communication device 27 .

The motion detection device 25 detects motion of the MR glasses 20 . The motion detection device 25 corresponds to an inertial sensor such as an acceleration sensor that detects acceleration and a gyro sensor that detects angular acceleration. The acceleration sensor detects acceleration in orthogonal X-, Y-, and Z-axes. The gyro sensor detects angular acceleration around the X-, Y-, and Z-axes. The motion detection device 25 can generate posture information indicating the posture of the MR glasses 20 based on the output information of the gyro sensor. The motion detection device 25 supplies orientation information relating to the orientation of the MR glasses 20 to the processing device 21 . In addition, the motion detection device 25 supplies motion information relating to the motion of the MR glasses 20 to the processing device 21 . The motion information includes acceleration data indicating three-axis acceleration and angular acceleration data indicating three-axis angular acceleration. The posture information and motion information supplied to the processing device 21 are transmitted to the terminal device 10 via the communication device 27 .

The imaging device 26 outputs imaging information obtained by imaging the outside world. Also, the imaging device 26 includes, for example, a lens, an imaging element, an amplifier, and an AD converter. The light condensed through the lens is converted into an image pickup signal, which is an analog signal, by the image pickup device. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is supplied to the processing device 21 . The imaging information supplied to the processing device 21 is transmitted to the terminal device 10 via the communication device 27 .

The communication device 27 is hardware as a transmission/reception device for communicating with other devices. The communication device 27 is also called a network device, a network controller, a network card, a communication module, etc., for example. The communication device 27 may include a connector for wired connection and an interface circuit corresponding to the connector. Further, the communication device 27 may have a wireless communication interface. Products conforming to wired LAN, IEEE1394, and USB are examples of connectors and interface circuits for wired connection. Also, as a wireless communication interface, there are products conforming to wireless LAN, Bluetooth (registered trademark), and the like.

The display 28 is a device that displays images. The display 28 displays various images under the control of the processing device 21 . The display 28 includes the lens 41L, the left-eye display panel, the left-eye optical member, and the lens 41R, the right-eye display panel, and the right-eye optical member, as described above. Various display panels such as a liquid crystal display panel and an organic EL display panel are preferably used as the display panel.

The speaker 29 is a device that emits sound. The speaker 29 emits various sounds under the control of the processing device 21 . For example, audio data, which is a digital signal, is converted into an audio signal, which is an analog signal, by a DA converter (not shown). The audio signal is amplified in amplitude by an amplifier (not shown). The speaker 29 emits sound indicated by the audio signal after the amplitude is amplified.

The processing device 21 functions as an acquisition unit 211 and a display control unit 212, for example, by reading the control program PR1 from the storage device 22 and executing it.

The acquisition unit 211 acquires image information indicating an image displayed on the MR glasses 20 from the terminal device 10 . More specifically, the acquisition unit 211 acquires, for example, second image information, which is transmitted from the terminal device 10 and will be described later.

The acquisition unit 211 also receives line-of-sight information input from the line-of-sight detection device 23 , position information input from the GPS device 24 , posture information and motion information input from the motion detection device 25 , and input from the imaging device 26 . Acquire imaging information. After that, the acquisition unit 211 outputs the acquired line-of-sight information, position information, posture information, motion information, and imaging information to the communication device 27 .

The display control unit 212 causes the display 28 to display an image indicated by the second image information based on the second image information acquired from the terminal device 10 by the acquisition unit 211 .

1-1-3: Configuration of Terminal Device FIG. 6 is a block diagram showing a configuration example of the terminal device 10. As shown in FIG. The terminal device 10 includes a processing device 11 , a storage device 12 , a communication device 13 , a display 14 , an input device 15 and an inertial sensor 16 . Elements of the terminal device 10 are interconnected by one or more buses for communicating information.

The processing device 11 is a processor that controls the terminal device 10 as a whole. Also, the processing device 11 is configured using, for example, a single chip or a plurality of chips. The processing unit 11 is configured using, for example, a central processing unit (CPU) including interfaces with peripheral devices, arithmetic units, registers, and the like. A part or all of the functions of the processing device 11 may be implemented by hardware such as DSP, ASIC, PLD, and FPGA. The processing device 11 executes various processes in parallel or sequentially.

The storage device 12 is a recording medium readable and writable by the processing device 11 . The storage device 12 also stores a plurality of programs including the control program PR2 executed by the processing device 11 .

The communication device 13 is hardware as a transmission/reception device for communicating with other devices. The communication device 13 is also called a network device, a network controller, a network card, a communication module, etc., for example. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. Further, the communication device 13 may have a wireless communication interface. Products conforming to wired LAN, IEEE1394, and USB are examples of connectors and interface circuits for wired connection. Also, as a wireless communication interface, there are products conforming to wireless LAN, Bluetooth (registered trademark), and the like.

The display 14 is a device that displays images and character information. The display 14 displays various images under the control of the processing device 11 . For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are preferably used as the display 14 .

The input device 15 accepts operations from the user U1 who wears the MR glasses 20 on his head. For example, the input device 15 includes a pointing device such as a keyboard, touch pad, touch panel, or mouse. Here, when the input device 15 includes a touch panel, the input device 15 may also serve as the display 14 .

The inertial sensor 16 is a sensor that detects inertial force. The inertial sensor 16 includes, for example, one or more of an acceleration sensor, an angular velocity sensor, and a gyro sensor. The processing device 11 detects the orientation of the terminal device 10 based on the output information from the inertial sensor 16 . Further, the processing device 11 receives selection of the virtual object VO, input of characters, and input of instructions in the celestial sphere virtual space VS based on the orientation of the terminal device 10 . For example, the user U1 directs the central axis of the terminal device 10 toward a predetermined area of the virtual space VS, and operates the input device 15 to select the virtual object VO arranged in the predetermined area. The user U1's operation on the input device 15 is, for example, a double tap. By operating the terminal device 10 in this way, the user U1 can select the virtual object VO without looking at the input device 15 of the terminal device 10 .

The processing device 11 functions as an acquisition unit 111, a display control unit 112, a determination unit 113, and an operation control unit 114 by reading the control program PR2 from the storage device 12 and executing it.

The acquisition unit 111 acquires line-of-sight information indicating the line of sight of the user U1. More specifically, the acquisition unit 111 acquires line-of-sight information output from the MR glasses 20 via the communication device 13 . In addition, the acquisition unit 111 acquires first image information, which is described below and indicates an image displayed on the MR glasses 20 , from the server 30 via the communication device 13 .

Based on the first image information acquired by the acquisition unit 111, the display control unit 112 moves the first virtual object VO1 and the second virtual object VO2 to positions not overlapping each other in the virtual space VS as seen from the user U1. display. Specifically, the first image information is image information indicating images of the first virtual object VO1 itself and the second virtual object VO2 itself. In addition, the display control unit 112 provides layout information such that the first virtual object VO1 and the second virtual object VO2 do not overlap with each other from the viewpoint of the user U1 in the virtual space VS viewed by the user U1 through the MR glasses 20, and Second image information including the first image information is transmitted to the MR glasses 20 via the communication device 13 .

FIG. 7 is an explanatory diagram of how the display control unit 112 displays the first virtual object VO1 and the second virtual object VO2. In the following description, it is assumed that X, Y and Z axes are orthogonal to each other in the virtual space VS. As an example, the X-axis extends in the horizontal direction of user U1. Further, as viewed from the user U1, the right direction along the X axis is the positive direction, and the left direction along the X axis is the negative direction. Also, the Y-axis extends in the front-rear direction of the user U1. Further, the forward direction along the Y-axis is defined as the positive direction, and the backward direction along the Y-axis is defined as the negative direction, as viewed from the user U1. A horizontal plane is formed by these X-axis and Y-axis. Also, the Z-axis is orthogonal to the XY plane and extends in the vertical direction of the user U1. Further, as viewed from the user U1, the upward direction along the Z axis is the positive direction, and the downward direction along the Z axis is the negative direction.

In the virtual space VS, the position of the pupil EL of the left eyeball or the pupil ER of the right eyeball of the user U1, that is, the coordinates of one end point of the line of sight V of the user U1 are (x, y, z)=(x0, y0, z0). Also, the display control unit 112 displays the first virtual object VO1 at the position of (x, y, z)=(x1, y1, z1).

Here, assume a cylinder Y as a region where the user U1's line of sight V can exist while the user U1 is viewing the first virtual object VO1. The cylinder Y has a first end point C1 at the position (x0, y0, z0) of the pupil EL of the left eyeball or the pupil ER of the right eyeball of the user U1, and the position (x1, y1, z1) of the first virtual object VO1 as It has a central axis L1 with a second end point C2. Furthermore, the radius R1 of the bottom surface of the cylinder Y is the length that allows the bottom surface to contain the first virtual object VO1. In FIG. 7, for convenience of explanation, the position of the pupil EL of the left eyeball or the pupil ER of the right eyeball of the user U1 does not match the position of the first end point C1. Similarly, in FIG. 7, the position (x1, y1, z1) of the first virtual object VO1 does not match the position of the second end point C2. However, for the purposes of the present invention, each position in these sets will match.

The display control unit 112 displays the second virtual object VO2 outside the cylinder Y in an inactive state in the virtual space VS. Specifically, the display position (x, y, z)=(x2, y2, z2) of the second virtual object VO2 is a position where the entire second virtual object VO2 is outside the range of the cylinder Y. becomes. Here, "active" means that the virtual object VO is in operation, or that the virtual object VO is selected by the user U1. On the other hand, "inactive" means that the motion of the virtual object VO has stopped, or that the virtual object VO has not been selected by the user U1. As an example, when the virtual object VO corresponds to a moving image application, "the virtual object VO is in operation" means that the moving image is being reproduced by the user U1's operation. Further, when the virtual object VO corresponds to an e-mail application, "the virtual object VO is in operation" means that the e-mail is sent/received or a text file indicating the content of the e-mail is opened by the operation of the user U1. It means that you are being killed. Further, when the virtual object VO corresponds to a music application, "the virtual object VO is in operation" means that the music is emitted from the speaker 29 by the operation of the user U1.

The display control unit 112 displays the second virtual object VO2 in the virtual space VS in the first state in which the line of sight V of the user U1 intersects the first virtual object VO1 and the first virtual object VO1 is active. Let Specifically, the display control unit 112 displays the entire second virtual object VO2 in a first state in which the entire line of sight V of the user U1 is positioned inside the cylinder Y and the first virtual object VO1 is active. It is displayed outside the cylinder Y.

The display position of the second virtual object VO2 is an area within a predetermined distance from the first virtual object VO1. Specifically, the second virtual object VO2 is displayed within the field of view of the user U1 while the user U1 is viewing the first virtual object VO1. With this process, the terminal device 10 can call the attention of the user U1 to the second virtual object VO2. The above "predetermined distance" is an example of the "first distance".

Returning to FIG. 6, the determination unit 113 determines that the first state in which the line of sight V of the user U1 intersects the first virtual object VO1 and the first virtual object VO1 is active is the line of sight V of the user U1. has transitioned to a second state that does not intersect with the first virtual object VO1.
After the determination unit 113 determines that the first state has transitioned to the second state, the motion control unit 114 transitions the second virtual object VO2 from inactive to active. Specifically, in FIG. 7, after the line of sight V of the user U1 changes from being within the cylinder Y to being outside the cylinder Y, the motion control unit 114 changes the second virtual object VO2 from inactive to active. transition to

In the second state, the motion control unit 114 preferably transitions the second virtual object VO2 from inactive to active after the line of sight V of the user U1 remains stationary for a predetermined period of time. Specifically, in the second state described above, the motion control unit 114 controls the second virtual object VO2 on the condition that the change in the line of sight V of the user U1 per unit time is within a predetermined range for a predetermined period of time. transition from inactive to active. As a result, the terminal device 10 can transition the second virtual object VO2 from inactive to active after the point of the line of sight V of the user U1 is not wandering and the state is almost stationary. The above "predetermined range" is an example of the "second range". Also, the above "predetermined time" is an example of the "second time".

Also, the operation control unit 114 transitions the first virtual object VO1 from active to inactive after transitioning from the first state to the second state. With this process, the terminal device 10 can switch which virtual object VO to activate, the first virtual object VO1 or the second virtual object VO2, according to the position of the line of sight V of the user U1. Also, the terminal device 10 can exclusively activate the first virtual object VO1 and the second virtual object VO2. The state transition of the first virtual object VO1 and the state transition of the second virtual object VO2 may be executed simultaneously, or one may precede the other.

1-1-4: Server Configuration FIG. 8 is a block diagram showing a configuration example of the server 30. As shown in FIG. The server 30 comprises a processing device 31 , a storage device 32 , a communication device 33 , a display 34 and an input device 35 . Each element of server 30 is interconnected by one or more buses for communicating information.

The processing device 31 is a processor that controls the server 30 as a whole. Also, the processing device 31 is configured using, for example, a single chip or a plurality of chips. The processing unit 31 is configured using, for example, a central processing unit (CPU) including interfaces with peripheral devices, arithmetic units, registers, and the like. A part or all of the functions of the processing device 31 may be implemented by hardware such as DSP, ASIC, PLD, and FPGA. The processing device 31 executes various processes in parallel or sequentially.

The storage device 32 is a recording medium readable and writable by the processing device 31 . The storage device 32 also stores a plurality of programs including the control program PR3 executed by the processing device 31 .

The communication device 33 is hardware as a transmission/reception device for communicating with other devices. The communication device 33 is also called a network device, a network controller, a network card, a communication module, or the like, for example. The communication device 33 may include a connector for wired connection and an interface circuit corresponding to the connector. Further, the communication device 33 may have a wireless communication interface. Products conforming to wired LAN, IEEE1394, and USB are examples of connectors and interface circuits for wired connection. Also, as a wireless communication interface, there are products conforming to wireless LAN, Bluetooth (registered trademark), and the like.

The display 34 is a device that displays images and character information. The display 34 displays various images under the control of the processing device 31 . For example, various display panels such as a liquid crystal display panel and an organic EL display panel are preferably used as the display 34 .

The input device 35 is a device that accepts operations by the administrator of the information processing system 1 . For example, the input device 35 includes a pointing device such as a keyboard, touch pad, touch panel, or mouse. Here, when the input device 35 includes a touch panel, the input device 35 may also serve as the display 34 .

The processing device 31 functions as an acquisition unit 311 and a generation unit 312, for example, by reading the control program PR3 from the storage device 32 and executing it.

The acquisition unit 311 acquires various data from the terminal device 10 using the communication device 33 . The data includes, for example, data indicating the operation content for the virtual object VO, which is input to the terminal device 10 by the user U1 wearing the MR glasses 20 on the head.

The generation unit 312 generates first image information indicating an image displayed on the MR glasses 20 . This first image information is transmitted to the terminal device 10 via the communication device 33 . As described above, the display control unit 112 provided in the terminal device 10 displays the first virtual object VO1 and the second virtual object VO2 as seen from the user U1 based on the first image information received via the communication device 13. are displayed at positions that do not overlap each other in the virtual space VS.

1-2: Operation of the First Embodiment FIG. 9 is a flow chart showing the operation of the terminal device 10 according to the first embodiment. The operation of the terminal device 10 will be described below with reference to FIG.

In step S<b>1 , the processing device 11 functions as the display control unit 112 . The processing device 11 displays the first virtual object VO1 in the virtual space VS.

In step S2, the processing device 11 functions as the acquisition unit 111. The processing device 11 acquires the line of sight V of the user U1.

In step S3, the processing device 11 functions as the display control unit 112. The processing device 11 displays the second virtual object VO2 in an inactive state so that it does not overlap the first virtual object VO1 as seen from the user U1 in the virtual space VS. It is assumed that the first virtual object VO1 is active at the stage of step S3 at the latest.

In step S4, the processing device 11 functions as the determination unit 113. The processing device 11 changes the line-of-sight V of the user U1 to intersect the first virtual object VO1 from the first state in which the line-of-sight V of the user U1 intersects the first virtual object VO1 and the first virtual object VO1 is active. It is determined whether or not the state has changed to the second state in which the state is not performed. If it is determined that the state has transitioned from the first state to the second state, that is, if the determination result of step S4 is affirmative, the processing device 11 executes the process of step S5. If it is not determined that the state has transitioned from the first state to the second state, that is, if the determination result of step S4 is negative, the processing device 11 executes the process of step S4.

In step S5, the processing device 11 functions as the operation control unit 114. The processing device 11 transitions the second virtual object VO2 from inactive to active.

In step S6, the processing device 11 functions as the operation control unit 114. The processing device 11 transitions the first virtual object VO1 from active to inactive. After that, the processing device 11 ends all the operations shown in FIG.

1-3: Effects of the First Embodiment According to the above description, the terminal device 10 as an information processing device includes the acquisition unit 111, the display control unit 112, and the operation control unit 114. FIG. The acquisition unit 111 acquires line-of-sight information indicating the line-of-sight V of the user U1. The display control unit 112 causes the first virtual object VO1 and the second virtual object VO2 to be displayed at positions not overlapping each other in the virtual space VS as seen from the user U1. The motion control unit 114 changes the line-of-sight V of the user U1 to the first virtual object VO1 from the first state in which the line-of-sight V of the user U1 indicated by the line-of-sight information intersects with the first virtual object VO1 and the first virtual object VO1 is active. After transitioning to a second state that does not intersect with the virtual object VO1, the second virtual object VO2 is transitioned from inactive to active.

By using the above configuration, the terminal device 10 can newly display the second virtual object VO2 so as not to overlap the previously displayed first virtual object VO1. In addition, the terminal device 10 can switch which virtual object VO to activate between the first virtual object VO1 and the second virtual object VO2 according to the direction of the line of sight V of the user U1.

Also, according to the above description, the operation control unit 114 transitions the first virtual object VO1 from active to inactive after transitioning from the first state to the second state.

By using the above configuration, the terminal device 10 can exclusively activate the first virtual object VO1 and the second virtual object VO2. For example, if the first virtual object VO1 is the first application and the second virtual object VO2 is the second application, the user U1 removes the line of sight V from the first virtual object VO1, and the first application is executed. You can stop working and run a second application. That is, the user U1 can switch the application to be operated only by removing the line of sight V from the first virtual object VO1 without inputting an instruction to the second virtual object VO2.

According to the above description, the display control unit 112 starts displaying the second virtual object VO2 in the virtual space VS in the first state.

By using the above configuration, the terminal device 10 can cause the second virtual object VO2 to appear in the first state. For example, the terminal device 10 can display a second application, which is the second virtual object VO2, while the first application, which is the first virtual object VO1, is running.

Further, according to the above description, the operation control unit 114, in the above-described second state, on the condition that the change per unit time of the line of sight V of the user U1 is within the second range continues for the second time. , transitions the second virtual object VO2 from inactive to active.

By using the above configuration, the terminal device 10 can transition the second virtual object VO2 from inactive to active after the point of the line of sight V of the user U1 is not wandering and the second virtual object VO2 is in a near stationary state.

Also, according to the above description, the display control unit 112 displays the second virtual object VO2 in the area within the first distance from the first virtual object VO1.

By using the above configuration, the terminal device 10 can increase the degree to which the user U1's attention can be drawn to the second virtual object VO2.

Further, according to the above description, the display control unit 112, in the virtual space VS, displays a second 2 Display the virtual object VO2.

By using the above configuration, the terminal device 10 can display the first virtual object VO1 and the second virtual object VO2 at positions that do not overlap each other in the virtual space VS as seen from the user U1.

2: Modifications The present disclosure is not limited to the embodiments illustrated above. Specific modification modes are exemplified below. Two or more aspects arbitrarily selected from the following examples may be combined.

2-1: Modification 1
In the terminal device 10 according to the embodiment described above, the display control unit 112 displays the second virtual object VO2 in the virtual space VS in the first state described above. However, the method of displaying the second virtual object VO2 is not limited to this method. For example, the display control unit 112 may display the second virtual object VO2 at a position that intersects the line of sight V of the user U1 after the line of sight V of the user U1 remains stationary for a predetermined time in the second state. Specifically, in the second state, the display control unit 112 sets the second virtual object VO2 to the second virtual object VO2 on the condition that the change per unit time of the line of sight V of the user U1 continues within a predetermined range for a predetermined time. may be displayed at a position intersecting with the line of sight V of the user U1. Since the second virtual object VO2 is displayed ahead of the line of sight V of the user U1, the user U1 does not need to search for the second virtual object VO2 in the virtual space VS. The above "predetermined time" is an example of the "first time". Also, the above "predetermined range" is an example of the "first range".

2-2: Modification 2
The terminal device 10 according to the above embodiment includes the display control section 112 and the operation control section 114 . In addition, the display control unit 112 causes the first virtual object VO1 and the second virtual object VO2 to be displayed at positions that do not overlap each other in the virtual space VS as seen from the user U1. Further, the motion control unit 114 changes the line-of-sight V of the user U1 from the first state in which the line-of-sight V of the user U1 indicated by the line-of-sight information and the first virtual object VO1 intersect and the first virtual object VO1 is active. After transitioning to a second state that does not intersect with one virtual object VO1, the second virtual object VO2 is transitioned from inactive to active. However, these operations may be performed by a device other than the terminal device 10. FIG. For example, the server 30 may include components similar to the display control section 112 and the operation control section 114 . In particular, in a mode in which the communication speed between the terminal device 10 and the server 30 is high and the amount of data transmitted from the server 30 to the terminal device 10 is large compared to a normal information processing system, the server 30 It is preferred to perform an action. In Modified Example 2, the server 30 is an example of an information processing device.

2-3: Modification 3
In the terminal device 10 according to the above-described embodiment, the cylinder Y was assumed as the region where the line of sight V of the user U1 may exist while the user U1 is viewing the first virtual object VO1. However, the shape of the area is not limited to a cylinder.

FIG. 10 is an explanatory diagram of an operation example of the display control unit 112 according to the third modification. The shape of said region may be, for example, a cone N, as shown in FIG. The cone N has the position (x0, y0, z0) of the pupil EL of the left eyeball or the pupil ER of the right eyeball of the user U1 as the first end point C1, and the position (x1, y1, z1) of the first virtual object VO1 as It has a central axis L2 with a second end point C2. The first end point C1 is the vertex of the cone N. Furthermore, the radius R2 of the base of the cone N is the length that allows the base to contain the first virtual object VO1. In FIG. 10, for convenience of explanation, the position of the pupil EL of the left eyeball or the pupil ER of the right eyeball of the user U1 does not match the position of the first end point C1. Similarly, in FIG. 10, the position (x1, y1, z1) of the first virtual object VO1 does not match the position of the second end point C2. However, for the purposes of the present invention, each position in these sets will match.

2-4: Modification 4
In the information processing system 1 according to the above-described embodiment, the terminal device 10 and the MR glasses 20 are implemented separately. However, the method of realizing the terminal device 10 and the MR glasses 20 in the embodiment of the present invention is not limited to this. For example, the terminal device 10 and the MR glasses 20 may be realized within a single housing by providing the MR glasses 20 with the same functions as the terminal device 10 .

2-5: Modification 5
The information processing system 1 according to the above embodiment includes the MR glasses 20 . However, in the information processing system 1, instead of the MR glasses 20, an HMD (Head Mounted Display) employing VR (Virtual Reality) technology, an HMD employing AR (Augmented Reality) technology, and an AR technology are employed. Any one of the AR glasses may be provided. Alternatively, the information processing system 1 may include, instead of the MR glasses 20, any one of a normal smartphone and tablet equipped with an imaging device. These HMDs, AR glasses, smartphones, and tablets are examples of display devices.

2-6: Modification 6
In the information processing system 1 according to the above embodiment, the generation unit 312 provided in the server 30 generates the first image information indicating the image displayed on the MR glasses 20 . However, the device that generates the first image information is not limited to the server 30 . For example, the terminal device 10 may generate the above first image information. In this case, the information processing system 1 does not have to include the server 30 as an essential component.

3: Others (1) In the above-described embodiment, the storage device 12, the storage device 22, and the storage device 32 are ROM and RAM. discs, Blu-ray discs), smart cards, flash memory devices (e.g. cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable discs, hard disks, floppies ) disk, magnetic strip, database, server or other suitable storage medium. Also, the program may be transmitted from a network via an electric communication line. Also, the program may be transmitted from the communication network NET via an electric communication line.

(2) In the embodiments described above, the information, signals, etc. described may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may refer to voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. may be represented by a combination of

(3) In the above-described embodiments, input/output information and the like may be stored in a specific location (for example, memory), or may be managed using a management table. Input/output information and the like can be overwritten, updated, or appended. The output information and the like may be deleted. The entered information and the like may be transmitted to another device.

(4) In the above-described embodiment, the determination may be made by a value (0 or 1) represented using 1 bit, or by a true/false value (Boolean: true or false). Alternatively, it may be performed by numerical comparison (for example, comparison with a predetermined value).

(5) The order of the processing procedures, sequences, flowcharts, etc. exemplified in the above embodiments may be changed as long as there is no contradiction. For example, the methods described in this disclosure present elements of the various steps using a sample order, and are not limited to the specific order presented.

(6) Each function illustrated in FIGS. 1 to 10 is realized by any combination of at least one of hardware and software. Also, the method of realizing each functional block is not particularly limited. That is, each functional block may be implemented using one device physically or logically coupled, or directly or indirectly using two or more physically or logically separated devices (e.g. , wired, wireless, etc.) and may be implemented using these multiple devices. A functional block may be implemented by combining software in the one device or the plurality of devices.

(7) The programs illustrated in the above embodiments, whether referred to as software, firmware, middleware, microcode, hardware description language or by other names, instructions, instruction sets, code, code shall be interpreted broadly to mean segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, and the like.

In addition, software, instructions, information, etc. may be transmitted and received via a transmission medium. For example, the software uses at least one of wired technology (coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), etc.) and wireless technology (infrared, microwave, etc.) to website, Wired and/or wireless technologies are included within the definition of transmission medium when sent from a server or other remote source.

(8) In each of the above aspects, the terms "system" and "network" are used interchangeably.

(9) Information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values from a predetermined value, or may be expressed using corresponding other information. may be represented as

(10) In the above-described embodiments, the terminal device 10 and the server 30 may be mobile stations (MS). A mobile station is defined by those skilled in the art as subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless It may also be called a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term. Also, in the present disclosure, terms such as "mobile station", "user terminal", "user equipment (UE)", "terminal", etc. may be used interchangeably.

(11) In the above-described embodiments, the terms "connected," "coupled," or any variation thereof refer to any direct or indirect connection between two or more elements. Any connection or coupling is meant, including the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. Couplings or connections between elements may be physical couplings or connections, logical couplings or connections, or a combination thereof. For example, "connection" may be replaced with "access." As used in this disclosure, two elements are defined using at least one of one or more wires, cables, and printed electrical connections and, as some non-limiting and non-exhaustive examples, in the radio frequency domain. , electromagnetic energy having wavelengths in the microwave and optical (both visible and invisible) regions, and the like.

(12) In the above-described embodiments, the phrase "based on" does not mean "based only on," unless expressly specified otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

(13) The terms "determining" and "determining" as used in this disclosure may encompass a wide variety of actions. "Judgement" and "determination" are, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (eg, lookup in a table, database, or other data structure); Also, "judgment" and "determination" are used for receiving (e.g., receiving information), transmitting (e.g., transmitting information), input, output, access (accessing) (for example, accessing data in memory) may include deeming that a "judgement" or "decision" has been made. In addition, "judgment" and "decision" are considered to be "judgment" and "decision" by resolving, selecting, choosing, establishing, comparing, etc. can contain. In other words, "judgment" and "decision" may include considering that some action is "judgment" and "decision". Also, "judgment (decision)" may be read as "assuming", "expecting", "considering", and the like.

(14) In the above-described embodiments, where "include," "including," and variations thereof are used, these terms are synonymous with the term "comprising." , is intended to be inclusive. Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive OR.

(15) In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure includes the plural nouns following these articles. good.

(16) In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean that "A and B are different from C". Terms such as "separate," "coupled," etc. may also be interpreted in the same manner as "different."

(17) Each aspect/embodiment described in the present disclosure may be used alone, may be used in combination, or may be used by switching according to execution. In addition, notification of predetermined information (for example, notification of “being X”) is not limited to explicit notification, but is performed implicitly (for example, not notification of the predetermined information). good too.

Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in this disclosure. The present disclosure can be practiced with modifications and variations without departing from the spirit and scope of the present disclosure as defined by the claims. Accordingly, the description of the present disclosure is for illustrative purposes and is not meant to be limiting in any way on the present disclosure.

REFERENCE SIGNS LIST 1 information processing system 10 terminal device 11 processing device 12 storage device 13 communication device 14 display 15 input device 16 inertial sensor 20 MR glass 21 processing device 22... Storage device 23... Line of sight detection device 24... GPS device 25... Motion detection device 26... Imaging device 27... Communication device 28... Display 30... Server 31... Processing device 32... Storage device 33... Communication device, 34... Display, 35... Input device, 41L, 41R... Lens, 91, 92... Temple, 93... Bridge, 94, 95... Frame, 111... Acquisition unit, 112... Display control unit, 113... Judgment Unit 114 Operation control unit 211 Acquisition unit 212 Display control unit 311 Acquisition unit 312 Generation unit C1, C2 End point L1, L2 Central axis PR1, PR2, PR3 Control program , R1, R2... radius, U1, U2, U3... user, VO... virtual object, VO1... first virtual object, VO2... second virtual object

Claims

an acquisition unit that acquires line-of-sight information indicating the user's line of sight;
a display control unit that displays the first virtual object and the second virtual object at positions that do not overlap each other in the virtual space as seen from the user;
A first state in which the line of sight of the user indicated by the line of sight information intersects the first virtual object and the first virtual object is active is a state in which the line of sight of the user does not intersect the first virtual object. an operation control unit that transitions the second virtual object from inactive to active after transitioning to two states;
Information processing device.
The information processing apparatus according to claim 1, wherein the action control unit transitions the first virtual object from active to inactive after the first state transitions to the second state.
The information processing apparatus according to claim 1 or 2, wherein said display control unit starts displaying said second virtual object in said virtual space in said first state.
In the second state, the display control unit controls the second virtual object to be displayed by the user on condition that a change in the line of sight of the user per unit time is within a first range for a first period of time. 3. The information processing apparatus according to claim 1, wherein the display is started at a position intersecting the line of sight of the .
In the second state, the motion control unit changes the second virtual object from being inactive on condition that a change in the line of sight of the user per unit time is within a second range for a second period of time. 4. The information processing apparatus according to any one of claims 1 to 3, configured to transition to active.
The information processing apparatus according to any one of claims 1 to 5, wherein the display control unit displays the second virtual object in a region within a first distance from the first virtual object.
wherein, in the virtual space, the display control unit displays the second virtual object outside a cylinder or cone having a central axis that matches the line of sight of the user and a bottom surface that includes the first virtual object. The information processing apparatus according to any one of claims 1 to 6.