WO2024075565A1

WO2024075565A1 - Display control device and display control method

Info

Publication number: WO2024075565A1
Application number: PCT/JP2023/034684
Authority: WO
Inventors: 毅石川; 京二郎永野; 洋史湯浅; 郁男山野
Original assignee: ソニーグループ株式会社
Priority date: 2022-10-04
Filing date: 2023-09-25
Publication date: 2024-04-11

Abstract

A display control device in one embodiment according to the present disclosure comprises: an acquisition unit that acquires, from a first controller, position/attitude information in a three-dimensional space; and a display control unit which, on the basis of the position/attitude information in the three-dimensional space, displays, in the three-dimensional space, a virtual coordinate plane on which two-dimensional coordinates can be designated. Moreover, the display control device may further comprise a first detection unit that, on the basis of input by means of the first controller, detects movement of the virtual coordinate plane in the three-dimensional space.

Description

Display control device and display control method

This disclosure relates to a display control device and a display control method for controlling operations on virtual objects.

With the development of video display technology, it is now possible to use devices for VR (Virtual Reality) and AR (Augmented Reality) to create virtual spaces, overlay content from virtual spaces onto real space, and view virtual objects in 3D as if they were real objects.

In relation to this technology, many UIs (User Interfaces) have been proposed that enable users to create and edit 3D models corresponding to virtual objects while viewing them in stereoscopic vision, by holding a controller to move the model in the air, or to place and move points and lines in the air (6DoF (Degree of Freedom) operations). As an example, a technology has been proposed that improves user operability by devising a mechanism for assigning a physical controller to a virtual controller (for example, Patent Document 1).

Patent No. 6859422

However, there is room for further improvement in the operations for selecting and editing virtual objects in 3D space.

For example, when a user operates a controller by lifting their hand in the air, the physical strain is greater than when using a conventional tabletop controller (such as a mouse), which can lead to problems such as hand tremors and reduced input accuracy. For this reason, controllers for air operation, which are primarily used in VR and AR, are currently difficult to use for creating or editing 3D models in applications that require high input accuracy (such as CAD).

Therefore, this disclosure proposes a display control device and a display control method that can improve input accuracy.

In order to solve the above problem, a display control device according to one embodiment of the present disclosure includes an acquisition unit that acquires position and orientation information in a three-dimensional space from a first controller, and a display control unit that displays a virtual coordinate plane in the three-dimensional space on which two-dimensional coordinates can be specified based on the position and orientation information in the three-dimensional space.

FIG. 2 is a diagram showing an overview of a display control process according to the embodiment. FIG. 1 is a diagram showing an example of a display control process according to an embodiment; FIG. 13 is a diagram showing an example of the display control process according to the embodiment; FIG. 1 is a diagram illustrating an example of the configuration of a display control device according to an embodiment. FIG. 4 is a sequence diagram showing a flow of a display control process according to the embodiment. FIG. 11 is a diagram for explaining a device operation according to a first modified example. 13 is a flowchart showing the flow of a display control process according to a first modified example. FIG. 11 is a diagram for explaining a display control process according to a second modified example. FIG. 11 is a diagram (2) for explaining the display control process according to the second modified example. FIG. 11 is a diagram (3) for explaining the display control process according to the second modified example. FIG. 11 is a diagram for explaining a display control process according to a third modified example. FIG. 13 is a diagram (2) for explaining the display control process according to the third modified example. FIG. 13 is a diagram for explaining the display control process according to the third modified example. FIG. 4 is a diagram for explaining the display control process according to the third modified example. FIG. 5 is a diagram for explaining a display control process according to a third modified example. FIG. 6 is a diagram for explaining the display control process according to the third modified example. FIG. 7 is a diagram for explaining a display control process according to the third modified example. FIG. 8 is a diagram for explaining a display control process according to a third modified example. FIG. 13 is a diagram for explaining a display control process according to a fourth modified example. FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the display control device.

The following describes the embodiments in detail with reference to the drawings. Note that in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate descriptions will be omitted.

The present disclosure will be described in the following order.
1. Embodiment 1-1. Overview of display control processing according to embodiment 1-2. Configuration of display control device according to embodiment 1-3. Processing procedure according to embodiment 1-4. Modifications 1-4-1. First modification: Operation of only one device 1-4-2. Second modification: Adjustment of operation feel 1-4-3. Third modification: Processing of superimposition on real space 1-4-4. Fourth modification: Deformation of surface based on pointer position 2. Other embodiments 3. Effects of display control device according to the present disclosure 4. Hardware configuration

1. EMBODIMENTS
(1-1. Overview of Display Control Process According to the Embodiment)
An example of a display control process according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an overview of the display control process according to the embodiment. Fig. 1 shows components of a display control system 1 that executes the display control process according to the embodiment.

As shown in FIG. 1, the display control system 1 includes a display control device 100, a three-dimensional device 20, and a planar device 30.

The display control device 100 is an information processing terminal for implementing VR and AR technologies. In an embodiment, the display control device 100 is a wearable display that is worn on the head of a user 10. Specifically, the display control device 100 is, for example, a head mounted display (HMD) or AR glasses.

The display control device 100 has a closed or transmissive display unit (display). For example, the display control device 100 displays a virtual space expressed by CG (Computer Graphics) or the like. Alternatively, the display control device 100 displays a virtual object expressed by CG or the like on the display unit by superimposing it on real space. In the example of FIG. 1, the display control device 100 displays an object 70 as an example of a virtual object. That is, in the example of FIG. 1, the object 70 and the space viewed by the user 10 are assumed to be a virtual space configured by CG or the like. Note that, as will be described later, the space viewed by the user 10 may be a so-called AR space in which a virtual object is superimposed on real space. In addition, the display control device 100 may have a configuration for outputting a predetermined output signal in addition to the display unit. For example, the display control device 100 may have a speaker or the like for outputting sound.

The 3D device 20 is an example of an input device according to an embodiment. In the embodiment, the 3D device 20 is operated by the user 10 and is used to input various information to the display control device 100. The 3D device 20 is a VR controller capable of inputting so-called 6DoF information. Specifically, the 3D device 20 is equipped with sensors such as an inertial sensor, an acceleration sensor, and a gravity sensor, and detects position and orientation information of the device itself. The 3D device 20 then transmits the detected position and orientation information of the device itself to the display control device 100.

The three-dimensional device 20 may be a controller held by the user 10 in one hand as shown in FIG. 1, or may be a pen-shaped controller. The three-dimensional device 20 is not limited to the example, and may be any device capable of acquiring position information in real space. For example, the three-dimensional device 20 may be an air mouse, a digital camera, a smartphone, or the like. In addition, if the display control device 100 can capture the position and orientation information of the three-dimensional device 20, the three-dimensional device 20 does not need to have a sensor of its own. For example, the three-dimensional device 20 may be a specified object, a human face, a finger, or the like, equipped with a marker that can be recognized by the display control device 100 or a specified external device (such as a video camera installed in real space).

The planar device 30 is an example of an input device according to an embodiment. In the embodiment, the planar device 30 is operated by the user 10 and is used to input various information to the display control device 100. The planar device 30 is, for example, a mouse controller or a trackball mouse that can input two-dimensional coordinate information. Specifically, the planar device 30 is equipped with a sensor such as an optical sensor, and detects coordinate information (two-dimensional information) on a plane based on the operation of the planar device. The planar device 30 then transmits the detected coordinate information to the display control device 100.

Here, we will explain the process when the user 10 specifies an arbitrary position of an object in a VR or AR space. Conventionally, a controller such as the three-dimensional device 20 has been used to select a position in a VR or AR space. However, operating a controller in the air places a greater physical strain than operating a controller on a conventional table, which can lead to problems such as hand tremors and reduced input accuracy.

As a solution to these problems, technology has been proposed that allows the use of highly accurate devices such as a mouse in three-dimensional space. However, planar devices such as a mouse are generally used to input coordinate information on a specific plane, making it difficult to point to any position in three-dimensional space. In response to this, processing has been explored that uses a mouse in combination with a controller that can input 6DoF information, such as a VR controller. However, operating two different pointing devices simultaneously would break down the conventional UI that was operated with a single pointer, so these multiple devices are usually implemented to operate exclusively. For this reason, technology that improves accuracy and operability when pointing in three-dimensional space has been desired.

The display control device 100 according to the present disclosure solves this problem by the following configuration. That is, the display control device 100 acquires position and orientation information in three-dimensional space from the first controller, and displays a virtual coordinate plane in three-dimensional space on which two-dimensional coordinates can be specified based on the position and orientation information in the three-dimensional space. Furthermore, the display control device 100 detects coordinates on the virtual coordinate plane from the second controller, thereby enabling the user 10 to accurately select a desired position in space. In this case, the first controller is a three-dimensional device 20 that the user 10 holds in one hand. Also, the second controller is a two-dimensional device 30 that the user 10 holds in the hand other than the hand holding the first controller (e.g., the dominant hand of the user 10).

In other words, the display control device 100 combines two different devices to enable the user 10 to select any position. Specifically, the display control device 100 displays a virtual coordinate plane (plane 50 shown in FIG. 1) in space, and uses the three-dimensional device 20 as an input means for moving the plane 50. The display control device 100 also displays a pointer 60 on the plane 50, and uses the planar device 30 as an input means for moving the pointer 60 two-dimensionally. This allows the user 10 to perform highly accurate pointing operations in virtual space by inputting using the planar device 30, which can specify the precise position of the pointer 60, after displaying the plane 50 at any location in space (for example, near the object 70).

The above display control process will be described with reference to FIG. 1. In the example shown in FIG. 1, the display control device 100 displays a surface 50 in a virtual space. A virtual hand is set on the surface 50, and the user 10 can obtain feedback as if he or she were holding the hand and moving the surface 50 by operating the three-dimensional device 20 holding the hand.

The user 10 places the surface 50 near the object 70. The user 10 then operates the planar device 30 placed on the table to specify the position of the pointer 60 on the surface 50. The coordinates on the surface 50 and the spatial coordinates in the virtual space are associated with each other. For this reason, the display control device 100 can, for example, obtain an intersection point between the extension line of the position of the pointer 60 and the outline (3D model) of the object 70 based on the position and orientation (angle) of the surface 50. In the example of FIG. 1, the display control device 100 obtains spatial coordinates 65, which is an arbitrary point on the object 70, as the position corresponding to the pointer 60. In other words, the user 10 can indicate a specific position on the object 70 by specifying the position of the pointer 60 on the surface 50. This allows the user 10 to select a point, edge, face, etc. that he or she wants to edit from the object 70 with high accuracy.

Next, another display example will be shown using FIG. 2. FIG. 2 is a diagram (1) showing an example of the display control process according to the embodiment. The example in FIG. 2 shows a state in which the user 10 operates the three-dimensional device 20 (step S1) and moves the surface 50 in a direction toward the same object 70 as in FIG. 1.

In this way, the user 10 can more accurately specify a specific position of the object 70 by moving the surface 50 to a position where it is almost in contact with the object 70. The example in FIG. 2 shows a situation in which the user 10 moves the surface 50, and then operates the planar device 30 to select a specific edge 66 of the object 70.

Next, another display example will be shown using FIG. 3. FIG. 3 is a diagram (2) showing an example of the display control process according to the embodiment. The example shown in the upper part of FIG. 3 shows a state in which the user 10 operates the three-dimensional device 20 to move the surface 50 in a direction toward the object 70, similar to FIG. 2.

On the other hand, in the example shown in the lower part of FIG. 3, the position of the surface 50 is fixed by the operation of the user 10. In this case, the user 10 can switch the operation target to the object 70 and operate the object 70 so as to move it closer to the surface 50. By using such a method, the user 10 can also move the surface 50 and the object 70 relatively closer to each other, thereby enabling accurate input.

As described above with reference to Figures 1 to 3, the display control process according to the embodiment displays a surface 50 in space, allowing the three-dimensional device 20 and the two-dimensional device 30 to be used together to specify a position in space, thereby improving operability and input accuracy.

Note that each device in FIG. 1 conceptually represents a function in the display control system 1, and may take various forms depending on the embodiment. For example, the display control device 100 may be composed of two or more devices each having a different function, which will be described later. Specifically, the display control device 100 may be a device in which the display unit and the information processing unit are separately configured. In this case, the information processing unit of the display control device 100 may be any information processing device, such as a server or a PC (Personal Computer).

(1-2. Configuration of the display control device according to the embodiment)
Next, a description will be given of the configuration of the display control device 100. Fig. 4 is a diagram showing an example of the configuration of the display control device 100 according to an embodiment.

As shown in FIG. 4, the display control device 100 has a communication unit 110, a storage unit 120, a control unit 130, a sensor unit 140, and a display unit 150. The display control device 100 may also have an input unit (such as an operation button or a touch panel) that accepts various operations from a user 10 who operates the display control device 100.

The communication unit 110 is realized, for example, by a NIC (Network Interface Card) or a network interface controller. The communication unit 110 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the three-dimensional device 20 and the planar device 30 via the network N. The network N is realized, for example, by a wireless communication standard or method such as Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), UWB (Ultra Wide Band), or LPWA (Low Power Wide Area).

The storage unit 120 is realized, for example, by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

The storage unit 120 stores various information related to the display control process according to the embodiment. For example, the storage unit 120 stores a 3D model of a virtual object to be displayed on the display unit 150.

The sensor unit 140 is a sensor that detects various types of environmental information. For example, the sensor unit 140 includes an outward-facing camera that captures images of the outside of the display control device 100 and an inward-facing camera that captures images of the user 10.

For example, the sensor unit 140 functions as a recognition camera for recognizing the space in front of the user's eyes. For example, the sensor unit 140 recognizes the spatial position of the three-dimensional device 20 by photographing the three-dimensional device 20, whose spatial position has been calibrated in advance, and acquires position and orientation information of the three-dimensional device 20.

The sensor unit 140 also recognizes a subject (e.g., a real object located in real space) located in front of the display control device 100. In this case, the sensor unit 140 acquires an image of the subject located in front of the user, and can calculate the distance from the display control device 100 (in other words, the position of the user's viewpoint) to the subject based on the parallax between images captured by the stereo camera. Alternatively, the sensor unit 140 may detect the distance in real space using a depth sensor capable of detecting the distance to a real object.

In addition to the function as a recognition camera, the sensor unit 140 may have a function of detecting various information related to the user's motion, such as the orientation, inclination, motion, and speed of the user's body. Specifically, the sensor unit 140 detects information related to the user's motion, such as information related to the user's head and posture, the motion of the user's head and body (acceleration and angular velocity), the direction of the field of view, and the speed of the viewpoint movement. For example, the sensor unit 140 functions as various motion sensors such as a three-axis acceleration sensor, a gyro sensor, and a speed sensor, and detects information related to the user's motion. More specifically, the sensor unit 140 detects at least one change in the position and posture of the user's head by detecting the components of the yaw direction, pitch direction, and roll direction as the motion of the user's head. Note that the sensor unit 140 does not necessarily need to be provided in the display control device 100, and may be, for example, an external sensor connected to the display control device 100 by wire or wirelessly.

The display unit 150 displays various information output from the control unit 130. For example, the display unit 150 is a display that outputs video to the user 10. The display unit 150 may also include an audio output unit (such as a speaker) that outputs audio.

The control unit 130 is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU, etc., executing a program stored inside the display control device 100 using a RAM or the like as a working area. The control unit 130 is also a controller, and may be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

As shown in FIG. 4, the control unit 130 has an acquisition unit 131, a first detection unit 132, a second detection unit 133, and a display control unit 134.

The acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 acquires position and orientation information in three-dimensional space from the three-dimensional device 20, which is the first controller. The acquisition unit 131 also acquires position information, which is information indicating a position on the surface 50, from the planar device 30, which is the second controller.

The first detection unit 132 detects the movement of the surface 50 in three-dimensional space based on the input from the stereoscopic device 20. Specifically, the first detection unit 132 detects (calculates) the spatial position and orientation information of the stereoscopic device 20 by combining the image of the stereoscopic device 20 captured by the sensor unit 140 and the position and orientation information transmitted from the stereoscopic device 20. That is, the position and orientation information of the stereoscopic device 20 is obtained by merging the image information captured by the outward camera of the sensor unit 140 and the IMU (Inertial Measurement Unit) information provided in the stereoscopic device 20. Note that the input from the stereoscopic device 20 refers to the position and orientation information (or the variation value thereof) measured by the IMU or the like as a result of the stereoscopic device 20 being operated in space being input to the display control device 100 via the communication unit 110.

For example, the first detection unit 132 detects the movement and position/orientation information of the surface 50 by detecting the fluctuation value of the position/orientation information obtained based on the result of the operation of the three-dimensional device 20 by the user 10 holding a handle virtually attached to the surface 50.

Note that, in addition to the above processing, various known technologies may be used to detect the position of the three-dimensional device 20. For example, if an infrared LED (Infrared Light Emitting Diode) is embedded in the three-dimensional device 20, the first detection unit 132 can detect the position and posture of the three-dimensional device 20 by observing the state in which the infrared LED is shining with the outward camera of the sensor unit 140 and combining it with IMU information. In addition, the first detection unit 132 can also detect the position and posture of the three-dimensional device 20 by using hand truck technology to capture the hand of the user 10 with the camera of the sensor unit 140 or a camera installed in the room and estimating the position and posture of the hand.

The first detection unit 132 may also detect the relative movement of the surface 50, rather than the movement of the surface 50 itself. In other words, the first detection unit 132 may detect the movement of an object whose position can be specified using the surface 50, based on an input from the three-dimensional device 20. In this case, the first detection unit 132 receives the designation of the object that the user 10 wishes to move, and performs processing to move the object closer to or farther away from the surface 50 in accordance with the operation of the user 10.

The second detection unit 133 detects a position specified on the surface 50 (for example, the position of a pointer indicating the specified position) based on an input from the planar device 30, which is the second controller. Note that since the surface 50 is given position and orientation information (such as the inclination of the plane) in space based on the operation of the three-dimensional device 20, the pointer located on the surface 50 also includes not only a two-dimensional position (the coordinates of the surface 50) but also position and orientation information corresponding to the surface 50.

The second detection unit 133 also detects a selected part of the object based on the position specified on the surface 50. For example, the second detection unit 133 detects a part of the object, such as a point, edge, or surface, that corresponds to the position indicated by the pointer, based on the specification by the user 10 or initial settings.

At this time, the second detection unit 133 may detect the part selected on the object based on the relationship between the angle of the surface 50 and the position of the pointer, and the position of the object. Specifically, the second detection unit 133 detects the point on the extension line of the pointer, which is determined from the angle of the surface 50 and the position of the pointer, that intersects with the object as the part designated by the user 10.

For example, the second detection unit 133 may detect, as the part specified by the user 10, a point where an extension of a line normal to the surface 50, starting from the pointer position, intersects with the object. Alternatively, the second detection unit 133 may detect, as the part specified by the user 10, a point where an extension of a line connecting the viewpoint position of the user 10 and the pointer position intersects with the object. Alternatively, the second detection unit 133 may detect the part selected on the object based on the position where a line extending horizontally from the specified pointer position on the surface 50 intersects with the object, regardless of the angle of the surface 50.

The display control unit 134 controls the display unit 150 to display the information output from the control unit 130. In other words, the display control unit 134 outputs the virtual space image rendered as video content to the output destination device. Note that the output destination device is not limited to a head-mounted display such as the display unit 150, but may also be an external display, a smartphone, a television, or other video output device.

For example, the display control unit 134 displays a surface 50, which is a virtual coordinate plane on which two-dimensional coordinates can be specified, in three-dimensional space based on the position and orientation information of the three-dimensional device 20 in three-dimensional space.

The display control unit 134 also controls the display of a pointer on the surface 50 based on the position information on the surface 50 input from the planar device 30. When a part of an object is selected based on the pointer position, the display control unit 134 also clearly indicates the selected position.

(1-3. Processing Procedure According to the Embodiment)
Next, a procedure of a process according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a sequence diagram showing a flow of a display control process according to the embodiment.

As shown in FIG. 5, the 3D device 20 acquires position and orientation information of its own device using an IMU or the like (step S11). Then, the 3D device 20 transmits the acquired position and orientation information to the display control device 100 (step S12).

The display control device 100 captures an image of the 3D device 20 using an outward-facing camera or the like (step S21). Then, the display control device 100 estimates the position and orientation of the 3D device 20 (step S22).

Then, the display control device 100 combines the position and orientation information acquired from the three-dimensional device 20 and the position and orientation information acquired by photographing to identify the position and orientation of the three-dimensional device 20 (step S23). The display control device 100 detects the position and orientation of the surface 50 in the virtual space based on the position and orientation information of the three-dimensional device 20 (step S24). The display control device 100 displays the surface 50 on the display unit 150 based on the detected position and orientation of the surface 50.

Once the position of the surface 50 is determined, the planar device 30 acquires planar coordinate information for identifying the position on the surface 50 (step S31). The planar device 30 then transmits the acquired planar coordinate information to the display control device 100 (step S32).

The display control device 100 detects the pointer position on the surface 50 based on the information acquired from the planar device 30 (step S25). Then, the display control device 100 detects the position specified by the user 10 within the object based on the position and orientation of the pointer (step S26). Note that the orientation information of the pointer in this case is calculated based on, for example, the angle of the surface 50.

(1-4. Modified Examples)
(1-4-1. First Modification: Operation of Only One Device)
The process according to the embodiment may be modified in various ways. For example, in the embodiment, the user 10 operates the surface 50 and the pointer using both the three-dimensional device 20 and the planar device 30. However, the user 10 may execute the information process according to the embodiment using either one of the devices.

In other words, the display control device 100 can perform the display control processing according to the present disclosure even if the first controller (three-dimensional device 20) and the second controller (planar device 30) are the same device. In this case, the first detection unit 132 detects the same device as the first controller when the same device is located at a predetermined distance from the plane in real space. Furthermore, the second detection unit 133 detects the same device as the second controller when the same device is located within a predetermined distance from the plane in real space.

Alternatively, when the first detection unit 132 receives a designation from a user 10 of the same device to use the same device as a first controller, it may detect the same device as a first controller. Furthermore, when the second detection unit 133 receives a designation from a user 10 of the same device to use the same device as a second controller, it may detect the same device as a second controller.

The above process will be explained using Figures 6A and 6B. Figure 6A is a diagram for explaining device operations related to the first modified example.

In the left part of FIG. 6A, the user 10 operates the three-dimensional device 20 and the planar device 30, as in the embodiment.

On the other hand, in the right part of FIG. 6A, the user 10 operates only the flat device 30. In this case, when the flat device 30 is lifted from the table, the display control device 100 recognizes this operation. Then, when the flat device 30 is in the air, the display control device 100 recognizes the flat device 30 as a device that performs the same operation as the three-dimensional device 20. Specifically, when the flat device 30 is in the air, the display control device 100 recognizes the movement of the device as a device for moving the surface 50.

When the user 10 places the planar device 30 on a table, the display control device 100 recognizes the planar device 30 as a device for operating a pointer on the surface 50. This process allows the user 10 to realize the processing according to the embodiment using only one device.

The display control device 100 can also handle the three-dimensional device 20 in the same way as the planar device 30. For example, when the three-dimensional device 20 is moved near the tabletop, the display control device 100 recognizes such movement and handles the three-dimensional device 20 as the planar device 30. More specifically, when the three-dimensional device 20 is located near the tabletop, the display control device 100 can invalidate the input of position and orientation from the three-dimensional device 20 and handle it as a planar device 30 that inputs position information by touching the desk like a touch pen.

When the display control device 100 performs the processing according to the embodiment using only one device, in addition to the above processing, the display control device 100 may explicitly accept a switching operation from the user 10. For example, when the stereoscopic device 20 has a function capable of identifying a two-dimensional position (such as a trackball), the user 10 can flexibly perform the above switching.

Furthermore, when the display unit 150 of the display control device 100 can be temporarily removed to the top (referred to as flip-up, etc.), the display control device 100 may treat the stereoscopic device 20 as the planar device 30 while it is flipped up. This is because the user 10 cannot see stereoscopically while it is flipped up.

The flow of the above process will be explained using FIG. 6B. FIG. 6B is a flowchart showing the flow of the display control process according to the first modified example.

The display control device 100 determines whether the surface 50 and the two devices performing the pointer operation are separate devices (step S41). If they are separate devices (step S41; Yes), the display control device 100 sets each device to a movement that conforms to the original settings, as in the embodiment.

If the surface 50 and the two devices performing the pointer operation are not separate devices (step S41; No), the display control device 100 determines whether the device is recognized on a tabletop (any plane in three-dimensional space) (step S42). If the device is on a tabletop (step S42; Yes), the display control device 100 treats the device as a flat-surface device 30.

On the other hand, if the device is not on the table (step S42; No), the display control device 100 determines whether or not the user 10 has specified the device as a flat-screen device 30 (step S43). If the user 10 has specified the device as a flat-screen device 30 (step S43; Yes), the display control device 100 treats the device as a flat-screen device 30.

On the other hand, if the device is not specified as a flat device 30 (step S43; No), the display control device 100 determines whether the HMD is flipped up (step S44).

If the HMD is not flipped up (step S44; No), the display control device 100 treats the device as a stereoscopic device 20. On the other hand, if the HMD is flipped up (step S44; Yes), the display control device 100 treats the device as a planar device 30.

(1-4-2. Second Modification: Adjustment of Operation Feel)
In the above embodiment, an example has been described in which the user 10 moves the surface 50 by operating the three-dimensional device 20. For example, the surface 50 is provided with a virtual handle, and the user 10 can move the surface 50 as if it were a real object by operating the three-dimensional device 20 and gripping the handle. However, in a virtual space, a situation is also assumed in which the user 10 wants to move the surface 50 farther than the distance the user 10 actually reaches out. Alternatively, a situation is also assumed in which the surface 50 is installed farther than the distance the user 10 reaches out, and the user 10 cannot grip the handle.

The processing for such a situation will be explained using FIG. 7. FIG. 7 is a diagram (1) for explaining the display control processing relating to the second modified example. In the left part of FIG. 7, the user 10 grasps the handle of the surface 50 with his/her hand, as in the embodiment, and moves the hand by a distance of 200 to move the surface 50.

On the other hand, in the right part of FIG. 7, the user 10 desires to move the surface 50 farther than the distance 205 that the user 10 actually moves his/her hand. In this case, the display control device 100 provides a guide to the user 10 by, for example, displaying a ray in the virtual space from the hand toward the surface 50. For example, the user 10 can display the guide by explicitly selecting a display mode that has been set in advance by the display control device 100, such as a remote operation mode. In this case, the display control device 100 may move the surface 50 a distance longer than the actual distance the user's 10 hand moves, using only the movement of the user's 10 hand, even if the user 10 does not hold the surface 50 with his/her hand.

In addition, when the surface 50 is placed far away, the display control device 100 may move the surface 50 by treating it as if the user 10 were holding it with their hand, based on a guide extended from the hand.

Furthermore, when the user 10 grasps an area in space that is not the handle of the surface 50, the display control device 100 may perform display control to move the object instead of the surface 50, as shown in FIG. 3.

The display control device 100 may also control not only the surface 50 but also objects so that they can be remotely operated. This will be explained using FIG. 8. FIG. 8 is a diagram (2) for explaining the display control process according to the second modified example.

In the example of FIG. 8, the display control device 100 displays a guide 210 for grasping the object 70 through the surface 50. The user 10 can grasp or select the object 70 via the guide 210. This allows the user 10 to perform operations such as grasping or moving an object 70 that is placed in a position that is out of reach in the virtual space.

Note that when the user 10 wishes to select a part of the object 70, if the object 70 and the surface 50 are in contact, the part of the object 70 that is in contact can be selected, but a situation may occur in which a part that is not in contact cannot be selected. In this case, the display control device 100 may switch the pointer operation mode, for example, to emit a ray (guide) from the pointer on the surface in a specific direction, so that a part of the object 70 can be selected. The specific direction is, for example, a direction that can be determined based on the viewpoint position and pointer position of the user 10, such as a normal direction to the surface 50 or a direction connecting the viewpoint position of the user 10 and the pointer. Furthermore, in order to make it easier for the user 10 to select a part of the object 70, the display control device 100 may move the pointer on the surface to the part that is in contact when the object 70 and the surface 50 are not in contact with each other.

The display control device 100 may also fix the position of the surface 50 so that the surface 50 does not move to a position in the virtual space that cannot be grasped. In other words, the display control unit 134 may display the surface 50 in three-dimensional space based on the viewpoint position of the user 10, regardless of the position and orientation information of the three-dimensional device 20 in three-dimensional space. This point will be explained using FIG. 9. FIG. 9 is a diagram (3) for explaining the display control process related to the second modified example.

In the example of FIG. 9, the display control device 100 controls so that the surface 50 is displayed at a fixed distance 215 from the display control device 100. In this case, the surface 50 moves in conjunction with the movement of the display control device 100. This allows the user 10 to move the surface 50 for selecting an arbitrary position of the object 70 by, for example, directing his or her line of sight toward the object 70 without using the stereoscopic device 20. Note that if the surface 50 always moves in conjunction with the display control device 100, it becomes difficult to keep the surface 50 in a fixed position, so the display control device 100 may turn the automatic movement of the surface 50 on or off in response to a request from the user 10.

Note that the display control device 100 may not only change the position and orientation of the surface 50, but also change the size. For example, the display control device 100 changes the size of the surface 50 in conjunction with an operation such as the user 10 rotating the mouse wheel. The display control device 100 may also change the color of the surface 50.

The display control device 100 may also perform processing such as cutting out a cross section of an object in virtual space, or hiding any object that is even slightly within the area in front of the surface 50, depending on the position where the surface 50 is installed. In other words, the display control device 100 may change the display content of an object depending on the position of the surface 50. This allows the display control device 100 to improve the visibility of the object.

(1-4-3. Third Modification: Processing for Superimposing on Real Space)
The display control device 100 may also superimpose the surface 50 on the real space, as in AR display. In this case, the display control device 100 may snap the surface 50 to a surface or edge of an object in the real world, and adjust the movement so that the user can easily adjust the position. That is, when the display control unit 134 displays the surface 50 in a three-dimensional space in the real space, the first detection unit 132 may detect the surface 50 at a position where it is attached (snapped) to an arbitrary plane in the real space based on an input from the stereoscopic device 20, so that the surface 50 is displayed superimposed on the plane when the surface 50 approaches the plane by a predetermined distance.

This point will be explained using FIG. 10 and subsequent figures. FIG. 10 is a diagram (1) for explaining the display control process according to the third modified example.

FIG. 10 shows a state in which a surface 50 is superimposed on real space. Specifically, the example in FIG. 10 shows a state in which a surface 50 with a handle 220 is displayed as if it is attached to the screen of a real display 225. When a plane to which the surface 50 can be snapped is recognized, the display control device 100 displays a guide 230 along the plane (the outer frame of the display 225 in this example).

For example, when the user 10 grasps the handle 220 and brings the surface 50 closer to the display 225, the display control device 100 recognizes that the display 225 and the surface 50 have come within a predetermined distance, and snaps the surface 50 to the display 225 along the guide 230.

As a result, the user 10 can use the surface 50 that is tightly attached when selecting an object or the like projected on the display 225, and can perform selection operations with precision even in the AR space. Note that the display control device 100 may attach the surface 50 to a side surface adjacent to the screen, rather than to the front surface of the display 225. This allows the user 10 to use the display 225 as a normal monitor, while simultaneously performing operations on the surface 50 that is virtually displayed next to it.

Another example will be described with reference to FIG. 11. FIG. 11 is a diagram (2) for explaining the display control process according to the third modified example.

In the example of FIG. 11, the display control device 100 recognizes the desk 240 as a flat surface to which the surface 50 is attached. In other words, when the user 10 grasps the handle 220 and brings the surface 50 closer to the desk 240, the display control device 100 recognizes that the top surface of the desk 240 and the surface 50 have come within a predetermined distance, and snaps the surface 50 to the top surface of the desk 240.

This allows the user 10 to use the display 225 as a normal monitor, while also utilizing the desk 240 on which the display 225 is placed as a separate area for performing other tasks.

Furthermore, information processing on surface 50 placed on a table will be explained using FIG. 12. FIG. 12 is a diagram (3) for explaining the display control processing related to the third modified example.

12 shows the user 10 selecting a virtual object 305 displayed behind the surface 50 using a touch pen 300, which is a virtual pointer for specifying coordinates on the surface 50. The touch pen 300 is displayed, for example, when the user 10 brings the 3D device 20 close to a tabletop, and is operated based on the IMU information of the 3D device 20, etc.

In the example of FIG. 12, the virtual object 305 is located behind the surface 50, i.e., below the desk surface, so if the actual movements are used, the desk surface will get in the way and the virtual object 305 cannot be selected with the touch pen 300 using the 3D device 20. For this reason, the display control device 100 makes adjustments regarding the selection operation.

Specifically, when the display control unit 134 virtually displays an object and a pointer for selecting the object (in this example, the touch pen 300) in a three-dimensional space in the real space, the second detection unit 133 may control the size or contact point of the pointer so that the user 10 can virtually touch the object using the pointer. This point will be explained using FIG. 13. FIG. 13 is a diagram (4) for explaining the display control process related to the third modified example.

The left part of FIG. 13 shows an example in which the display control device 100 moves the touch pen 300 by the same amount as the actual movement of the 3D device 20. In this case, the touch pen 300 collides with the desk surface, and the user 10 does not get the sensation of touching the virtual object 305.

For this reason, when the display control device 100 recognizes that the virtual object 305 is located behind the surface 50, it dynamically changes the length of the touch pen 300, as shown on the right side of FIG. 13. Specifically, the display control device 100 adjusts the 3D device 20 so that it actually collides with the desk surface when the touch pen 300 comes into contact with the virtual object 305. This allows the display control device 100 to provide accurate force feedback to the user 10, and to perform accurate display control processing that does not penetrate the virtual object 305.

FIG. 14 shows an example of the display of the touch pen 300. FIG. 14 is a diagram (5) for explaining the display control process according to the third modified example.

For example, the display control device 100 determines the length of the touch pen 300 based on the distance to the virtual object 305 that the user 10 is trying to select. Specifically, the display control device 100 sets the distance 310 between the desk surface and the 3D device 20 to be the same as the distance 315 between the tip of the touch pen 300 and the virtual object 305.

The user 10 can also change the size of the surface 50 depending on the situation at the work location. That is, the first detection unit 132 may control the position and size of the surface 50 virtually displayed in real space based on the operation of the three-dimensional device 20 by the user 10. This point will be explained using FIG. 15. FIG. 15 is a diagram (6) for explaining the display control process according to the third modified example.

In the example shown in FIG. 15, the user 10 performs work using a surface 50 that is attached to a desk 240. At this time, the user 10 grasps the handle 220 and performs an action such as spreading the surface 50, thereby expanding the surface 50 so that it covers the flat surface of the desk 240. In other words, the display control device 100 expands or reduces the size of the surface 50 in response to the operation of the user 10. This allows the user 10 to effectively use the flat surface in real space on which the surface 50 is attached, thereby achieving better operability.

Furthermore, when the user 10 selects a virtual object, the display control device 100 may control the designated position to allow the user 10 to make an appropriate selection, for example by extending the designated position. In other words, the second detection unit 133 may control the operation of the pointer to extend the selection position using the pointer and allow the user 10 to virtually operate the object. This point will be explained using FIG. 16. FIG. 16 is a diagram (7) for explaining the display control process related to the third modified example.

In the example of FIG. 16, a surface 50 is displayed at a 45 degree angle with respect to the display 225. In addition, a camera 320, which is a virtual object, is displayed behind the surface 50. In this case, the user 10 cannot directly operate the camera 320 because it does not exist on the surface 50.

At this time, the display control device 100 extends a ray from the pointer 330 displayed by the user 10 and displays a guide 335 indicating the extended line, and a pointer 340 which is the intersection of the guide 335 and the camera 320. This allows the user 10 to operate the camera 320 using the extended pointer 340, even if the user 10 cannot actually move the pointer 330 to a position that touches the camera 320.

The display control device 100 can display the surface 50 at any angle in the AR display. This will be explained using FIG. 17. FIG. 17 is a diagram (8) for explaining the display control process according to the third modified example.

In the example of FIG. 17, the user 10 grasps the handle 220 and moves the surface 50. In this case, the display control device 100 may display the surface 50 at a right angle (90 degrees) to the plane in space (the screen of the display 225 in this example), or at an angle of 45 degrees or 0 degrees (i.e., the desk surface). In other words, the display control device 100 can accept any angle from the user 10, and therefore can create a work area that meets the needs of the user 10.

When the user 10 wishes to design or edit a virtual object, the display control device 100 may display the surface 50 at a position offset a predetermined distance from the object. In this case, the user 10 can perform selection operations on the surface 50 offset from the object, or can perform tasks such as sketching the object, thereby providing a work environment with excellent visibility.

(1-4-4. Fourth Modification: Deformation of Surface Based on Pointer Position)
The display control device 100 may change the size of the surface 50 itself in response to an operation by the user 10. That is, the second detection unit 133 may perform control so as to change the position or size of the surface 50 based on a position (e.g., a display position of a pointer) designated on the surface 50 by the user 10. This point will be described with reference to Fig. 18. Fig. 18 is a diagram for explaining a display control process according to the fourth modified example.

In the example at the top of FIG. 18, the pointer 350 is displayed at the right edge of the surface 50. In this case, the display control device 100 may display the surface 50 and the pointer 350 in a fixed manner, as shown in the lower left part of FIG. 18. The display control device 100 may also move the surface 50 to the right when the pointer 350 reaches the right edge, as shown in the lower center part of FIG. 18. The display control device 100 may also enlarge the display of the surface 50 when the pointer 350 reaches the right edge, as shown in the lower right part of FIG. 18. The display control device 100 may switch between these display control processes at the selection of the user 10. This allows the user 10 to automatically change the work area in accordance with the movement of the pointer 350, thereby obtaining better operability.

2. Other Embodiments
The processing according to each of the above-described embodiments may be implemented in various different forms other than the above-described embodiments.

Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

Furthermore, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

Furthermore, the above-mentioned embodiments and variations can be combined as appropriate to the extent that they do not cause any contradictions in the processing content.

Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

(3. Effects of the display control device according to the present disclosure)
As described above, the display control device according to the present disclosure (the display control device 100 in the embodiment) includes an acquisition unit (the acquisition unit 131 in the embodiment) and a display control unit (the display control unit 134 in the embodiment). The acquisition unit acquires position and orientation information in a three-dimensional space from a first controller (the three-dimensional device 20 in the embodiment). The display control unit displays a virtual coordinate plane (the surface 50 in the embodiment) on which two-dimensional coordinates can be specified in the three-dimensional space based on the position and orientation information in the three-dimensional space.

In this way, the display control device according to the present disclosure can improve input accuracy, such as pointing to a specific location on an object in space, by displaying a plane on which two-dimensional coordinates can be specified in three-dimensional space.

The display control device also includes a first detection unit (first detection unit 132 in this embodiment) that detects movement in the three-dimensional space of the virtual coordinate plane based on an input from the first controller. For example, the first detection unit detects movement of an object whose position can be specified using the virtual coordinate plane based on an input from the first controller.

In this way, the display control device uses a VR controller or the like to allow the user to move freely around the surface. This allows the user to perform input operations with a high degree of freedom.

The display control device also detects a position specified on the virtual coordinate plane based on an input from a second controller (in this embodiment, the planar device 30). For example, the second detection unit detects a part selected on the object based on a position specified on the virtual coordinate plane. Specifically, the second detection unit detects a part selected on the object based on the angle of the virtual coordinate plane and a position specified by the user on the virtual coordinate plane.

In this way, the display control device obtains input from the user using a planar device such as a mouse, and can therefore receive more accurate input than a VR controller, etc.

The second detection unit may also detect, as the selected part of the object, a point where an extension of a line normal to the virtual coordinate plane starting from the specified position intersects with the object, or a point where an extension of a line connecting the user's viewpoint position and the specified position intersects with the object.

In this way, the display control device can arbitrarily switch the behavior that determines the location selected by the pointer on the screen. This allows the user to avoid stressful behavior, such as the wrong location being selected due to the pointer position, and achieve the operability that they desire.

In addition, in the display control process executed by the display control device, the first controller and the second controller may be the same device. In this case, the first detection unit may detect the same device as the first controller when the same device is located at a position away from a plane in real space by a predetermined distance. Furthermore, the second detection unit may detect the same device as the second controller when the same device is located within a predetermined distance from a plane in real space. Alternatively, the first detection unit may detect the same device as the first controller when a designation to use the same device as the first controller is received from a user of the same device. The second detection unit may detect the same device as the second controller when a designation to use the same device as the second controller is received from a user of the same device.

In this way, the display control device can realize the display control processing according to the present disclosure in a single device, and can provide users with information processing that achieves high input accuracy even in a variety of equipment environments.

The display control unit also displays a virtual coordinate plane in three-dimensional space based on the user's viewpoint position, regardless of the position and orientation information of the first controller in three-dimensional space.

In this way, the display control device can reduce the number of operation steps required by the user by displaying a fixed surface, thereby realizing information processing that places less of a burden on the user.

The display control unit also displays a virtual coordinate plane in the three-dimensional space of the real space. When the virtual coordinate plane approaches an arbitrary plane in the real space by a predetermined distance, the first detection unit detects the virtual coordinate plane at a position adsorbed to the plane based on an input from the first controller so that the virtual coordinate plane is displayed superimposed on the plane.

In this way, the display control device can achieve a behavior that is like snapping a surface to a plane in real space. This allows the user to easily set a location on a tabletop device that is convenient for use as a surface, thereby improving work efficiency.

The display control unit also virtually displays an object and a pointer for selecting the object in a three-dimensional space in the real space. The second detection unit controls the size or contact point of the pointer so that the user can virtually touch the object using the pointer.

In this way, the display control device can provide the user with an operational feel that matches the sensation of real space by adjusting the size of the pointer and the contact point (i.e., the feedback sensation) to suit the real space.

The first detection unit also controls the position and size of the virtual coordinate plane virtually displayed in real space based on the user's operation of the first controller.

In this way, the display control device can change the size of the screen according to the user's desires, allowing the user to freely adjust the work area, thereby improving operability.

The second detection unit also extends the position selected by the pointer and controls the operation of the pointer so that the user can virtually operate the object.

In this way, the display control device can improve user operability by performing processing to eliminate inconveniences in the virtual space, such as objects being located far away.

The second detection unit also controls the change of the position or size of the virtual coordinate plane based on a position specified on the virtual coordinate plane by the user.

In this way, the display control device can freely change the size of the screen depending on the position of the pointer, providing the user with a comfortable working environment.

(4. Hardware Configuration)
Information devices such as the display control device 100 according to each embodiment described above are realized by a computer 1000 having a configuration as shown in Fig. 19, for example. The display control device 100 will be described below as an example. Fig. 19 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the display control device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input/output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

The CPU 1100 operates based on the programs stored in the ROM 1300 or the HDD 1400 and controls each component. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

The ROM 1300 stores boot programs such as the Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000.

HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a display control program related to the present disclosure, which is an example of program data 1450.

The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

The input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input/output interface 1600. The CPU 1100 also transmits data to an output device such as a display, edger, or printer via the input/output interface 1600. The input/output interface 1600 may also function as a media interface that reads programs and the like recorded on a specific recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase change rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, and semiconductor memories.

For example, when the computer 1000 functions as the display control device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes a display control program loaded onto the RAM 1200 to realize the functions of the control unit 130, etc. Also, the display control program according to the present disclosure and data in the storage unit 120 are stored in the HDD 1400. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from other devices via the external network 1550.

The present technology can also be configured as follows.
(1)
an acquisition unit that acquires position and orientation information in a three-dimensional space from the first controller;
a display control unit that displays, in the three-dimensional space, a virtual coordinate plane on which two-dimensional coordinates can be specified, based on position and orientation information in the three-dimensional space;
A display control device comprising:
(2)
a first detection unit that detects a movement in a three-dimensional space of the virtual coordinate plane based on an input by the first controller;
The display control device according to (1) further comprising:
(3)
The first detection unit is
detecting a movement of an object whose position can be specified using the virtual coordinate plane based on an input by a first controller;
The display control device according to (2).
(4)
a second detection unit that detects a position designated on the virtual coordinate plane based on an input from a second controller;
The display control device according to (2) or (3), further comprising:
(5)
The second detection unit is
detecting a selected portion of an object based on a position specified on the virtual coordinate plane;
The display control device according to (4).
(6)
The second detection unit is
detecting a part selected in the object based on an angle of the virtual coordinate plane and a position designated by a user on the virtual coordinate plane;
The display control device according to (5) above.
(7)
The second detection unit is
a point where an extension line extending from the specified position in a normal direction to the virtual coordinate plane intersects with the object, or a point where an extension line connecting the user's viewpoint and the specified position intersects with the object, is detected as the selected part of the object;
The display control device according to (6) above.
(8)
the first controller and the second controller are the same device;
The display control device according to any one of (4) to (7).
(9)
The first detection unit is
When the same device is located at a position away from a plane in real space by a predetermined distance, the same device is detected as the first controller;
The second detection unit is
When the same device is located within a predetermined distance from a plane in real space, the same device is detected as the second controller.
The display control device according to (8).
(10)
The first detection unit is
when receiving a designation from a user of the same device to use the same device as the first controller, detecting the same device as the first controller;
The second detection unit is
when a designation to use the same device as the second controller is received from a user of the same device, the same device is detected as the second controller;
The display control device according to (8).
(11)
The display control unit is
displaying the virtual coordinate plane in the three-dimensional space based on a viewpoint position of a user, regardless of position and orientation information of the first controller in the three-dimensional space;
The display control device according to any one of (2) to (10).
(12)
The display control unit is
Displaying the virtual coordinate plane on the three-dimensional space of a real space;
The first detection unit is
when the virtual coordinate plane approaches an arbitrary plane in real space by a predetermined distance based on an input by the first controller, the virtual coordinate plane is detected at a position where the virtual coordinate plane is attracted to the arbitrary plane so that the virtual coordinate plane is displayed superimposed on the arbitrary plane.
A display control device according to any one of (5) to (7).
(13)
The display control unit is
virtually displaying the object and a pointer for selecting the object in the three-dimensional space in the real space;
The second detection unit is
controlling a size or a contact point of the pointer so that the user can virtually touch the object using the pointer;
The display control device according to (12).
(14)
The first detection unit is
controlling a position and a size of the virtual coordinate plane virtually displayed in a real space based on an operation of the first controller by a user;
The display control device according to (12) or (13).
(15)
The second detection unit is
extending a position selected by the pointer and controlling an operation of the pointer so that the user can virtually operate the object;
The display control device according to (13).
(16)
The second detection unit is
and controlling the virtual coordinate plane so as to change a position or a size of the virtual coordinate plane based on a position designated on the virtual coordinate plane by a user.
The display control device according to any one of (4) to (7).
(17)
The computer
acquiring position and orientation information in a three-dimensional space from a first controller;
displaying a virtual coordinate plane in the three-dimensional space on the basis of the position and orientation information in the three-dimensional space, on which two-dimensional coordinates can be specified;
A display control method comprising:

REFERENCE SIGNS LIST 1 Display control system 10 User 20 Three-dimensional device 30 Planar device 50 Surface 100 Display control device 110 Communication unit 120 Storage unit 130 Control unit 131 Acquisition unit 132 First detection unit 133 Second detection unit 134 Display control unit 140 Sensor unit 150 Display unit

Claims

an acquisition unit that acquires position and orientation information in a three-dimensional space from the first controller;
a display control unit that displays, in the three-dimensional space, a virtual coordinate plane on which two-dimensional coordinates can be specified, based on position and orientation information in the three-dimensional space;
A display control device comprising:
a first detection unit that detects a movement in a three-dimensional space of the virtual coordinate plane based on an input by the first controller;
The display control device according to claim 1 , further comprising:
The first detection unit is
detecting a movement of an object whose position can be specified using the virtual coordinate plane based on an input by a first controller;
The display control device according to claim 2 .
a second detection unit that detects a position designated on the virtual coordinate plane based on an input from a second controller;
The display control device according to claim 2 , further comprising:
The second detection unit is
detecting a selected portion of an object based on a position specified on the virtual coordinate plane;
The display control device according to claim 4.
The second detection unit is
detecting a part selected in the object based on an angle of the virtual coordinate plane and a position designated by a user on the virtual coordinate plane;
The display control device according to claim 5 .
The second detection unit is
a point where an extension line extending from the specified position in a normal direction to the virtual coordinate plane intersects with the object, or a point where an extension line connecting the user's viewpoint and the specified position intersects with the object, is detected as the selected part of the object;
The display control device according to claim 6.
the first controller and the second controller are the same device;
The display control device according to claim 4.
The first detection unit is
When the same device is located at a position away from a plane in real space by a predetermined distance, the same device is detected as the first controller;
The second detection unit is
When the same device is located within a predetermined distance from a plane in real space, the same device is detected as the second controller.
The display control device according to claim 8.
The first detection unit is
when receiving a designation from a user of the same device to use the same device as the first controller, detecting the same device as the first controller;
The second detection unit is
when a designation to use the same device as the second controller is received from a user of the same device, the same device is detected as the second controller;
The display control device according to claim 8.
The display control unit is
displaying the virtual coordinate plane in the three-dimensional space based on a viewpoint position of a user, regardless of position and orientation information of the first controller in the three-dimensional space;
The display control device according to claim 2 .
The display control unit is
Displaying the virtual coordinate plane on the three-dimensional space of a real space;
The first detection unit is
when the virtual coordinate plane approaches an arbitrary plane in real space by a predetermined distance based on an input by the first controller, the virtual coordinate plane is detected at a position where the virtual coordinate plane is attracted to the arbitrary plane so that the virtual coordinate plane is displayed superimposed on the arbitrary plane.
The display control device according to claim 5 .
The display control unit is
virtually displaying the object and a pointer for selecting the object in the three-dimensional space in the real space;
The second detection unit is
controlling a size or a contact point of the pointer so that the user can virtually touch the object using the pointer;
The display control device according to claim 12.
The first detection unit is
controlling a position and a size of the virtual coordinate plane virtually displayed in a real space based on an operation of the first controller by a user;
The display control device according to claim 12.
The second detection unit is
extending a position selected by the pointer and controlling an operation of the pointer so that the user can virtually operate the object;
The display control device according to claim 13.
The second detection unit is
and controlling the virtual coordinate plane so as to change a position or a size of the virtual coordinate plane based on a position designated on the virtual coordinate plane by a user.
The display control device according to claim 5 .
The computer
acquiring position and orientation information in a three-dimensional space from a first controller;
displaying a virtual coordinate plane in the three-dimensional space on the basis of the position and orientation information in the three-dimensional space, on which two-dimensional coordinates can be specified;
A display control method comprising: