JP2023117420A - Information processing method, program, and system - Google Patents

Abstract

To provide an information processing method, a program, and a system that provide high intuitiveness of operation.SOLUTION: An information processing method that is executed by a computer system includes controlling: setting of a virtual viewpoint of a user for a real space; and setting of an operable range in which operation related to the real space can be executed for the real space by the user. This can provide high intuitiveness of operation. In addition, the operable range is set to a range that a controller held by the user can reach, that is, a range that a hand of the user can reach; and therefore, a human physical spatial-cognitive ability can be utilized.SELECTED DRAWING: Figure 1

Description

The present technology relates to an information processing method, program, and system applicable to virtual representation and the like.

The image display system described in Patent Literature 1 displays a virtual reality image in which the set measurement object and the observation viewpoint have the same actual positional relationship. A difference between the length of the object observed in real space and the length of the object observed in virtual reality is input. A ratio for enlarging or reducing the virtual reality image is calculated based on the input value, and the virtual reality image is enlarged or reduced. Thus, it is disclosed to display a virtual reality image in which the difference between the impression of the observer and the impression of the object in the image is eliminated (paragraphs [0032] to [0035] FIG. 7 of Patent Document 1, etc.).

JP 2008-52641 A

There is a demand for a technology capable of exhibiting high intuitiveness in operations for such virtual representations.

In view of the circumstances as described above, an object of the present technology is to provide an information processing method, a program, and a system capable of exhibiting high intuition in operation.

In order to achieve the above object, an information processing method according to an embodiment of the present technology is an information processing method executed by a computer system, which includes setting a user's virtual viewpoint for a real space, and setting a user's virtual viewpoint for the real space. and controlling the setting of the operable range in which the operation related to the real space is executable.

In this information processing method, the setting of the user's virtual viewpoint for the real space and the setting of the operable range in which the user can perform an operation regarding the real space are controlled. This makes it possible to exhibit high intuition in operation.

The control step may set a first virtual viewpoint within the real space, and set a second virtual viewpoint different from the first virtual viewpoint within the operable range.

The controlling step may change a scale of the operable range.

The information processing method may further include a detection step of detecting candidate planes for setting the virtual viewpoint or the operable range from the real space.

The control step may set a position a predetermined distance away from the detected candidate plane as the virtual viewpoint.

The control step may set a position a predetermined distance away from the detected candidate plane as a reference point that serves as a reference of the operable range.

The control step may control the virtual viewpoint or the operable range based on the set reference point.

The information processing method may further include a setting step of setting the position of the virtual viewpoint and the scale of the operable range based on the size of the real space. In this case, the control step may change the set position of the virtual viewpoint and the set scale of the operable range with reference to the reference point.

The detecting step may detect the candidate plane based on a predetermined axis of the real space.

The control step may change the scale of the operable range based on the position of the virtual viewpoint.

The control step may set the position of the virtual viewpoint based on the scale of the operable range.

The information processing method may further include a presentation step of presenting to the user a GUI (Graphical User Interface) capable of controlling the virtual viewpoint and the operable range.

The presenting step may present a virtual viewpoint image when the user views the real space from the virtual viewpoint. In this case, the GUI can set a first virtual viewpoint within the real space and set a second virtual viewpoint different from the first virtual viewpoint within the operable range. may

The GUI may be capable of setting the candidate plane within the operable range.

The real space may be a three-dimensional map created by sensors.

The control step may change the scale of the operable range based on the three-dimensional map created by the sensor.

The sensor may be mounted on a mobile object.

The GUI may be capable of generating a route along which the moving body travels according to the user's operation.

A program according to an embodiment of the present technology causes a computer system to execute the following steps.
A control step of controlling the setting of a user's virtual viewpoint with respect to the real space and the setting of an operable range in which the user can perform an operation with respect to the real space.

An information processing system according to one embodiment of the present technology includes a mobile body and an information processing device.
The mobile body moves in the real space.
The information processing device has a control unit that controls setting of a user's virtual viewpoint for the real space and setting of an operable range in which the user can perform an operation regarding the real space. .

1 is a diagram schematically showing the appearance of a virtual representation system; FIG. 1 is a block diagram showing a functional configuration example of a virtual representation system; FIG. 4 is a flow chart showing control of a virtual viewpoint and an operable range; FIG. 4 is a schematic diagram showing control of a virtual viewpoint and an operable range; 10 is a flow chart showing control for setting a new virtual viewpoint from a set virtual viewpoint; FIG. 5 is a schematic diagram showing an example of control for setting a second reference point; FIG. 4 is a schematic diagram showing an example of a GUI; 9 is a flow chart showing another control example of the virtual viewpoint and the operable range; FIG. 5 is a schematic diagram showing an example of selection of candidate planes; 9 is a flow chart showing another control example of the virtual viewpoint and the operable range; 10 is a flow chart showing an example of control for setting a virtual body; FIG. 4 is a schematic diagram showing an example of a virtual body; FIG. 10 is a flow chart showing a control example for setting a new virtual body from a virtual body; FIG. FIG. 10 is a schematic diagram showing an example of control for setting a second virtual body; FIG. 4 is a schematic diagram showing a setting example of a virtual body; FIG. 10 is a schematic diagram showing another example of control of the virtual viewpoint and the operable range; It is a schematic diagram which shows the external appearance of HMD. It is a block diagram which shows the hardware structural example of an information processing apparatus.

Hereinafter, embodiments according to the present technology will be described with reference to the drawings.

<First Embodiment>
FIG. 1 is a diagram schematically showing the appearance of a virtual representation system according to a first embodiment of the present technology.
As shown in FIG. 1 , the virtual representation system 100 has a mobile object 10 , an information processing device 20 and a user device 30 . The mobile object 10, the information processing apparatus 20, and the user device 30 are communicably connected via wires or wirelessly. The form of connection between devices is not limited, and wireless LAN communication such as WiFi or short-range wireless communication such as Bluetooth (registered trademark), for example, can be used.

The mobile object 10 is, for example, a drone capable of autonomous flight.
In this embodiment, the moving body 10 has a sensor unit 14 capable of observing its surroundings. For example, the sensor unit 14 includes an imaging device such as a stereo camera, digital camera, or monocular camera. Further, for example, the sensor unit 14 may be arranged with a 360-degree camera capable of photographing the moving body 10 in 360 degrees or a stereo camera in front, back, left, right, up and down (different directions). In addition to this, sensor devices such as ToF (Time of Flight) sensors, laser ranging sensors, contact sensors, ultrasonic sensors, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), and sonar may be used. .
Note that the mobile object 10 is not limited to a drone, and may be, for example, a wheeled robot, a multi-legged robot, or a robot with legs having a multi-joint structure.

Further, in this embodiment, the sensing result of the sensor unit 14 mounted on the moving body 10 is supplied to the information processing device 20 .

The information processing device 20 generates a three-dimensional map around the mobile object 10 based on the sensing results supplied from the mobile object 10 .
A three-dimensional map is a three-dimensional map that displays the environment around the mobile object 10 . For example, in FIG. 1, the information processing device 20 generates a three-dimensional map including a person 1, a car 2, and a pedestrian bridge 3. FIG.
Note that the method of generating the three-dimensional map is not limited. For example, CAD (Computer Aided Design) or the like may be used.

The information processing device 20 also controls the setting of the virtual viewpoint 6 of the user 5 with respect to the 3D map and the setting of the operable range 7 with respect to the 3D map. In this embodiment, the information processing apparatus 20 changes the setting of the position of the virtual viewpoint 6 and the scale of the operable range 7 according to the operation of the user 5 .
A virtual viewpoint is a virtual viewpoint of the user 5 set at an arbitrary location (coordinates) within the three-dimensional map. For example, the user 5 can see the mobile object 10 and the pedestrian bridge 3 from a bird's-eye view from the set virtual viewpoint 6 .
The operable range is a range within which the user 5 can operate the 3D map. Also, the scale of the operable range includes the size of the operable range such as volume and area, and the shape of a circle, cylinder, rectangular parallelepiped, and the like.
The user's operation includes setting the virtual viewpoint 6, changing the scale of the operable range 7, and generating a route along which the moving object 10 moves.
Note that the scale of the operable range 7 is not limited, and may be set to any size and shape.

User device 30 is a terminal operated by user 5 . In this embodiment, the user device 30 uses an HMD 31 (Head Mounted Display) such as a VR headset and a controller 32 .

The HMD 31 has various sensors capable of detecting, for example, the posture of the user 5, the position of the eyes, the line of sight, and the like.
The controller 32 is equipped with, for example, an inertial measurement unit (IMU) or the like that detects acceleration, angular velocity, and the like due to button or user 5 operation.
The user 5 can visually recognize the 3D map from the virtual viewpoint 6 via the HMD 31 . Also, the user 5 can set the virtual viewpoint 6 within the operable range 7 , change the scale of the operable range 7 , and generate the route of the moving object 10 via the controller 32 .
For example, the user 5 can generate the trajectory of the moving body 10 when moving the right hand holding the controller 32 . Also, for example, it is possible to set the position of the hand of the user 5 as a new virtual viewpoint. That is, it can be said that the operable range 7 is a distance within which the user 5 holding the controller 32 can reach.

FIG. 2 is a block diagram showing a functional configuration example of the virtual representation system 100. As shown in FIG.
As shown in FIG. 2 , the virtual representation system 100 has a mobile object 10 , an information processing device 20 and a user device 30 .

The moving body 10 has a driving system control section 11 , an external information detection section 12 , a state detection section 13 and a sensor section 14 .

The driving system control unit 11 generates various control signals for the moving body 10 and controls various devices related to the driving system of the moving body 10 .
For example, the moving body 10 includes servo motors that can specify angles and torques provided in the joints of four legs, a motion controller that decomposes and replaces the movement of the robot itself into movements of the four legs, and each Equipped with a feedback control device using a sensor in the motor and a sensor on the sole of the foot.
Further, for example, the moving body 10 includes a driving force generator for generating driving force such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, a steering mechanism for adjusting the steering angle, A braking device that generates a braking force, an ABS (Antilock Brake System), an ESC (Electronic Stability Control), an electric power steering device, and the like may be provided.

The external information detection unit 12 detects information on the outside of the moving body 10 based on the sensing result of the sensor unit 14 . For example, the external information detection unit 12 performs detection processing, recognition processing, and tracking processing of an object around the moving body 10, and detection processing of the distance to the object.
Further, for example, the external information detection unit 12 performs detection processing of the environment around the moving object 10 . Surrounding environments to be detected include, for example, weather, temperature, humidity, brightness, and road conditions.
In this embodiment, the external information detection unit 12 supplies data indicating the result of detection processing to the information acquisition unit 21 .

The state detector 13 detects the state of the moving body 10 based on data or signals from the driving system controller 11 . For example, the state detection unit 13 detects the speed, acceleration, steering angle, presence/absence and content of abnormality of the mobile body 10, and other states of devices mounted on the mobile body.
In this embodiment, the state detection unit 13 supplies data indicating the result of detection processing to the information acquisition unit 21 .

The sensor unit 14 observes the surroundings of the mobile object 10 . In this embodiment, the sensing result acquired by the sensor unit 14 is output to the external information detection unit 12 and the information acquisition unit 21 .

The information processing apparatus 20 has hardware necessary for configuring a computer, such as processors such as CPU, GPU, and DSP, memories such as ROM and RAM, and storage devices such as HDD (see FIG. 18). For example, the information processing method according to the present technology is executed by the CPU loading a program according to the present technology pre-recorded in the ROM or the like into the RAM and executing the program.
For example, the information processing device 20 can be realized by any computer such as a PC. Of course, hardware such as FPGA and ASIC may be used.
In the present embodiment, the operable range changing unit and the virtual viewpoint setting unit are configured as functional blocks by the CPU executing a predetermined program. Of course, dedicated hardware such as an IC (integrated circuit) may be used to implement the functional blocks.
The program is installed in the information processing device 20 via various recording media, for example. Alternatively, program installation may be performed via the Internet or the like.
The type of recording medium on which the program is recorded is not limited, and any computer-readable recording medium may be used. For example, any computer-readable non-transitory storage medium may be used.

As shown in FIG. 2, the information processing apparatus 20 includes an information acquisition unit 21, a map generation unit 22, a candidate plane detection unit 23, a reference point setting unit 24, an operable range change unit 25, a virtual viewpoint setting unit 26, and a GUI. (Graphical User Interface) It has a presentation unit 27 .

The information acquisition unit 21 acquires various information. In this embodiment, the information acquisition unit 21 acquires various information from the mobile object 10 and the user device 30 . For example, sensing results regarding obstacles such as objects and walls around the moving body 10 are obtained. Further, for example, operation information of the user 5 input by the user device 30 is acquired.
In addition, in the present embodiment, the information acquisition unit 21 supplies various acquired information to the map generation unit 22 . The information acquisition unit 21 also supplies the operation information input to the user device 30 to the driving system control unit 11 .

The map generation unit 22 acquires a three-dimensional map around the mobile object 10 . In this embodiment, the map generation unit 22 generates a three-dimensional map based on the sensing results acquired by the information acquisition unit 21 . For example, the map generating unit 22 may generate the self-location and three-dimensional map of the moving body 10 by SLAM (Simultaneous Localization and Mapping) or the like.
Specifically, based on the detection results supplied from the sensor unit 14, the map generation unit 22 accumulates time-series information supplied in time series in the database, and based on the accumulated time-series information, Estimate self-position and output as time-series information self-position.
The map generation unit 22 also estimates the self-position based on the current detection result supplied from the sensor unit 14 and outputs it as current information self-position. Then, the map generator 22 integrates or switches the time-series information self-position and the current information self-position, and outputs them as a self-position estimation result. Further, the map generation unit 22 detects the attitude of the moving body 10 based on the detection result supplied from the state detection unit 13, detects a change in the attitude, detects a large change in the self-position, When it is considered that the position estimation accuracy is degraded, the self-position may be estimated only from the current self-position information.
The map generation unit 22 may also acquire a three-dimensional map prepared in advance. For example, when moving the mobile object 10 on the first floor of a building, a map of the first floor of the building may be acquired.

The candidate plane detection unit 23 detects candidate planes from the three-dimensional map.
A candidate plane is a plane for setting a virtual viewpoint or an operable range. In this embodiment, a predetermined plane within the three-dimensional map generated by the map generation unit 22 is detected as a candidate plane. For example, the X plane, Y plane, and Z plane in the 3D map are detected as candidate planes. Alternatively, for example, a predetermined direction in the three-dimensional map is set as the direction of gravity, and planes perpendicular to the direction of gravity are detected as candidate planes.
Further, in the present embodiment, the candidate planes detected by the candidate plane detection unit 23 are supplied to the reference point setting unit 24 and the virtual viewpoint setting unit 26 .

The reference point setting unit 24 sets a reference point that serves as a reference for the operable range. In this embodiment, the reference point setting unit 24 sets reference points based on the candidate planes detected by the candidate plane detection unit 23 . For example, a position separated by a predetermined distance in the vertical direction of the candidate plane is set as the reference point. In this case, the reference point may be set in the vertical direction from the center of gravity of the candidate plane, or the user may set the distance and position between the candidate plane and the reference point.
In this embodiment, the positional information of the reference point set by the reference point setting section 24 is supplied to the operable range changing section 25 and the virtual viewpoint setting section 26 .

The operable range changing unit 25 changes the scale of the operable range based on the set reference point. In this embodiment, the operable range changing unit 25 changes the scale of the spherical operable range around the reference point. That is, the operable range changing unit 25 changes the size of the radius of the sphere.
Further, the operable range changing unit 25 associates the scale of the operable range with the position information of the reference point. For example, control may be performed such that the farther the reference point is from the candidate plane, the larger the scale of the operable range.

The virtual viewpoint setting unit 26 sets the position of the virtual viewpoint based on the detected candidate plane. For example, a position a predetermined distance away from the candidate plane in the vertical direction is set as the virtual viewpoint. In this case, the virtual viewpoint may be set in the vertical direction from the center of gravity of the candidate plane, or the user may set the distance and position between the candidate plane and the virtual viewpoint.
The virtual viewpoint setting unit 26 also sets the position of the virtual viewpoint corresponding to the scale of the operable range. For example, the larger the scale of the operable range, the farther the position of the virtual viewpoint is from the candidate plane. Similarly, the position of the virtual viewpoint corresponding to the positional information of the reference point may be set. For example, a virtual viewpoint may be set at a predetermined distance from the set reference point in the vertical direction of the candidate plane.
That is, a table in which the scale of the operable range and the position of the virtual viewpoint are associated with the position information of the reference point may be prepared in advance. In this case, by setting the reference point, the scale of the operable range and the position of the virtual viewpoint are uniquely determined.
The setting of the virtual viewpoint includes adjustment of a virtual inter-pupillary distance (IPD) in VR (Virtual Reality). For example, the interpupillary distance is appropriately set according to the size of the scale of the operable region.

The GUI presenting unit 27 presents the GUI to the user 5 . In the present embodiment, the GUI presenting unit 27 presents a GUI that allows a three-dimensional map to be visually recognized from a virtual viewpoint and a virtual viewpoint and operable range to be controlled. For example, the user 5 can select the detected candidate plane and set the position of the reference point via the GUI.
Also, the GUI presentation unit 27 presents a GUI capable of generating the route of the moving object 10 . For example, the user 5 can observe the three-dimensional map and the moving body 10 from a virtual viewpoint via the GUI and generate a route from the current location of the moving body 10 to the destination point.

Note that, in the present embodiment, the operable range changing unit 25 and the virtual viewpoint setting unit 26 set the user's virtual viewpoint with respect to the real space, and perform an operation on the real space that allows the user to perform an operation related to the real space. A step corresponding to the control step for controlling the setting of the possible range is executed.
In this embodiment, the candidate plane detection unit 23 executes a step corresponding to a detection step of detecting a candidate plane for setting a virtual viewpoint or an operable range from the real space.
In this embodiment, the reference point setting unit 24, the operable range changing unit 25, and the virtual viewpoint setting unit 26 set the position of the virtual viewpoint and the scale of the operable range based on the size of the real space. Execute the steps corresponding to the setting steps to be performed.
In this embodiment, the GUI presenting unit 27 executes a step corresponding to a presenting step of presenting a GUI capable of controlling the virtual viewpoint and the operable range.
In addition, in this embodiment, the three-dimensional map corresponds to the actual space.

FIG. 3 is a flow chart showing control of the virtual viewpoint and the operable range.

Three-dimensional measurement of the actual space is performed by the sensor unit 14 of the moving body 10 (step 101). Alternatively, a known map prepared in advance is read by the information acquisition unit 21 . A three-dimensional map is generated by the map generation unit 22 based on the sensing result of the sensor unit 14 .

A candidate plane is detected from the three-dimensional map by the candidate plane detector 23 (step 102). For example, the candidate plane detection unit 23 detects a plurality of candidate planes from planes perpendicular to a predetermined axial direction (for example, Z-axis direction). Further, the candidate planes detected by the candidate plane detector 23 are presented to the user 5 (step 103). User 5 selects the presented candidate plane (step 104).

The reference point setting unit 24 sets the two-dimensional position of the reference point on the selected candidate plane (step 105). For example, the reference point setting unit 24 sets the X coordinate and the Y coordinate on the selected candidate plane of the XY plane. Also, the reference point setting unit 24 vertically moves the set reference point by a predetermined distance from the two-dimensional position (step 106). That is, a reference point is set by steps 105 and 106 .
Note that the order of steps 105 and 106 may be exchanged. For example, it may be moved vertically from the selected candidate plane, and the two-dimensional position of the reference point may be set from the moved plane.

The operable range changing unit 25 changes the scale of the operable range based on the set reference point (step 107).

The virtual viewpoint setting unit 26 sets the position of the virtual viewpoint based on the reference point and the scale of the operable range (step 108). For example, the virtual viewpoint setting unit 26 sets the virtual viewpoint at a position that is vertically away from the candidate plane and further from the reference point than the reference point.

FIG. 4 is a schematic diagram showing control of the virtual viewpoint and the operable range. In FIG. 4, the three-dimensional map is omitted for simplification.
FIG. 4A is a schematic diagram showing step 102. FIG. That is, the candidate plane 41 is detected from the measured three-dimensional map 40 by the sensor unit 14 of the moving body 10 . For example, by setting a predetermined direction (direction of gravity) in the three-dimensional map 40 , planes perpendicular to the direction of gravity are detected as candidate planes 41 .

FIG. 4B is a schematic diagram showing steps 103-106.
The user selects the detected candidate plane 41 and sets the two-dimensional position of the reference point 42 on the selected candidate plane. Also, the user can set a position, which is vertically moved by a predetermined distance from the set two-dimensional position, as the reference point 42 .

FIG. 4C is a schematic diagram showing steps 107 and 108. FIG.
The operable range changing unit 25 changes the scale of the operable range 43 based on the set reference point 42 . For example, in FIG. 4C, a scaled operable range 43 that covers the three-dimensional map 40 is set around the reference point 32 . Also, the virtual viewpoint setting unit 26 sets a virtual viewpoint 44 at a position a predetermined distance away from the reference point 42 in the direction opposite to the direction of gravity.

FIG. 5 is a flow chart showing control for setting a new virtual viewpoint from the set virtual viewpoint.

The position of the virtual viewpoint and the scale of the operable range are determined by steps 101 to 108 shown in FIG. 3 (step 201). Here, for convenience, the reference point, virtual viewpoint, and operable range determined in step 201 are referred to as the first reference point, first virtual viewpoint, and first operable range. The newly determined reference point, virtual viewpoint, and operable range are referred to as a second reference point, second virtual viewpoint, and second operable range.

The operable range changing unit 25 and the virtual viewpoint setting unit 26 control the first virtual viewpoint and the first operable range in order to set the second reference point (step 202). A specific method will be described with reference to FIG.

A new reference point is set by the reference point setting unit 24 from the controlled first virtual viewpoint and the first operable range (step 203). For example, a second reference point is set based on the position or candidate plane selected by the user 5 .

The operable range changing unit 25 changes the scale of the second operable range (step 204).

A second virtual viewpoint is set by the virtual viewpoint setting unit 26 (step 205). For example, the virtual viewpoint setting unit 26 sets the position of the virtual viewpoint based on the position of the second reference point and the scale of the operable range.
User 5 can move from a first virtual viewpoint to a second virtual viewpoint. That is, it becomes possible to view the three-dimensional map from the second virtual viewpoint.

FIG. 6 is a schematic diagram showing a control example for setting the second reference point. FIG. 6 illustrates an example of controlling the first virtual viewpoint and the first operable range in order to set the second reference point in step 202 .

FIG. 6A is a schematic diagram of setting the second reference point 54 from the first operable range 53. FIG. The example shown in FIG. 6A is an example in which the position of the first virtual viewpoint and the scale of the first operable range are not controlled.

As shown in FIG. 6A, a first reference point 51, a first virtual viewpoint 52, and a first operable range 53 are illustrated in a three-dimensional map 50. FIG. The user 5 visually recognizes the three-dimensional map 50 from the first virtual viewpoint 52 and can perform various operations within the first operable range 53 via the controller 32 .
In this embodiment, the user 5 sets the second reference point 54 within the first operable range 53 via the controller 32 . For example, the user 5 can set the position as the coordinates of the second reference point 54 by placing the hand holding the controller 32 at a predetermined position.

The user 5 also controls the second virtual viewpoint 55 and the second operable range 56 from the set second reference point 54 . For example, user 5 controls second virtual viewpoint 55 and second operable range 56 via controller 32 . Specifically, the position of the second virtual viewpoint 55 may be set to the position of the hand holding the controller 32 . Also, the scale of the second operable range 56 may be changed by various operations using the controller 32 such as drawing a circle with the controller 32 .

FIG. 6B is a schematic diagram showing a control example for controlling the first virtual viewpoint and the first operable range. Also, in FIG. 6B, the three-dimensional map is omitted for simplification.

In the example shown in FIG. 6B, it is assumed that the position of the second reference point 64 desired by the user 5 is farther than the first operable range 63 . That is, this is an example of control when the user 5 wants to set a position that cannot be set by the controller 32 as the second reference point.

In this case, the virtual viewpoint setting unit 26 sets the position of the first virtual viewpoint 62 so that the entire three-dimensional map 60 can be overlooked. For example, the first virtual viewpoint 62 is set at a position where the area of the generated three-dimensional map on the XY plane, YZ plane, or XZ plane fits within the angle of view of the HMD 31 (the field of view of the user 5).
In this case, the position of the first reference point 61 and the scale of the first operable range 63 are set to predetermined numerical values.

The user 5 sets a second reference point 64 within a first operable range 63 from a first virtual viewpoint 62 set so as to overlook the entire three-dimensional map 60 . The user 5 also controls the second virtual viewpoint 65 and the second operable range 66 from the set second reference point 64 .

FIG. 6C is a schematic diagram showing another control example for controlling the first virtual viewpoint and the first operable range.
In the example shown in FIG. 6C, the first operable range 73 is changed so as to include the entire generated three-dimensional map 70 . That is, this is another example of control when the user 5 wants to set a position that cannot be set by the controller 32 as the second reference point.
Further, in FIG. 6C, the map generation unit 22 generates a rectangular parallelepiped three-dimensional map 70 .

The operable range changing unit 25 changes the scale of the first operable range 73 based on the size of the generated three-dimensional map 70 . For example, the operable range changing unit 25 changes the scale of the first operable range 73 based on the volume of the cuboid containing the generated three-dimensional map 70 .
In this case, the position of the first reference point 71 and the position of the first virtual viewpoint 72 are set to predetermined numerical values.

The user 5 sets a second reference point 74 within a first operable range 73 from a first virtual viewpoint 72 set to encompass the entire three-dimensional map 70 . The user 5 also controls the second virtual viewpoint 75 and the second operable range 76 from the set second reference point 74 .

Note that the control example for setting the second reference point is not limited. It may be controlled to a scale of 1 operable range.

FIG. 7 is a schematic diagram showing an example of a GUI.
As shown in FIG. 7, the GUI presentation unit 27 presents various GUIs. FIG. 7 shows a state in which the user 5 observes the three-dimensional map generated by the map generation unit 22 from the first virtual viewpoint. In FIG. 7, a hand-shaped virtual controller is displayed on the three-dimensional map at a location corresponding to the position of the controller 32 used by the user 5 .

FIG. 7A is a schematic diagram showing an example of a control GUI for controlling the virtual viewpoint and the operable range.
As shown in FIG. 7A, the control GUI 80 presents detected candidate planes 81 for controlling the second virtual viewpoint and the second operable range. User 5 can select candidate plane 81 via virtual controller 82 . For example, the user 5 can select the candidate plane 81 by moving the hand (the controller 32 ) so as to superimpose the virtual controller 82 on the candidate plane 81 .
The user 5 can also set a second reference point 83 from the selected candidate plane 81 . For example, the distance by which the virtual controller 82 is moved from the selected candidate plane 81 is the position of the second reference point 83 . Similarly, user 5 can control second virtual viewpoint 84 and second operable range 86 .

Note that the display and operation for controlling the virtual viewpoint and the operable range are not limited. For example, the plurality of detected candidate planes may be presented to the user 5 so as to blink. Further, for example, control of the reference point, virtual viewpoint, and operable range may be executed for each shape of the virtual controller 82, such as the state in which the index finger is extended or the state in which the hand is open.
Note that the shape of the virtual controller 82 is not limited. For example, it may have a shape that imitates the shape of the controller 32 .

FIG. 7B is a schematic diagram showing an example of a route GUI for generating the route of the mobile object 10. FIG.

The user 5 creates a route 86 for the mobile object 10 via the route GUI 85 . For example, the user 5 generates a trajectory when moving the virtual controller 82 as the path of the moving object 10 . In this embodiment, the virtual controller 82 representing the right hand of the user 5 generates the route of the mobile object 10 .

Various operations of the moving body 10 may be performed via the route GUI 85 . For example, the imaging timing of the camera mounted on the moving body 10 may be set. Further, for example, the speed or flight pattern of the moving body 10 may be controlled. That is, it is possible to control the trajectory, speed, etc. defined as a pattern, such as a turn or a figure of eight. For example, for a flight pattern such as a turn or a figure of eight, the speed, curvature, etc. of the moving body 10 when performing a turn or a figure of eight can be set.
Parameters such as speed and attitude associated with the flight pattern may be defaulted. That is, how to move a predetermined flight pattern may be set by default.

Also, the virtual viewpoint when operating the control GUI 80 and the route GUI 85 may be set at the position of the camera mounted on the moving body 10 . In that case, the user 5 can visually recognize the three-dimensional map or the actually captured image from the angle of view of the camera. Also, if the camera mounted on the moving object 10 is a 360-degree camera, it is possible to rotate the field of view according to the operation of the user 5 .

As described above, in the information processing method according to the first embodiment, setting of the virtual viewpoint 6 of the user 5 with respect to the 3D map, and setting of the operable range 7 in which the user 5 can perform operations regarding the 3D map. Control settings. This makes it possible to exhibit high intuition in operation.

Conventionally, when operating a mobile object from a subjective viewpoint, the field of view is limited, and it is necessary to understand the motion characteristics of the mobile object before performing complicated operations. On the other hand, prior flight planning using a map requires a GPS (Global Positioning System, Global Positioning Satellite) or requires a map in advance, which limits the circumstances in which it can be applied. In addition, many pre-mapped routings are two-dimensional.

Therefore, in the present technology, an operable range in which an operation regarding the user's virtual viewpoint and the user's space, which is set in the space, is executable is controlled. As a result, even if a spatial map has not been obtained in advance, the route of the moving object can be designated within a three-dimensional range. In addition, it is possible to flexibly select the virtual viewpoint and the operable range, and it is possible to obtain the range of the place to be designated and the required designation accuracy.

<Second embodiment>
A virtual representation system of a second embodiment according to the present technology will be described. In the following description, descriptions of parts similar to the configuration and operation of the virtual representation system described in the above embodiment will be omitted or simplified.

In the first embodiment, the reference point was set from the candidate plane. In the second embodiment, virtual viewpoints are set from candidate planes. Also, after the virtual viewpoint is set, the scale of the operable range is changed.

FIG. 8 is a flow chart showing another control example of the virtual viewpoint and the operable range.

Three-dimensional measurement of the actual space is performed by the sensor unit 14 of the moving body 10 (step 301). Alternatively, a known map prepared in advance is read by the information acquisition unit 21 . A three-dimensional map is generated by the map generation unit 22 based on the sensing result of the sensor unit 14 .

A candidate plane is detected from the three-dimensional map by the candidate plane detector 23 (step 302). For example, the candidate plane detector 23 detects a plurality of candidate planes from planes perpendicular to a predetermined axial direction. Further, the candidate planes detected by the candidate plane detector 23 are presented to the user 5 (step 303). User 5 selects the presented candidate plane (step 304).

The virtual viewpoint setting unit 26 sets the two-dimensional position of the virtual viewpoint on the selected candidate plane (step 305). For example, the virtual viewpoint setting unit 26 sets the X coordinate and the Y coordinate on the selected candidate plane (XY plane). Also, the virtual viewpoint setting unit 26 moves the set two-dimensional position of the virtual viewpoint in the vertical direction by a predetermined distance (step 306). That is, a virtual viewpoint is set by steps 305 and 306 .
Note that the order of steps 305 and 306 may be interchanged. For example, the virtual viewpoint may be moved vertically from the selected candidate plane and the two-dimensional position of the virtual viewpoint may be set from the moved plane.

The operable range changing unit 25 changes the scale of the operable range based on the set virtual viewpoint (step 307).

The reference point setting unit 24 sets the position of the reference point based on the position of the virtual viewpoint and the scale of the operable range (step 308). For example, the reference point setting unit 24 sets the center of the operable range, or a position that is the same as the two-dimensional position of the virtual viewpoint and is separated from the candidate plane by a predetermined distance in the vertical direction.
Note that the order of steps 307 and 308 may be exchanged. For example, the operable range changing unit 25 may change the operable range based on the position of the virtual viewpoint and the position of the reference point.
A table in which the scale of the operable range and the position of the reference point are associated with the position of the virtual viewpoint may be prepared in advance. In this case, the scale of the operable range and the position of the reference point are uniquely determined by setting the virtual viewpoint.

In the first embodiment, planes perpendicular to the direction of gravity are detected as candidate planes. Any surface on the three-dimensional map may be detected as a candidate plane without being limited to this.

FIG. 9 is a schematic diagram showing an example of selection of candidate planes. Note that the three-dimensional map is omitted in FIG. 9 for the sake of simplification.

FIG. 9A is a schematic diagram illustrating detection of candidate planes.
The candidate plane detection unit 23 detects arbitrary planes of the three-dimensional map 90 as candidate planes. In this embodiment, multiple candidate planes are detected according to each plane of the three-dimensional map 90 .

FIG. 9B is a schematic diagram showing setting of the position of the virtual viewpoint.
The user selects the detected candidate plane 91 and sets the two-dimensional position of the virtual viewpoint 92 on the selected candidate plane 91 . Also, the user can set a position, which is vertically moved by a predetermined distance from the set two-dimensional position, as the virtual viewpoint 92 .

FIG. 9C is a schematic diagram showing control of the operable range and reference points.
The operable range changing unit 25 changes the scale of the operable range 93 based on the set virtual viewpoint. Also, the reference point setting unit 24 sets a reference point 94 at a position a predetermined distance away from the virtual viewpoint 92 in the direction of the candidate plane 91 . Alternatively, the reference point 94 may be set at the center of the operable range 93 by the reference point setting section 24 .

FIG. 10 is a flow chart showing another control example of the virtual viewpoint and the operable range.
FIG. 10 shows an example in which candidate planes are determined by selecting a predetermined direction in the three-dimensional map. For example, when a plane perpendicular to the direction of gravity is automatically selected when the direction of gravity is selected, the user need not perform control to select a candidate plane.

Three-dimensional measurement of the actual space is performed by the sensor unit 14 of the moving body 10 (step 401). Alternatively, a known map prepared in advance is read by the information acquisition unit 21 . A three-dimensional map is generated by the map generation unit 22 based on the sensing result of the sensor unit 14 .

A predetermined direction is selected from the three-dimensional map by the user 5 (step 402). The candidate plane detector 23 selects a candidate plane based on the selected predetermined direction.

The reference point setting unit 24 sets the two-dimensional position of the reference point on the selected candidate plane (step 403).
Also, the reference point setting unit 24 vertically moves the set reference point by a predetermined distance from the two-dimensional position (step 404).

The operable range changing unit 25 changes the scale of the operable range based on the set reference point (step 405).

The virtual viewpoint setting unit 26 sets the position of the virtual viewpoint based on the reference point and the scale of the operable range (step 406).

<Third Embodiment>
In the first embodiment, the user 5 was able to observe the three-dimensional map from a virtual viewpoint and perform various operations using the virtual controller. In the third embodiment, the GUI presenting unit 27 presents a GUI displaying a virtual body including a virtual viewpoint and a virtual controller of the user.

FIG. 11 is a flowchart showing an example of control for setting the virtual body.

Three-dimensional measurement of the actual space is performed by the sensor unit 14 of the moving body 10 (step 501). Alternatively, a known map prepared in advance is read by the information acquisition unit 21 . A three-dimensional map is generated by the map generation unit 22 based on the sensing result of the sensor unit 14 .

A candidate plane is detected from the three-dimensional map by the candidate plane detector 23 (step 502). For example, the candidate plane detector 23 detects a plurality of candidate planes from planes perpendicular to a predetermined axial direction. Further, the candidate plane detected by the candidate plane detector 23 is presented to the user 5 (step 503). User 5 selects the presented candidate plane (step 504).

The selected candidate plane is moved on the axis by the virtual viewpoint setting unit 26 (step 505). For example, according to a predetermined axis such as the direction of gravity set by the user 5, the candidate plane is moved on the axis in the positive or negative direction.
The virtual viewpoint setting unit 26 designates the two-dimensional position of the head (virtual viewpoint) of the virtual body on the moved candidate plane (step 506).

The height of the virtual body is specified by the operable range changing unit 25 (step 507).
A virtual body has a head, a torso, arms and legs that contain a virtual viewpoint. Specifically, both arms include both hands that are virtual controllers. For example, the user 5 can operate the hands of the virtual body by moving the controller. That is, the arm length of the virtual body becomes the scale of the operable range. Also, the virtual body can be said to be an avatar of the user 5 .
In this embodiment, the length of both arms is preset based on the height of the virtual body. For example, a table or the like in which the height and the length of both arms are linked is recorded, and the operable range changing unit 25 changes the length of both arms (scale of the operable range) from the designated height according to the table. be done. Of course, the height and the length of both arms of the virtual body may be determined according to the relationship between the position of the HMD 31 worn by the user 5 and the position of the controller 32, for example.

A virtual body is set by the virtual viewpoint setting unit 26 and the operable range changing unit 25 based on the designated virtual body (step 508).

Note that the method of setting the virtual body is not limited. For example, instead of step 506, the position where the virtual body stands on the candidate plane may be designated. Further, for example, the height of the virtual body may be designated, and the virtual viewpoint and the operable range may be set according to a predetermined algorithm. Further, for example, the position of the head of the virtual body may be designated from the two-dimensional position of the candidate plane.

FIG. 12 is a schematic diagram showing an example of a virtual body. Note that FIG. 12 omits the lower half of the body such as the waist and legs of the virtual body.

As shown in FIG. 12, user 5 specifies the position of head 112 of virtual body 111 on candidate plane 110 . For example, the user 5 sets the center of the head 112 of the virtual body 111 on the predetermined axis 113 on the candidate plane 110 .

User 5 specifies the height of the virtual body. For example, the height can be specified so that the feet of the virtual body 111 are in contact with the candidate plane 110 . Also, in this embodiment, the size of the virtual body is determined by designating the height of the virtual body. For example, the lengths of the head 117 and both arms 118 of the virtual body 116 are determined to increase as the height of the virtual body 116 increases. Similarly, the smaller the height of the virtual body 111 (the smaller one), the smaller the lengths of the head 112 and both arms 113 of the virtual body 111 are determined.
The position of the eyes 114 (119) of the virtual body 111 (116) may also be determined based on the height of the virtual body. For example, the interpupillary distance may be obtained from the HMD 31 worn by the user 5, and the positions of the eyes (virtual viewpoint) of the virtual body may be determined from the positional relationship between the position (height) of the HMD 31 and the interpupillary distance. .
That is, the position of the virtual viewpoint and the scale of the operable range may be associated with the height of the virtual body.

Also, the hand 115 (120) of the virtual body 111 (116) has the function of a virtual controller. The user 5 can use the hand 115 (120) of the virtual body 111 (116) to select a candidate plane, generate a route for the moving body 10, and the like.
For example, when the user 5 stretches the arm to the maximum, the virtual body 111 (116) also stretches the arm 113 (118) to the maximum. The length of the arm 113 (118) of the virtual body 111 (116) at that time becomes the operable range 121 of the user 5. FIG. In this case, the scale of the operable range 121 is changed based on the height of the virtual body. Of course, if the arm is short (the height of the virtual body is small), the scale of the operable range 121 will also be small.

Note that control of the virtual viewpoint and the operable range of the virtual body is not limited. For example, even if the height of the virtual body is short, the arms of the virtual body may be set long. Further, for example, the operable range may be set to be equal to or longer than the arm length of the virtual body. In this case, when the user 5 tilts the body, the gyro sensor or the like mounted on the HMD 31 may acquire information about the body tilt, and the virtual body may similarly tilt.

FIG. 13 is a flowchart showing a control example for setting a new virtual body from virtual bodies.

The height of the virtual body is determined by steps 501 to 508 shown in FIG. 11 (step 601). That is, the position of the virtual viewpoint and the scale of the operable range are determined.
Here, the virtual body determined in step 601 is referred to as the first virtual body for convenience. A newly determined virtual body is referred to as a second virtual body.

The first virtual body is controlled by the operable range changing section 25 and the virtual viewpoint setting section 26 to set a new virtual body (step 602).
The position of the second virtual body is set by the operation of the user 5 via the virtual body (step 603). A specific method will be described with reference to FIG.

User 5 specifies the height of the second virtual body (step 604). Specifically, the operable range changing unit 25 and the virtual viewpoint setting unit 26 control the virtual viewpoint and the operable range based on the specified height of the virtual body.
The user 5 can change the operation from the first virtual body to the second virtual body. That is, various operations are possible using the second virtual body.

FIG. 14 is a schematic diagram showing a control example for setting the second virtual body. In FIG. 14, a specific example in steps 602 and 603 will be described.

FIG. 14A is a schematic diagram when setting a virtual body on a candidate plane.

As shown in FIG. 14A, the user 5 can select, as a candidate plane 132, a location designated by the first virtual body 130 with a finger 131 (virtual controller). For example, the user 5 can select a linear position from the hand 131 of the virtual body 130 as a candidate plane. In this case, the GUI presenting unit 27 presents a state in which a dotted line 133 like a laser pointer is emitted from the hand 131 of the virtual body 130 . A candidate plane 132 is selected by intersecting the dotted line 133 with a plane in the three-dimensional map.

A second virtual body 134 is set according to steps 505 to 508 from the selected candidate plane.

FIG. 14B is a schematic diagram showing an example of setting candidate planes.

As shown in FIG. 14B, user 5 can generate candidate plane 142 using hand 141 of first virtual body 140 . For example, when the user 5 moves the hand 141 of the virtual body 140 so as to draw a square in the air, the square can be generated as the candidate plane 142 .

The second virtual body 143 is set by designating the standing position and height of the second virtual body 143 from the generated candidate plane.
Note that the method for generating candidate planes is not limited. For example, the user 5 uses the controller 32 to select a candidate plane generation mode to generate candidate planes at arbitrary locations. Further, for example, candidate planes may be generated by performing a predetermined action such as spreading the hands of the virtual body.

FIG. 14C is a schematic diagram showing a control example for controlling the first virtual body. Also, in FIG. 14C, the three-dimensional map is omitted for simplification.

In the example shown in FIG. 14C, the case of setting the second virtual body at a position out of reach of the first virtual body 150 is taken as an example.

As shown in FIG. 14C, the first virtual body is controlled so that the entire three-dimensional map 150 can be overlooked. For example, the eye position of the first virtual body 151 (virtual position of the viewpoint). Further, for example, the operable range changing unit 25 may set the arm length (operable range scale) of the first virtual body based on the volume of the cuboid containing the generated three-dimensional map 150. good.

User 5 sets second virtual body 152 via controlled first virtual body 151 . For example, a point designated by the hand of the first virtual body 151 may be set as the candidate plane or the position of the head of the second virtual body 152 .

FIG. 14D is a schematic diagram showing a control example for controlling the first virtual body.
In FIG. 14D, the first virtual body 161 is enlarged so that the arms of the first virtual body 161 reach the entire three-dimensional map 160 . Specifically, the operable range changing unit 25 changes the scale of the operable range based on the size of the three-dimensional map 160 . That is, the arm length of the first virtual body 161 is changed. Also, the height of the first virtual body 161 is determined based on the changed arm length of the first virtual body 161 .

In this case, part of the controlled first virtual body 161 may interfere with the three-dimensional map 160 . For example, in FIG. 14D , the lower body of the first virtual body 161 may be drawn (represented) so as to penetrate the three-dimensional map 160 .
The user 5 sets the second virtual body 162 via the controlled first virtual body 161 .

Here, an example of an algorithm for determining a position during manipulation by a virtual body will be shown. Note that the position in this algorithm indicates the position of the user's eyes and the position of the virtual viewpoint, or the position of the controller and the position of the virtual controller.

When the virtual body is set, the user's position Pr0 in reality at that time, the user's position Pv0 in the three-dimensional map at that time, and the scale S of the virtual body are determined.
Also, when the user manipulates the virtual body such as by moving the hand or moving, the user's current position Prc in reality and the user's current position Pvc in the three-dimensional map are measured.

The virtual body scale indicates the set magnification of the virtual body. For example, when the height of the user is 1, the height of the set virtual body is S. Specifically, when the user is 170 cm and the height of the virtual body is increased to 340 cm, the scale is 2.

Here, the formula for obtaining the user's position Pvc on the current three-dimensional map is expressed as follows.
Pvc=S×(Prc−Pr0)+Pv0

The above formula is executed for every rendered frame. As a result, the virtual controller can be operated in conjunction with the operation of the controller 32 by the user 5 .

This enables intuitive operation even in a virtual space such as a three-dimensional map.
In addition, since the operable range is set within the reach of the controller held by the user, that is, within the reach of the user's hand, it is possible to utilize the physical ability of humans to perceive space. Therefore, the three-dimensional position and orientation can be specified accurately with a small number of steps and at a low learning cost.

<Other embodiments>
The present technology is not limited to the embodiments described above, and various other embodiments can be implemented.

In the above embodiment, the virtual viewpoint is used as the eyes of the virtual body, and the operable range is used as the arm length of the virtual body. Instead of being limited to this, the coordinates of the reference point may be set as the waist position of the virtual body.

In the above embodiment, the virtual viewpoint was set within the operable range. The virtual viewpoint is not limited to this, and the virtual viewpoint may be set at an arbitrary position outside the operable range.

In the above embodiment, the virtual body was set on the candidate plane based on the direction of gravity. The virtual body is not limited to this, and the virtual body may be set on any candidate plane. For example, the virtual body may be set so that both feet are in contact with the wall or ceiling. In this case, the user does not need to look up when viewing the ceiling of the 3D map from the virtual viewpoint. That is, it becomes possible to reduce the physical burden on the user.

In the above embodiment, the virtual viewpoint and the operable range are determined by specifying the height of the virtual body. Without being limited to this, the virtual body may be determined based on the size of the 3D map.

FIG. 15 is a schematic diagram showing a setting example of a virtual body.

As shown in FIG. 15 , the map generation unit 22 generates a three-dimensional map 170 based on the sensing results obtained by the sensor unit 14 mounted on the mobile object 10 .

The candidate plane detection unit 23 detects candidate planes 171 that include the three-dimensional map 170 . For example, the candidate plane detection unit 23 detects the candidate plane 171 based on the area on the XY plane, YZ plane, or XZ plane of the three-dimensional map.

A reference point setting unit 24 sets a reference point at the center of gravity of the detected candidate plane. In this embodiment, the coordinates of the set reference point are the waist height of the virtual body.
The operable range changing section 25 changes the operable range based on the size of the three-dimensional map 170 . For example, the operable range changing unit 25 sets, as the operable range 174, the length up to the reach of the virtual body 172 when the virtual body 172 moves one step from the state in which both arms 173 are spread.
The height of the virtual body 172 is determined from the set waist height and arm length of the virtual body.

In the above embodiments, the reference point, virtual viewpoint, and operable range were controlled from the candidate plane. The control is not limited to this, and control may be performed so that a region specified by the user becomes the operable range.

FIG. 16 is a schematic diagram showing another example of control of the virtual viewpoint and the operable range.
As shown in FIG. 16, the user 5 designates an area 181 to be operated within the three-dimensional map 180 generated by the map generation unit 22 .

The operable range changing unit 25 changes the scale of the operable range 182 so that the specified area 181 is included. For example, the scale of the operable range 182 is changed so that it coincides with the major axis of the cylinder with the region 181 as the bottom.
The reference point setting unit 24 sets the reference point 183 based on the changed operable range 182 . For example, the reference point setting unit 24 sets the center of the operable range 182 as the reference point 183 .
The virtual viewpoint setting unit 26 sets the position of the virtual viewpoint 184 based on the scale of the operable range 182 and the position of the reference point.
Alternatively, the height of the virtual body may be determined so that the hand of the virtual body can reach the designated area 181 . In this case, the scale of the operable range 182 changed by the operable range changing unit 25 becomes the arm length of the virtual body. Also, the position of the reference point 183 set by the reference point setting unit 24 is the waist height of the virtual body. The position of the virtual viewpoint 184 set by the virtual viewpoint setting unit 26 is the head of the virtual body.

In the above embodiment, the 3D map was generated by the map generator 22 . It is not limited to this, and the map generation unit 22 may be mounted on the moving body 10 . That is, a three-dimensional map may be generated by the moving body 10 and supplied to the information processing device 20 .

In the above-described embodiment, a virtual body is used to express the operation of the user 5 and the image of the virtual viewpoint. It is not limited to this, and any expression may be used. For example, it may be possible to observe a 3D map from the viewpoint of a bird flying at the same height (same Z coordinate) as the virtual viewpoint.

In the above embodiment, the HMD 31 was used as the user device 30 . It is not limited to this, and a form terminal such as an AR (Augmented Reality) glass, a smartphone, or a tablet terminal may be used.

FIG. 17 is a schematic diagram showing the appearance of the HMD 31. As shown in FIG. FIG. 17A is a perspective view schematically showing the appearance of the HMD 31, and FIG. 17B is a perspective view schematically showing how the HMD 31 is disassembled.

The HMD 31 has a base portion 190 , a wearing band portion 191 , a headphone portion 192 , a front camera 193 , a display unit 194 and a cover portion 195 .

The base portion 190 is a member arranged in front of the left and right eyes of the user, and is provided with a forehead support portion 196 that contacts the forehead of the user.

The wearing band portion 191 is worn on the user's head. As shown in FIG. 8, the mounting band portion 191 has a temporal band 197 and a parietal band 198 . The temporal band 197 is connected to the base portion 190 and worn so as to surround the user's head from the temporal region to the occipital region. The parietal band 198 is connected to the temporal band 197 and worn to surround the user's head from the temporal region to the parietal region.

The headphone section 192 is connected to the base section 190 and arranged to cover the left and right ears of the user. The headphone section 192 is provided with left and right speakers. The position of the headphone section 192 can be controlled manually or automatically. The configuration for that purpose is not limited, and any configuration may be adopted.

The front camera 193 is provided as a stereo camera capable of capturing an image of the real space in front of the user. The forward camera 193 can generate a camera image in which the real space is captured.

The display unit 195 is inserted into the base portion 190 and positioned in front of the user's eyes. A display is arranged inside the display unit 195 . Any display device using liquid crystal, EL (Electro-Luminescence), or the like, for example, may be used as the display. Also, the display unit 195 is provided with a lens system (not shown) that guides the image displayed by the display to the left and right eyes of the user.

The cover portion 196 is attached to the base portion 190 and configured to cover the display unit 195 . The HMD 31 configured in this manner functions as an immersive head-mounted display configured to cover the user's field of view. For example, the HMD 31 displays a three-dimensional virtual space. By wearing the HMD 31, the user can experience virtual reality (VR) or the like.

FIG. 18 is a block diagram showing a hardware configuration example of the information processing device 20. As shown in FIG.

The information processing apparatus 20 includes a CPU 201, a ROM 202, a RAM 203, an input/output interface 205, and a bus 204 that connects these to each other. A display unit 206, an input unit 207, a storage unit 208, a communication unit 209, a drive unit 210, and the like are connected to the input/output interface 205. FIG.

The display unit 206 is a display device using liquid crystal, EL, or the like, for example. The input unit 207 is, for example, a keyboard, pointing device, touch panel, or other operating device. When input unit 207 includes a touch panel, the touch panel can be integrated with display unit 206 .

The storage unit 208 is a non-volatile storage device such as an HDD, flash memory, or other solid-state memory. The drive unit 210 is a device capable of driving a removable recording medium 211 such as an optical recording medium or a magnetic recording tape.

A communication unit 209 is a modem, a router, and other communication equipment for communicating with other devices that can be connected to a LAN, WAN, or the like. The communication unit 209 may use either wired or wireless communication. The communication unit 209 is often used separately from the information processing device 20 .
In this embodiment, the communication unit 209 enables communication with other devices via the network.

Information processing by the information processing apparatus 20 having the hardware configuration as described above is realized by cooperation of software stored in the storage unit 208 or the ROM 202 or the like and hardware resources of the information processing apparatus 20 . Specifically, the information processing method according to the present technology is realized by loading a program constituting software stored in the ROM 202 or the like into the RAM 203 and executing the program.

The program is installed in the information processing device 20 via the recording medium 211, for example. Alternatively, the program may be installed in the information processing device 20 via a global network or the like. In addition, any computer-readable non-transitory storage medium may be used.

An information processing method, a program, and a system according to the present technology are executed by linking a computer installed in a communication terminal with another computer that can communicate via a network, etc., and an information processing apparatus according to the present technology is constructed. may be

That is, the information processing method, program, and system according to the present technology can be executed not only in a computer system configured by a single computer, but also in a computer system in which a plurality of computers operate in conjunction with each other. In the present disclosure, a system means a set of multiple components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device housing a plurality of modules within a single housing, are both systems.

Execution of the information processing method, program, and system according to the present technology by a computer system, for example, when detection of a candidate plane, change of an operable range, setting of a virtual viewpoint, etc. are executed by a single computer, and It includes both cases where each process is executed by a different computer. Execution of each process by a predetermined computer includes causing another computer to execute part or all of the process and obtaining the result.

That is, the information processing method, program, and system according to the present technology can also be applied to a configuration of cloud computing in which a plurality of devices share and jointly process one function via a network.

The configurations of the candidate plane detection unit, the operable range change unit, the virtual viewpoint setting unit, and the like, the control flow of the communication system, and the like, which have been described with reference to the drawings, are merely one embodiment, and are within the scope of the present technology. and can be arbitrarily transformed. That is, any other configuration, algorithm, or the like for implementing the present technology may be employed.

Note that the effects described in the present disclosure are merely examples and are not limited, and other effects may be provided. The description of a plurality of effects above does not mean that those effects are necessarily exhibited at the same time. It means that at least one of the above-described effects can be obtained depending on the conditions, etc., and of course, effects not described in the present disclosure may also be exhibited.

It is also possible to combine at least two characteristic portions among the characteristic portions of the respective forms described above. That is, various characteristic portions described in each embodiment may be combined arbitrarily without distinguishing between each embodiment.

Note that the present technology can also adopt the following configuration.
(1)
An information processing method in which a computer system executes a control step of controlling the setting of a user's virtual viewpoint with respect to a real space and the setting of an operable range in which the user can perform an operation related to the real space. .
(2) The information processing method according to (1),
The information processing method, wherein the control step sets a first virtual viewpoint within the real space, and sets a second virtual viewpoint different from the first virtual viewpoint within the operable range.
(3) The information processing method according to (1) or (2),
The information processing method, wherein the control step changes a scale of the operable range.
(4) The information processing method according to any one of (1) to (3), further comprising:
An information processing method, comprising: detecting a candidate plane for setting the virtual viewpoint or the operable range from the real space.
(5) The information processing method according to (4),
The information processing method, wherein the control step sets a position a predetermined distance away from the detected candidate plane as the virtual viewpoint.
(6) The information processing method according to (4) or (5),
The information processing method, wherein the control step sets a position a predetermined distance away from the detected candidate plane as a reference point serving as a reference of the operable range.
(7) The information processing method according to (6),
The information processing method, wherein the control step controls the virtual viewpoint or the operable range based on the set reference point.
(8) The information processing method according to (6), further comprising:
a setting step of setting the position of the virtual viewpoint and the scale of the operable range based on the size of the real space;
In the information processing method, the control step changes the set position of the virtual viewpoint and the set scale of the operable range with reference to the reference point.
(9) The information processing method according to (4),
The information processing method, wherein the detection step detects the candidate plane based on a predetermined axis of the real space.
(10) The information processing method according to any one of (1) to (9),
The information processing method, wherein the control step changes a scale of the operable range based on the position of the virtual viewpoint.
(11) The information processing method according to any one of (1) to (10),
The information processing method, wherein the control step sets the position of the virtual viewpoint based on the scale of the operable range.
(12) The information processing method according to any one of (1) to (11), further comprising:
An information processing method, comprising: presenting to the user a GUI (Graphical User Interface) capable of controlling the virtual viewpoint and the operable range.
(13) The information processing method according to (12),
The presenting step presents a virtual viewpoint image when the user views the real space from the virtual viewpoint,
The GUI is capable of setting a first virtual viewpoint within the real space and setting a second virtual viewpoint different from the first virtual viewpoint within the operable range. Information processing method .
(14) The information processing method according to (12) or (13),
The information processing method, wherein the GUI can set the candidate plane within the operable range.
(15) The information processing method according to any one of (1) to (14),
The information processing method, wherein the real space is a three-dimensional map created by a sensor.
(16) The information processing method according to (15),
The information processing method, wherein the control step changes a scale of the operable range based on the three-dimensional map created by the sensor.
(17) The information processing method according to (15),
The information processing method, wherein the sensor is mounted on a moving object.
(18) The information processing method according to (17),
The information processing method, wherein the GUI is capable of generating a moving route of the moving object by the user's operation.
(19)
A program for causing a computer system to execute a control step of controlling setting of a user's virtual viewpoint with respect to a real space and setting of an operable range in which the user can perform an operation related to the real space.
(20)
a moving body that moves in a real space;
an information processing apparatus having a control unit that controls setting of a user's virtual viewpoint for the real space and setting of an operable range in which the user can perform an operation regarding the real space. information processing system.

5 User 6 Virtual Viewpoint 7 Operable Range 10 Mobile 14 Sensor Unit 20 Information Processing Device 23 Candidate Plane Detector 24 Reference Point Setting Unit 25 Operable Range Changing Unit 26 Virtual Viewpoint Setting Unit 100... Virtual expression system

Claims (20)

An information processing method in which a computer system executes a control step of controlling the setting of a user's virtual viewpoint with respect to a real space and the setting of an operable range in which the user can perform an operation related to the real space. . The information processing method according to claim 1,
The information processing method, wherein the control step sets a first virtual viewpoint within the real space, and sets a second virtual viewpoint different from the first virtual viewpoint within the operable range. The information processing method according to claim 1,
The information processing method, wherein the control step changes a scale of the operable range. The information processing method according to claim 1, further comprising:
An information processing method, comprising: detecting a candidate plane for setting the virtual viewpoint or the operable range from the real space. The information processing method according to claim 4,
The information processing method, wherein the control step sets a position a predetermined distance away from the detected candidate plane as the virtual viewpoint. The information processing method according to claim 4,
The information processing method, wherein the control step sets a position a predetermined distance away from the detected candidate plane as a reference point serving as a reference of the operable range. The information processing method according to claim 6,
The information processing method, wherein the control step controls the virtual viewpoint or the operable range based on the set reference point. The information processing method according to claim 6, further comprising:
a setting step of setting the position of the virtual viewpoint and the scale of the operable range based on the size of the real space;
In the information processing method, the control step changes the set position of the virtual viewpoint and the set scale of the operable range with reference to the reference point. The information processing method according to claim 4,
The information processing method, wherein the detection step detects the candidate plane based on a predetermined axis of the real space. The information processing method according to claim 1,
The information processing method, wherein the control step changes a scale of the operable range based on the position of the virtual viewpoint. The information processing method according to claim 1,
The information processing method, wherein the control step sets the position of the virtual viewpoint based on the scale of the operable range. The information processing method according to claim 1, further comprising:
An information processing method, comprising: presenting to the user a GUI (Graphical User Interface) capable of controlling the virtual viewpoint and the operable range. The information processing method according to claim 12,
The presenting step presents a virtual viewpoint image when the user views the real space from the virtual viewpoint,
The GUI is capable of setting a first virtual viewpoint within the real space, and setting a second virtual viewpoint different from the first virtual viewpoint within the operable range. Information processing method . The information processing method according to claim 12,
The information processing method, wherein the GUI can set the candidate plane within the operable range. The information processing method according to claim 1,
The information processing method, wherein the real space is a three-dimensional map created by a sensor. The information processing method according to claim 15,
The information processing method, wherein the control step changes a scale of the operable range based on the three-dimensional map created by the sensor. The information processing method according to claim 15,
The information processing method, wherein the sensor is mounted on a moving object. The information processing method according to claim 17,
The information processing method, wherein the GUI is capable of generating a moving route of the moving object by the user's operation. A program for causing a computer system to execute a control step of controlling setting of a user's virtual viewpoint with respect to a real space and setting of an operable range in which the user can perform an operation related to the real space. a moving body that moves in a real space;
an information processing apparatus having a control unit that controls setting of a user's virtual viewpoint for the real space and setting of an operable range in which the user can perform an operation related to the real space. information processing system.

Images