WO2022176450A1

WO2022176450A1 - Information processing device, information processing method, and program

Info

Publication number: WO2022176450A1
Application number: PCT/JP2022/000900
Authority: WO
Inventors: 良徳大橋; 大佑松本
Original assignee: ソニーグループ株式会社
Priority date: 2021-02-22
Filing date: 2022-01-13
Publication date: 2022-08-25

Abstract

[Problem] To provide an information processing device, an information processing method, and a program with which it is possible to enable information of the virtual object to be operated on by a user to be handled in a geographical coordinate system so as to realize further improvement of user experience. [Solution] An information processing device comprising a control unit that exercises control on displaying a virtual object correlated to a real space. The control unit converts virtual space position information of the virtual object having been operated on by a user to geographical coordinate information on the basis of geographical coordinate information of the information processing device having been acquired on the basis of captured image in which the real space is imaged and virtual space position information that corresponds to the geographical coordinate information.

Description

Information processing device, information processing method, and program

The present disclosure relates to an information processing device, an information processing method, and a program.

In recent years, various technologies have been developed to fuse and view real and virtual spaces. For example, augmented reality (AR) technology has been developed as a technology that superimposes images (called virtual objects), which are virtual space information, on the real space in a state in which the user can directly see the real space. It is Display devices that provide augmented reality include, for example, an optically transmissive HMD (Head Mounted Display).

Also, when displaying a virtual object superimposed on the real space, the real space can be recognized by analyzing the image captured by the camera. For example, Patent Literature 1 below discloses a technique of recognizing a real space based on a captured image and creating an environment map.

Japanese Patent No. 5380789

Here, it was difficult to link the virtual object information processed in the virtual space coordinate system with the map service that manages the information in the geographic coordinate system.

Therefore, in the present disclosure, an information processing device, an information processing method, and an information processing device capable of further improving the user experience by enabling information of a virtual object, which is a target of user operation, to be handled in a geographic coordinate system, and Suggest a program.

According to the present disclosure, a control unit is provided for controlling display of a virtual object associated with a real space, and the control unit receives virtual space position information of the virtual object, which is a target operated by a user, Proposal of an information processing device that converts into geographic coordinate information based on geographic coordinate information of the information processing device acquired based on a captured image of a real space and virtual space position information corresponding to the geographic coordinate information do.

According to the present disclosure, the processor displays the virtual object associated with the real space, and the virtual space position information of the virtual object, which is the target of the user's operation, is captured in the captured image of the real space. and converting into geographical coordinate information based on the virtual space position information corresponding to the geographical coordinate information of the information processing device acquired based on the above.

According to the present disclosure, a computer is caused to function as a control unit that controls display of a virtual object associated with a real space, and the control unit controls the virtual space of the virtual object that is the target of the user's operation. A program for converting location information into geographic coordinate information based on geographic coordinate information of an information processing device acquired based on a captured image of real space and virtual space location information corresponding to the geographic coordinate information. Suggest.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram illustrating an overview of a cooperation system between display of a virtual object and a map service according to an embodiment of the present disclosure; FIG. 5 is a diagram showing an example of a screen on which POI information and icon images of virtual objects are arranged on a map image in the operation terminal according to the present embodiment; FIG. 10 is a diagram illustrating a case of operating a virtual object using the operating terminal according to the embodiment; 1 is a block diagram showing an example of the configuration of an AR display device and an operation terminal according to this embodiment; FIG. 4 is a sequence diagram showing the flow of position estimation processing based on captured images according to the present embodiment; FIG. FIG. 4 is a sequence diagram showing the flow of AR display processing in the AR display device according to this embodiment; FIG. 10 is a sequence diagram showing the flow of processing for displaying a layout of virtual objects on a map in the operation terminal according to the present embodiment; FIG. 10 is a sequence diagram showing the flow of placement processing of a new virtual object according to this embodiment; FIG. 10 is a sequence diagram showing the flow of information update processing of a virtual object operated on the operating terminal according to the embodiment; It is a figure explaining an example of selection operation of the virtual object using the operation terminal by this embodiment. It is a figure explaining an example of selection operation of the virtual object using the operation terminal by this embodiment. FIG. 10 is a diagram illustrating an example of an operation for changing the position of a virtual object using the operating terminal in the position/rotation editing mode according to the embodiment; FIG. 10 is a diagram illustrating an example of an operation for changing the position of a virtual object using the operating terminal in the position/rotation editing mode according to the embodiment; FIG. 11 is a diagram illustrating an example of a pitch/yaw rotation operation of a virtual object using the operation terminal in the position/rotation edit mode according to the embodiment; FIG. 10 is a diagram illustrating an example of a roll rotation operation of a virtual object using the operation terminal in the position/rotation edit mode according to the embodiment; FIG. 10 is a diagram illustrating an example of a virtual object enlargement/reduction operation using the operation terminal in the rotation/scale edit mode according to the present embodiment; FIG. 11 is a diagram illustrating an example of a Roll rotation operation of a virtual object using the operation terminal in the rotation/scale editing mode according to the embodiment; FIG. 11 is a diagram illustrating selection determination of other operation options for a virtual object in a selected state according to the present embodiment;

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

Also, the description shall be given in the following order.
1. Overview 2. Configuration example 3. Operation processing 3-1. Position estimation based on captured image 3-2. AR display processing in AR display device 10 3-3. Arrangement display of virtual objects on map on operation terminal 20 3-4. Placement of New Virtual Object 3-5. 3. Updating information of virtual objects operated by the operation terminal 20; 5. Operation example of a virtual object using the operation terminal 20; supplement

<<1. Overview＞＞
As an embodiment of the present disclosure, a map service (specifically, a map data server) that manages geographic information using virtual object information and location information (geographic coordinate information) in a geographic coordinate system that indicates at least latitude and longitude. An example of an external device) will be described. A virtual object is an image (2D image or 3DCG) presented by a display device as virtual space information. Further, in the display device, interaction in the virtual space can be realized by changing the position, posture, and display mode of the virtual object according to the user's operation input to the virtual object.

(Organization of issues)
Here, in response to a desire to fix the result of an operation performed by the user in the virtual space (for example, moving a virtual object, setting a new virtual object, etc.) to the location where the operation was performed, the coordinate system of the virtual space It is possible to specify the position in the virtual space, but since the virtual space position information is not linked to the latitude and longitude information, it is fixed to the location in the real space (real space) as one of the geographic information on the map service It was difficult to handle. In addition, it is possible to use the GPS (Global Positioning System) function provided on smartphones, etc., but the GPS function has low position accuracy, and when assuming cooperation with a map service, the position in the real space and the position on the map There is a risk that the deviation from the position will become large.

Therefore, in one embodiment according to the present disclosure, the information of the virtual object, which is the target of the user's operation, can be handled in a geographic coordinate system, so that it is reflected in a device that manages geographic information (for example, a map service). etc., to further improve the user experience.

(System configuration)
FIG. 1 is a diagram illustrating an overview of a cooperation system between display of a virtual object and a map service according to an embodiment of the present disclosure. In this embodiment, as an example, a case of using so-called AR technology for superimposing and displaying a virtual object in real space will be described.

As shown in FIG. 1, the cooperation system according to the present embodiment includes an AR display device 10 that displays a virtual object, an operation terminal 20 that receives an operation input for the virtual object, and position estimation based on feature points in the real space. and a map data server 40 for providing geographic images and the like.

(AR display device 10)
The AR display device 10 is an example of an information processing device that superimposes and displays a virtual object on a real space (so-called AR display). The AR display device 10 may be, for example, an HMD worn on the user's head, a terminal held by the user such as a smartphone, a mobile phone terminal, a tablet terminal, or a wearable device worn by the user. There may be. Also, the AR display by the AR display device 10 may be a method in which a virtual object is displayed on a transmissive display in a state in which the real space is directly visible through the transmissive display. In addition, the AR display by the AR display device 10 displays a front real space image (captured image) acquired from a camera on a non-transmissive display in real time (using a video see-through display that performs so-called through display), A method of superimposing and displaying a virtual object on an image may be used.

Also, the AR display device 10 according to the present embodiment communicates with the position estimation server 30 and the map data server 40 via a network. The AR display device 10 transmits captured images of the surrounding area or feature points (in the real space) extracted from the captured images to the position estimation server 30, and acquires position information (geographical coordinate information) of the current location from the position estimation server 30. . In this embodiment, more accurate position estimation is realized by using position estimation based on captured images. Acquisition of position information from the position estimation server 30 based on the captured image is performed, for example, at a frequency of about once per second. The AR display device 10 may further utilize self-position recognition (for example, VIO (Visual Inertial Odometry)) using captured images and motion sensor data for relative positional changes in a short period of time.

The AR display device 10 also transmits the geographical coordinate information acquired from the position estimation server 30 to the map data server 40 and acquires the surrounding geographical information from the map data server 40 . The geographic information according to the present embodiment includes information about virtual objects (virtual object-related information), and the AR display device 10 is fixed to a location in the real space based on the geographic information acquired from the map data server 40. AR display of virtual objects may be performed. In addition, the AR display device 10 is connected to the operation terminal 20 for wired or wireless communication, and controls the display of virtual objects according to the user's operation input using the operation terminal 20, thereby realizing interaction in the virtual space. .

(Operation terminal 20)
The operation terminal 20 may be a dedicated controller, a wearable device worn on a user's hand or foot, or a smart phone (a general-purpose communication terminal). Note that the location information (geographical coordinate information) of the current location may be continuously transmitted from the AR display device 10 to the operation terminal 20 (for example, at a frequency of about 60 times per second), or the operation terminal 20 may capture images of the surroundings. An image or a feature point (in a real space) extracted from the captured image may be transmitted to the position estimation server 30 and position information (geographical coordinate information) of the current location may be obtained from the position estimation server 30 . The operation terminal 20 also transmits the geographical coordinate information of the current location to the map data server 40, acquires the surrounding geographical information (map image, POI information, virtual object related information) from the map data server 40, and converts the acquired information to It is also possible to display it on the display unit of the operation terminal 20 .

(Position estimation server 30)
The position estimation server 30 is an information processing device that collates the feature points of the real object extracted from the captured image with the point cloud database 301 and performs position estimation (acquisition of geographical coordinate information). The position estimation server 30 is also called a so-called VPS (Visual Positioning System) that measures positions from images. Also, the location estimation server 30 may be a cloud server composed of a plurality of servers.

The position estimation server 30 has a point cloud database 301 constructed by collecting feature points (three-dimensional information) of objects (real objects) in real space in advance. The point cloud database 301 is a database in which feature points (three-dimensional information) of objects in real space are associated with geographic coordinate information. In this embodiment, the geographic coordinate information is, for example, latitude/longitude/altitude information. In the present embodiment, the AR display device 10 or the operation terminal 20 transmits the captured image of the surrounding area or the feature points obtained from the captured image to the position estimation server 30, and the position estimation server 30 checks the point cloud database 301. This makes it possible to perform position estimation with higher precision than GPS. By collating the point cloud database 301, the position estimation server 30 obtains latitude/longitude/altitude information (geographical coordinate information), which is the absolute geographical position information of the AR display device 10 or the operation terminal 20, and attitude (for example, azimuth). specify (implementation of so-called localization). Note that the details of position estimation are not particularly limited. The technology for constructing the point cloud database 301 is not particularly limited, either. By matching with the feature point group extracted from the captured image, the feature points and the geographic coordinate information can be associated in advance.

Correlation between Real Space and Virtual Space In the point cloud database 301 according to the present embodiment, feature points (three-dimensional information) of an object in the real space are further associated with virtual space position information. That is, the correspondence between the real space and the virtual space is made in advance in the point cloud database 301 . In this specification, the virtual space position information is xyz coordinate information in the coordinate system of the virtual space. Such xyz coordinates indicate a position within a 10m to 100m square area, for example, with a certain geographic location as the origin.

As a result, the AR display device 10 or the operation terminal 20 transmits to the position estimation server 30 the captured image of the surroundings (or the information of the feature points extracted from the captured image). geographic coordinate information (latitude/longitude/altitude information) and attitude information (for example, azimuth), and the corresponding virtual space position coordinate information (xyz coordinates) and attitude information (for example, rotation matrix) can be obtained. Further, the current location information acquisition may be performed, for example, at a frequency of about once per second.

(Map data server 40)
The map data server 40 is an information processing device (map service server) that manages geographical information based on geographical coordinate information (latitude and longitude information). The map data server 40 may be a cloud server consisting of multiple servers.

Geographic information is information (text, image) about a place, such as a map image (either a 2D map image or a 3D map image), POI (point of interest) information (names of surrounding buildings and shops). and marks, including geographic coordinate information), and virtual object related information (virtual object ID, 3D image data, placed geographic coordinate information and posture information, sound data, movement of the virtual object, etc. defined scripts, etc.). That is, in the present embodiment, information on virtual objects is also handled as information on locations (virtual objects are fixed at locations in real space by geographic coordinate information). This makes it possible to display, for example, on a map image, the arrangement of virtual objects in addition to general POI information. The placement display of the virtual objects on the map will be described later with reference to FIG.

As shown in FIG. 1, the map data server 40 according to the present embodiment includes a map image database 401 storing map images associated with geographical coordinate information and POI information associated with the geographical coordinate information. and an AR database 403 that stores virtual object-related information associated with geographical coordinate information. The map data server 40 can return surrounding map image data, POI information, and virtual object-related information based on the geographical coordinate information (at least latitude and longitude information) transmitted from the AR display device 10 or the operation terminal 20. . The “surroundings” may be, for example, an area of about 100 m square around the user's current location (latitude and longitude). The AR display device 10 or the operation terminal 20 may acquire the peripheral information from the map data server 40 once, and then acquire the peripheral information again after moving to some extent. Further, when the map data server 40 updates the information in the area, the updated information may be notified from the map data server 40 to the AR display device 10 or the operation terminal 20 .

The AR display device 10 or the operation terminal 20 can superimpose POI information (building names, etc.) and virtual objects received from the map data server 40 on the real space for AR display. At this time, for example, the AR display device 10 can realize AR display by converting the POI information and the geographical coordinate information of the virtual object received from the map data server 40 into virtual space position information. The details of the conversion will be described using mathematical formulas to be described later. The virtual space position information is calculated using the virtual space position information of the AR display device 10 based on the physical positional relationship. An example of AR display of a virtual object on the AR display device 10 and an operation of the virtual object using the operation terminal 20 will be described later with reference to FIG.

(Regarding the placement and display of virtual objects on the map)
FIG. 2 is a diagram showing an example of a screen on which POI information and icon images of virtual objects are arranged on a map image in the operation terminal 20 according to this embodiment.

As described above, in the present embodiment, the virtual object information, which is an example of the surrounding geographic information received from the map data server 40, includes geographic coordinate information (at least latitude and longitude information). 20 displays icon images (

icon images

252a, 252b, . can be expressed as This allows the user to intuitively grasp the position of the virtual object associated with the location in the real space on the map. Note that the map image used is not limited to the 2D image shown in FIG. 2, and may be a 3DCG image.

The geographical coordinate information of the current location, which the operating terminal 20 transmits to the map data server 40 in order to obtain the geographical information of the surrounding area, may be the information received from the AR display device 10, or may be the information received from the AR display device 10. Information obtained from the server 30 may be used.

In addition, the operation terminal 20 transmits not only the geographical information around the current location, but also arbitrary latitude and longitude information, address, station name, etc. (POI information) input by the user to the map data server 40, and provides geographical information around an arbitrary location. Information may be obtained and displayed.

Also, the POI information and the placement display of the virtual objects on the map may be displayed not only on the display unit of the operation terminal 20, but also on the display unit of the AR display device 10.

(Operation of virtual object performed using operation terminal 20)
FIG. 3 is a diagram illustrating a case of operating a virtual object using the operating terminal 20 according to this embodiment. As shown in FIG. 3, the operation terminal 20 is, for example, a smart phone, and can be held by a user. Also, the AR display device 10 is realized by a transmissive spectacle type device in which a transmissive display is provided in the lens portion, as shown in FIG.

　On the AR display device 10 worn by the user, virtual objects 50a to 50c are superimposed and displayed in the real space (the user's field of view), as shown in FIG. At this time, the user uses the operation terminal 20 to select at least one of the virtual objects 50a to 50c, and continues to use the operation terminal 20 to move, delete, copy, or copy the selected virtual object 50. Various operations such as posture control and display mode (size, color, shape, etc.) change can be performed. Specifically, the user can operate the virtual object 50 by moving the operation terminal 20 or performing a touch operation on the operation display section 250 of the operation terminal 20 . The movement of the operation terminal 20 (motion data detected by the motion sensor 240) and the user's touch operation (single-tap, double-tap, long press, swipe, pinch-in/pinch-out, etc.) on the operation display unit 250 of the operation terminal 20 , is continuously transmitted to the AR display device 10 . In addition, the AR display device 10 continuously captures images of the user's visual field range with the camera 130 provided facing outward, and analyzes the captured images (visual field images) (extraction of feature points, acquisition of depth information, etc.). , recognition of surrounding real objects (object recognition), recognition of three-dimensional space, tracking, and the like are performed.

As shown in FIG. 3, the AR display device 10 displays a pointing image L like a ray extending in a straight direction from the tip (upper end) of the operating terminal 20, and a virtual object 50c that collides with the pointing image L. It can be selected as an operation target. Since the instruction image L follows the direction of the operation terminal 20, the user can control the direction of the instruction image L by moving the top end (upper end) of the operation terminal 20 up, down, left, or right to select an arbitrary virtual object 50. can do. The selection is determined by, for example, tapping the operation display unit 250 of the operation terminal 20 .

The AR display device 10 performs AR display processing according to the user operation based on the tracking result of the operation terminal 20 obtained by analyzing the captured image and operation information such as motion data and touch operation transmitted from the operation terminal 20. Realize interaction in virtual space. In addition, when performing AR display processing according to a user operation, the AR display device 10 appropriately uses the virtual space position information and orientation information of the AR display device 10 and the operation terminal 20 and the virtual space position information and orientation information of the virtual object. may be used.

(storage of results of user operations on virtual objects)
The AR display device 10 according to this embodiment can transmit the result of operating a virtual object in the virtual space to the map data server 40 as update information and store it in the AR database 403 . By reflecting the result of the operation by the user in the AR database 403 of the map data server 40, it becomes possible to share the result of the operation with other users. The result of the operation is, for example, placement of a new virtual object, movement, deletion, copy of the virtual object, control of posture, change in display mode, and the like. The AR display device 10 transmits virtual object update information (virtual object ID, geographic coordinate information, orientation information, shape features, 3DCG, etc.) to the map data server 40 . Note that the AR display device 10 uses position information and orientation information of the virtual space when AR displaying a virtual object. The information (rotation matrix) is converted into geographical coordinate information (latitude, longitude, elevation) and attitude information (azimuth angle) and transmitted to the map data server 40 . The details of the conversion will be explained using mathematical formulas described later.

Note that the operation result may be saved from the operation terminal 20 to the map data server 40. For example, the user operates the operation terminal 20 to generate a new virtual object (for example, by inputting a character to generate a 3DCG of the character), and the AR display performed on the display unit of the operation terminal 20 is performed on the generated virtual object. The location is fixed by arranging it in the real space using it or by arranging it on the map displayed on the display unit of the operation terminal 20 . The operation terminal 20 transmits the generation and placement of such a new virtual object as update information (operation result) to the map data server 40 and stores it in the AR database 403 . In this case also, the virtual space position information (xyz coordinates) and orientation information (rotation matrix) of the new virtual object are converted into geographic coordinate information (latitude, longitude, elevation) and orientation information (azimuth), and the map data It is sent to the server 40 .

The overview of the cooperation system according to this embodiment has been described above. Next, specific configuration examples and operation processing of the AR display device 10 and the operation terminal 20 included in the cooperation system will be sequentially described.

<<2. Configuration example >>
FIG. 4 is a block diagram showing an example of the configuration of the AR display device 10 and the operation terminal 20 according to this embodiment. The AR display device 10 and the operation terminal 20 are connected for communication by wire or wirelessly, and transmit and receive data.

<2-1. AR display device 10>
As shown in FIG. 4 , the AR display device 10 has a communication section 110 , a control section 120 , a camera 130 , a motion sensor 140 , a display section 150 and a storage section 160 . The AR display device 10 is an example of an information processing device that displays a virtual object associated with a real space.

(Communication unit 110)
The communication unit 110 communicates with an external device by wire or wirelessly, and transmits and receives data. For example, the communication unit 110 connects to a network and transmits/receives data to/from a server on the network. In addition, the communication unit 110, for example, wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), mobile communication network (LTE (Long Term Evolution), 3G (3rd generation mobile communication system), 4G (fourth-generation mobile communication system), 5G (fifth-generation mobile communication system)), etc., to connect with external devices and networks. The communication unit 110 according to this embodiment can transmit and receive data to and from the operation terminal 20 , the position estimation server 30 , and the map data server 40 .

(control unit 120)
The control unit 120 functions as an arithmetic processing device and a control device, and controls overall operations within the AR display device 10 according to various programs. The control unit 120 is realized by an electronic circuit such as a CPU (Central Processing Unit), a microprocessor, or the like. The control unit 120 may also include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

The control unit 120 according to the present embodiment analyzes (space recognition, etc.) captured images of the surrounding area acquired by the camera 130, motion data acquired by the motion sensor 140, virtual space position information acquired from the position estimation server 30, And based on the virtual object related information and the like acquired from the map data server 40, the display unit 150 performs control (AR display control) to superimpose and display the virtual object on the real space. Main functions of the control unit 120 will be described below.

The control unit 120 according to this embodiment also functions as a coordinate information acquisition unit 121, a coordinate information conversion unit 122, and a virtual space processing unit 123.

The coordinate information acquisition unit 121 transmits to the position estimation server 30 the captured image of the surrounding area acquired by the camera 130 or the data of the feature points extracted from the captured image. A process of acquiring geographic coordinate information (latitude, longitude, elevation) and corresponding virtual space position information (xyz coordinates) is performed. In addition, the coordinate information acquisition unit 121 can also acquire geographical orientation information (eg, azimuth angle) of the AR display device 10 and virtual space orientation information (eg, rotation matrix) from the position estimation server 30 . For example, when the AR display device 10 is worn on the user's head, the posture information corresponds to the user's head orientation (face orientation). Acquisition of position information and orientation information of the AR display device 10 in the geographic and virtual spaces is performed, for example, every second. In addition, the coordinate information acquisition unit 121 obtains analysis results (tracking, etc.) of the captured image acquired by the camera 130 and a motion sensor for relative positional changes in a short period of time based on the acquired position information and orientation information. Self-localization using 140 motion data (for example, VIO (Visual Inertial Odometry)) may be utilized.

The coordinate information conversion unit 122 performs mutual conversion between geographic coordinate information (latitude, longitude, elevation) and attitude information (azimuth) and virtual space position information (xyz coordinates) and attitude information (rotation matrix). A specific calculation method will be described later using mathematical formulas. As a result, geographical coordinate information can be associated with a virtual object and managed, realizing cooperation with a map service. For example, the results of operations performed in the virtual space can be reflected in the map service and shared with other users.

The virtual space processing unit 123 performs AR display control to superimpose and display the virtual object on the real space. Algorithms for AR display control are not particularly limited, but the following processing is performed, for example. The virtual space processing unit 123 analyzes the captured image of the user's field of view (an example of the captured image of the surroundings) acquired by the camera 130, and recognizes the real space. Real space recognition includes, for example, three-dimensional object recognition, plane area detection, and tracking. The virtual space processing unit 123 may, for example, extract and track feature points from the captured image, and perform recognition processing of the three-dimensional space. The virtual space processing unit 123 may further acquire depth information from the camera 130 and use it for recognizing the real space. The virtual space processing unit 123 may further acquire motion data (motion of the AR display device 10 itself) from the motion sensor 140 and use it for real space recognition (particularly tracking).

The virtual space processing unit 123 is fixed at a location based on the virtual space position information and orientation information of the AR display device 10 acquired from the position estimation server 30 and the virtual object related information acquired from the map data server 40. To appropriately superimpose and display a virtual object (for example, 3DCG) on a real space. For example, the virtual space processing unit 123 can control the display position and posture of the virtual object according to the position of the plane area in the real space and the shape and position of the real object so as not to create an unnatural state. In addition, the virtual space processing unit 123 appropriately controls the display of the virtual object according to the operation information for the virtual object by the user transmitted from the operation terminal 20, and executes the interaction in the virtual space. The virtual space processing unit 123 also controls transmission of the operation result from the communication unit 110 to the map data server 40 .

(Camera 130)
The camera 130 is an imaging unit that has a function of imaging the real space. For example, when the AR display device 10 is implemented by a transmissive glasses type device as shown in FIG. 3, the camera 130 is a camera ( external camera). The angle of view of camera 130 is preferably an angle of view that includes at least the field of view of the user. As a result, the virtual space processing unit 123 can perform spatial recognition of the real space (field of view) that the user sees through the display unit 150 based on the captured image acquired by the camera 130 . Note that the camera 130 may be singular or plural. Camera 130 may also be configured as a so-called stereo camera.

(motion sensor 140)
The motion sensor 140 is a sensor that detects motion of the AR display device 10 . Motion sensor 140 includes, for example, an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor. Also, the motion sensor 140 may be a sensor (eg, an IMU (Inertial Measurement Unit)) including, for example, a 3-axis acceleration sensor and a 3-axis gyro sensor.

(Display unit 150)
The display unit 150 has a function of displaying images. For example, the display unit 150 is implemented by a transmissive display. A transmissive display is a display that allows light in the real space to be directly delivered to the user's eyes. In the case of a transmissive display, the user can directly view the real space through the transmissive display. A transmissive display may be, for example, an optical see-through display. The optical see-through display can employ known forms including a half-mirror system, a light guide plate system, a retinal direct drawing system, and the like. Further, the display unit 150 may be a so-called video see-through display that displays an image captured by a camera (captured image of the real space in front of the eyes) on a non-transmissive display.

The display unit 150 according to this embodiment displays, for example, a virtual object (eg, 3DCG) under the control of the control unit 120.

(storage unit 160)
The storage unit 160 is realized by a ROM (Read Only Memory) that stores programs and calculation parameters used in the processing of the control unit 120, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

Although the configuration of the AR display device 10 has been specifically described above, the configuration of the AR display device 10 according to the present disclosure is not limited to the example shown in FIG. For example, the AR display device 10 may be realized by multiple devices. Specifically, for example, a display device (including at least a display unit 150, a camera 130, and a motion sensor 140) realized by a transmissive glasses-type device or the like, and an information processing terminal realized by a smartphone, a tablet terminal, a PC, or the like (including at least the control unit 120).

In addition, the AR display device 10 may further have a speaker and a microphone. Audio information of the virtual space (for example, sound effects of virtual objects, etc.) can be reproduced from the speaker. The speakers may be configured as headphones, earphones, or bone conduction speakers, for example. Also, the microphone picks up the user's uttered voice and implements operation input by voice.

<2-2. Operation terminal 20>
As shown in FIG. 4 , operation terminal 20 has communication section 210 , control section 220 , camera 230 , motion sensor 240 , operation display section 250 and storage section 260 .

(Communication unit 210)
The communication unit 210 communicates with an external device by wire or wirelessly, and transmits and receives data. For example, the communication unit 210 connects to a network and transmits/receives data to/from a server on the network. In addition, the communication unit 210, for example, wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), mobile communication network (LTE (Long Term Evolution), 3G (third generation mobile communication system), 4G (fourth-generation mobile communication system), 5G (fifth-generation mobile communication system)), etc., to connect with external devices and networks. The communication unit 210 according to this embodiment can transmit and receive data to and from the AR display device 10, the position estimation server 30, and the map data server 40. FIG.

(control unit 220)
The control unit 220 functions as an arithmetic processing device and a control device, and controls overall operations within the operation terminal 20 according to various programs. The control unit 220 is implemented by an electronic circuit such as a CPU (Central Processing Unit), a microprocessor, or the like. The control unit 220 may also include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

The control unit 220 according to this embodiment also functions as a coordinate information acquisition unit 221, an operation information transmission control unit 222, and a display control unit 223.

The coordinate information acquisition unit 221 transmits to the position estimation server 30 the captured image of the surrounding area acquired by the camera 230 or the data of the feature points extracted from the captured image. A process of acquiring coordinate information (latitude, longitude and altitude) and corresponding virtual space position information (xyz coordinates) is performed. In addition, the coordinate information acquisition unit 221 can also acquire geographical attitude information (eg, azimuth angle) of the operation terminal 20 and virtual space attitude information (eg, rotation matrix) from the position estimation server 30 . For example, when the operation terminal 20 is a smartphone held by the user, the orientation information is the camera 130 provided on the operation terminal 20 (for example, the outward facing camera provided on the surface (back side) opposite to the surface on which the operation display unit 250 is provided). corresponds to the direction of The geographical and virtual space position information and attitude information of the operation terminal 20 are acquired every second, for example. Further, the coordinate information acquisition unit 221 analyzes captured images continuously acquired by the camera 230 and based on the motion data of the operation terminal 20 continuously acquired by the motion sensor 240, based on the relative coordinates in a short period of time. Calculation of position change (eg, utilizing VIO) may also be performed.

Here, as an example, it is described that the coordinate information acquisition unit 221 acquires the geographical coordinate information of the operation terminal 20 from the position estimation server 30, but the present embodiment is not limited to this, and the coordinate information acquisition unit 221 , for example, may obtain the geographical coordinate information of the AR display device 10 from the AR display device 10 and regard it as the coordinate information of the operation terminal 20 . The AR display device 10 may transmit highly accurate geographic coordinate information acquired using VIO to the operation terminal 20 at a high frequency (for example, about 60 times per second).

The operation information transmission control unit 222 transmits motion data (movement information of the operation terminal 20) acquired from the motion sensor 240 and information of user's touch operation acquired from the operation display unit 250 to the AR display device 10 as operation information. controls sending to. The operation information may be information of button operation or switch operation.

The display control unit 223 controls the display of the operation display unit 250. For example, the display control unit 223 may perform display control according to a user's touch operation acquired from the operation display unit 250 . Further, the display control unit 223 may transmit the geographical coordinate information of the operation terminal 20 to the map data server 40 and display the peripheral geographical information acquired from the map data server 40 on the operation display unit 250. . Specifically, as described with reference to FIG. 2, POI information and virtual objects may be arranged and displayed on the map image.

In addition, as shown in FIG. 3 , when a virtual object presented on the AR display device 10 is operated using the operation terminal 20, the display control unit 223 has a layout that facilitates operation without looking closely at the hand. Alternatively, a simple cross as shown in FIG. 3 may be displayed on the operation display unit 250. Alternatively, a tap operation or a swipe operation may be performed by placing a finger on an arbitrary position without arranging a specific button image on the operation display section 250 .

The display control unit 223 can also perform AR display in which a virtual object is superimposed on the captured image (video of the real space) acquired by the camera 230 . Information related to the display of such virtual objects is included in the peripheral geographic information acquired from the map data server 40 by the operating terminal 20 transmitting the geographic coordinate information to the map data server 40 . When the AR display is performed on the operation display unit 250, the user can select, move, delete, copy, control the posture, and display the virtual object by touching the virtual object displayed on the operation display unit 250. An operation input such as a mode change can be performed. The display control unit 223 can control the display of the virtual object and execute the interaction in the virtual space according to the user's touch operation. Touch operations are, for example, operations such as single tap, double tap, long press, swipe, drag & drop, pinch in/pinch out. In addition, the user can input characters from, for example, a software keyboard displayed on the operation display unit 250, generate a virtual object (3DCG) of the character, and perform control to newly place it at an arbitrary location in the real space. is also possible. The results of such operations performed in the virtual space are transmitted from the AR display device 10 or the operation terminal 20 to the map data server 40 and stored in the AR database 403 so that they can be shared with other users. becomes.

(Operation display unit 250)
The operation display unit 250 has an input function of attaching an operation input by the user and a display function of displaying an image. For example, the operation display unit 250 is realized by a touch panel display. Thereby, the operation display unit 250 can detect a touch operation on the display screen. As an example, the operation display unit 250 according to this embodiment is assumed to be a non-transmissive display.

(storage unit 260)
The storage unit 260 is implemented by a ROM (Read Only Memory) that stores programs, calculation parameters, and the like used in the processing of the control unit 220, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

Although the configuration of the operation terminal 20 has been specifically described above, the configuration of the operation terminal 20 according to the present disclosure is not limited to the example shown in FIG. For example, the operation terminal 20 may be realized by a general-purpose smart phone, a tablet terminal, a handy game machine, a wearable device worn on a finger or wrist, or the like.

<<3. Operation processing >>
Next, operation processing of the cooperation system according to this embodiment will be specifically described with reference to the drawings.

<3-1. Position Estimation Based on Captured Image>
FIG. 5 is a sequence diagram showing the flow of position estimation processing based on captured images according to the present embodiment. As shown in FIG. 5, first, the AR display device 10 acquires a captured image by the camera 130 (step S103), and transmits the captured image or feature points extracted from the captured image to the position estimation server 30 (step S106). .

Next, the position estimation server 30 collates the feature points of the captured image with the point cloud database 301, geographic coordinate information (latitude, longitude, elevation) and orientation information (for example, azimuth), and corresponding virtual space position information (xyz coordinates) and attitude information (for example, rotation matrix) are acquired (step S109).

Next, the position estimation server 30 transmits the acquired geographical coordinate information and the like and the virtual space position information and the like to the AR display device 10 (step S112).

Then, the AR display device 10 appropriately merges it with the data of the motion sensor 140 (using VIO) to obtain current position information with a higher frame rate (step S115).

In order to narrow down the area to be matched by referring to the point cloud database 301 on the side of the position estimation server 30, the AR display device 10 uses GPS information representing the rough position of the AR display device 10 and Wi-Fi position estimation data. The result, information on the connected mobile phone base station, or the like may be transmitted together with the captured image. Also, when 5G becomes popular, the location estimation server 30 can also narrow down the area by the unique ID of the edge server transmitted from the AR display device 10 .

<3-2. AR Display Processing in AR Display Device 10>
FIG. 6 is a sequence diagram showing the flow of AR display processing in the AR display device 10 according to this embodiment. As shown in FIG. 6 , first, the coordinate information acquisition unit 121 of the AR display device 10 acquires the geographic coordinate information of the AR display device 10 based on the captured image of the periphery (user's field of view) acquired by the camera 130 . and virtual space position information and the like are acquired (step S123). Specifically, the process (acquisition process from the position estimation server 30) described with reference to FIG. 5 is performed.

Next, the virtual space processing unit 123 of the AR display device 10 sets the acquired geographical coordinate information and the like and the virtual space position information and the like as the base position (current location) of the AR display device 10 (step S126).

Next, the virtual space processing unit 123 of the AR display device 10 transmits the geographical coordinate information (current location) to the map data server 40 (step S129), and receives POI information and virtual object related information around the current location from the map data server 40. Acquire (step S132). The virtual object-related information includes, for example, the ID of the virtual object, 3DCG data for displaying (or generating) the virtual object, 3D model, various parameters, geographic coordinate information and posture information of the virtual object, and furthermore, virtual This includes object sound effect information and the like. Note that the AR display device 10 may periodically acquire update information (difference) from the map data server 40 after receiving the virtual object related information in the area at the time of initial connection.

Next, the coordinate information conversion unit 122 of the AR display device 10 converts the POI information and the geographical coordinate information of the virtual object acquired from the map data server 40 into a virtual space in order to display them in a virtual space associated with the real space. Processing for conversion into position information is performed (step S135).

Here, the interconversion processing of coordinates will be explained. First, the position: PG (x, y, z) and the orientation (rotation: RG) is as shown in the following formula (Equation 1). Here, rotation information is generated by designating only the Yaw component as an example of the orientation information of the virtual space.

Geographical coordinate information (longitude1, latitude1, elevation1), which is the Geo Pose of the virtual object 1 obtained from the map data server 40 and the orientation ( Rotation) information (azimuth1) is converted into position information (PP1) and rotation (RP1) which are Pose in virtual space (dedicated coordinate system for each area). At this time, the geographic coordinate information (latitude, longitude, elevation) of the virtual object 1 is transformed into a position in the Euclidean space on a global scale: PG1 (x, y, z) and rotation information, and converted into virtual space (dedicated coordinates for each area can be converted into position information (PP1) and rotation (RP1), which is Pose in the system).

The calculation method described above is an example, and the present embodiment is not limited to this. For example, the coordinate information conversion unit 122 converts the virtual coordinates of the AR display device 10 so as to correspond to the relative positional relationship between the geographical coordinate information/attitude information of the AR display device 10 and the geographical coordinate information/attitude information of the virtual object 1 . Based on the spatial position information/posture information, the geographic coordinate information/posture information of the virtual object 1 can be converted into virtual space position information/posture information.

Then, the virtual space processing unit 123 performs AR display processing of the virtual object using the virtual space position information/orientation information of the virtual object (step S138).

<3-3. Arrangement Display of Virtual Objects on Map on Operation Terminal 20>
FIG. 7 is a sequence diagram showing the flow of processing for arranging and displaying virtual objects on a map in the operation terminal 20 according to this embodiment.

As shown in FIG. 7 , first, the coordinate information acquisition unit 221 of the operation terminal 20 acquires the geographical coordinate information and the like of the operation terminal 20 and the virtual space position information and the like based on the captured image of the surrounding area acquired by the camera 230 . Acquire (step S203). Specifically, it is obtained from the position estimation server 30 based on the feature points of the captured image. In addition, the coordinate information acquisition unit 221 may recognize the self-position at a higher frame rate using the motion data of the operation terminal 20 with respect to the relative positional change in a short time (for example, using VIO).

Next, the display control unit 223 of the operation terminal 20 sets the acquired geographical coordinate information and the like and the virtual space position information and the like to the basic position (current location) of the operation terminal 20 (step S206).

Next, the display control unit 223 of the operation terminal 20 transmits the geographical coordinate information (current location) to the map data server 40 (step S209), and from the map data server 40, the map image around the current location, the POI information, and the virtual object related Information is acquired (step S212). As in the case of the AR display device 10, the operation terminal 20 periodically acquires update information (difference) from the map data server 40 after receiving the virtual object related information in the area at the time of initial connection. good too.

Next, the display control unit 223 of the operation terminal 20 displays the POI information and the icon image of the virtual object (included in the virtual object related information) acquired from the map data server 40 as shown in FIG. 2 based on the geographical coordinate information. As shown, the operation display unit 250 performs processing for displaying on the map image (step S135). Thereby, the user can intuitively grasp the arrangement of the virtual objects associated with the real space, and can enjoy the integration of the real space and the virtual space.

<3-4. Placement of New Virtual Object>
FIG. 8 is a sequence diagram showing the flow of processing for arranging a new virtual object according to this embodiment. In this embodiment, as one of the operations related to the virtual object in the virtual space, the user can place (fix) the virtual object at an arbitrary location. The newly placed virtual object may be a 3DCG prepared in advance as a placeable virtual object, a 3DCG arbitrarily edited by the user, or a 3DCG newly generated by the user. good too. As a result, for example, AR advertisements (an example of virtual objects) can be placed near stores that exist in real space, various AR characters (an example of virtual objects) can be placed anywhere in a theme park, It is possible to place AR items (game items, etc., an example of virtual objects) and AR products (indicating that they can be purchased as data or as real items, an example of virtual objects) anywhere in becomes.

As shown in FIG. 8 , first, the coordinate information acquisition unit 121 of the AR display device 10 acquires the geographic coordinate information of the AR display device 10 based on the captured image of the periphery (user's field of view) acquired by the camera 130 . and virtual space position information and the like are acquired (step S303). Specifically, the process (acquisition process from the position estimation server 30) described with reference to FIG. 5 is performed.

Next, the virtual space processing unit 123 of the AR display device 10 sets the acquired geographical coordinate information and the like and the virtual space position information and the like as the base position (current location) of the AR display device 10 (step S306).

Next, the virtual space processing unit 123 of the AR display device 10 receives a new virtual object registration request (step S309). For example, the user uses the operation terminal 20 to move a new virtual object superimposed and displayed on the real space by the AR display device 10 and perform an operation to arrange the new virtual object at an arbitrary place in the real space.

Next, the coordinate information conversion unit 122 of the AR display device 10 transforms the virtual space position coordinate information of the newly placed virtual object into the base position (virtual space coordinate information etc. and geographic coordinate information etc.) of the AR display device 10. Based on this, processing for conversion into geographical coordinate information is performed (step S312).

Here, the interconversion processing of coordinates will be explained. The coordinate information conversion unit 122 converts the position (PP1) and rotation (RP1), which are Pose, of the virtual object 1 in the virtual space (dedicated coordinate system for each area) to geographic Convert to coordinate information (longitude1, latitude1, elevation1) and attitude (rotation) information (azimuth1).

The calculation method described above is an example, and the present embodiment is not limited to this. For example, the coordinate information conversion unit 122 converts the AR display device 10 so as to correspond to the relative positional relationship between the virtual space position information/posture information of the virtual object 1 and the virtual space position information/posture information of the AR display device 10 . The virtual space position information/attitude information of the virtual object 1 can be converted into the geographical coordinate information/attitude information based on the geographical coordinate information/attitude information.

Subsequently, the virtual space processing unit 123 of the AR display device 10 transmits information regarding the newly placed virtual object to the map data server 40 (step S315). The information to be transmitted (virtual object-related information) includes, for example, the ID of the virtual object, 3DCG data for displaying (or generating) the virtual object, the 3D model, various parameters, and the geographical coordinates converted in step S312. Information/posture information is included.

Then, the map data server 40 updates the AR database 403 with the virtual object-related information received from the AR display device 10 (step S318). Since the virtual object-related information includes the geographical coordinate information and the like of the newly placed virtual object, it can be reflected in the map data server 40 that manages the geographical information based on the geographical coordinate information. . Updates to the AR database 403 can be sent to other users within the area.

As described above, in the cooperation system according to the present embodiment, the information (operation result) of the virtual object newly arranged by the user can be reflected in the map data server 40 by performing the conversion process to the geographic coordinate information. .

Note that the AR display device 10 transmits the operation result (information about the newly placed virtual object) to the AR display devices of other users near the user by P2P (peer-to-peer). good too. It is possible to experience the operation results in the virtual space with lower latency than via the cloud. In this system, it is assumed that the AR display device 10 acquires update differences from the map data server 40 at regular intervals (for example, every second). Priority may be given to the operation result.

In the example described above, it is assumed that a new virtual object is laid out from scratch on site, but the present embodiment is not limited to this. etc., and fine adjustment of the position and orientation may be performed on site using the operation terminal 20 and the AR display device 10 .

<3-5. Information Update of Virtual Object Operated on Operation Terminal 20>
FIG. 9 is a sequence diagram showing the flow of information update processing for a virtual object operated on the operation terminal 20. As shown in FIG.

As shown in FIG. 9 , first, the coordinate information acquisition unit 221 of the operation terminal 20 acquires the geographical coordinate information and the like of the operation terminal 20 and the virtual space position information and the like based on the peripheral captured image acquired by the camera 230 . Acquire (step S403). Specifically, it is obtained from the position estimation server 30 based on the feature points of the captured image. Acquisition of geographical coordinate information and the like from the position estimation server 30 may be performed at a frequency of about once per second. In addition, the coordinate information acquisition unit 221 may recognize the self-position with higher accuracy by using the motion data of the operation terminal 20 with respect to the relative positional change in a short time (for example, using VIO). Note that the recognition of the self-position of the operation terminal 20 regarding changes in the self-position may be performed at a frequency of about 120 times per second.

Next, the operating terminal 20 receives the user's operation input for the virtual object (step S406), and transmits the position and orientation information of the operating terminal 20 and the operation information to the AR display device 10 (step S409). The operation information may include touch operations (information associated with various tap operations and swipe operations such as tap states, tap positions, swipe states, and swipe positions). Also, the position and orientation information of the operation terminal 20 can be recognized with higher accuracy based on motion data and captured images, such as by using VIO.

Next, the AR display device 10 determines processing for the virtual object according to the position/orientation information and the operation information of the operation terminal 20 (step S412). For example, as shown in FIG. 3, it determines the result of collision determination between a line extending straight from the tip of the operation terminal 20 and the virtual object, and determines processing according to the touch operation.

Next, the AR display device 10 transmits the determination of the above process as update information of the virtual object to other users' terminals (other AR display devices) located nearby by P2P (step S415). The virtual object update information includes, for example, the ID of the virtual object and information on the determined process. Further, the determined processing may include changing the position information and orientation information of the virtual space. In this case, the information about the determined processing may include post-change virtual space position information and orientation information.

Also, the coordinate information conversion unit 122 of the AR display device 10 converts the virtual space coordinate information of the changed (updated) virtual object into geographic coordinate information (step S418). The coordinate information conversion unit 122 may apply the above-described formula (Formula 3) to convert the virtual space coordinate information and orientation information of the changed (updated) virtual object into geographic coordinate information and orientation information.

Subsequently, the AR display device 10 transmits update information of the changed (updated) virtual object (including the ID of the virtual object, geographical coordinate information, determined processing, etc.) to the map data server 40 (step S412). ).

Next, the map data server 40 updates the data by saving the virtual object update information received from the AR display device 10 in the AR database 403 (step S424). Since the update information of the virtual object includes the geographic coordinate information of the changed (updated) virtual object, etc., it can be reflected in the map data server 40 that manages the geographic information based on the geographic coordinate information. becomes. Updates to the AR database 403 can be sent to other users within the area. In this way, the edited content (update, change) of the virtual object is shared in real time.

The AR display device 10 then reflects the determined processing, that is, updates the display of the virtual object (step S427). Note that the update information of the AR database 403 can also be periodically notified to the operation terminal 20 . For example, even on the map displayed on the operation terminal 20, updates such as changes in the arrangement of virtual objects are reflected in real time.

Above, each operation process according to the present embodiment has been specifically described. Note that the operation processing shown in each sequence diagram is an example, and the present disclosure is not limited to this. For example, this disclosure is not limited to the order of steps shown in each sequence diagram. At least some steps may be processed in parallel, may be processed in reverse order, or may be skipped.

For example, the process shown in step S415 shown in FIG. 9, the process shown in steps S418 to S424, and the process shown in step S427 may be performed in parallel.

<<4. Example of operation of virtual object using operation terminal 20 >>
Next, an example of operating a virtual object using the operating terminal 20 according to this embodiment will be described with reference to FIGS. 10 to 18. FIG. Here, a case where a smartphone is used as a controller will be described as an example.

10 and 11 are diagrams illustrating an example of a virtual object selection operation using the operation terminal 20 according to this embodiment. As shown on the left side of FIG. 10 , an instruction image L, such as a ray of light traveling straight from the tip of the operation terminal 20 (smartphone), is superimposed and displayed in real space on the display unit 150 (transmissive display) of the AR display device 10 . FIG. 10 also shows virtual objects 51a to 51c superimposed and displayed on the real space on the display unit 150. As shown in FIG.

As shown on the left side of FIG. 10, the indication image L can be displayed on an extension line of the central axis C that is at the center of the operation terminal 20 in the short direction and that is orthogonal to the short direction (perpendicular to the tip side of the operation terminal 20). Since the instruction image L is display-controlled by the AR display device 10 so as to follow the center axis C, as shown in the right side of FIG. By doing so, an arbitrary virtual object 51 can be selected.

Confirmation of the selection operation is determined by performing a tap operation on the operation terminal 20, as shown in FIG. 11, for example. The AR display device 10 selects the virtual object 51b that most closely collides with the instruction image L when the tap operation is performed. Feedback of the selection state may be performed, for example, by changing the display mode of the virtual object 51b or lighting the surroundings.

In the selected state, the position/rotation edit mode and the rotation/scale edit mode may be switched. For example, you can tap and hold to enter position/rotation editing mode, and a single tap to enter rotation/scale editing mode.

12 to 15 are diagrams explaining an example of an operation method using the operation terminal 20 in the position/rotation edit mode according to this embodiment. For example, as shown in FIGS. 12 and 13 , the position of the virtual object 51b in the selected state can be changed by changing the orientation of the operation terminal 20 up, down, left, right, or moving it horizontally while the user keeps tapping. changes. By maintaining the length of the instruction image L (that is, the distance between the operation terminal 20 and the virtual object 51b in the selected state) constant, a change in the position and orientation of the operation terminal 20 causes the virtual object 51b in the selected state to move. Can change position.

Also, as shown in FIG. 14, by swiping the user's tapping finger in the vertical or horizontal direction, the virtual object 51b in the selected state can be rotated (pitch rotation, yaw rotation). Further, as shown in FIG. 15 , by rotating the operating terminal 20 in the vertical direction (while tapping is continued), the virtual object 51b in the selected state can be rotated in the Roll direction.

16 and 17 are diagrams illustrating an example of an operation method using the operation terminal 20 in the rotation/scale edit mode according to this embodiment. For example, when the rotation/scale editing mode is set by single-tapping, the user can enlarge or reduce the virtual object 51b in the selected state by pinch-in/pinch-out operations, as shown in FIG. In addition, the user can rotate the virtual object 51b in the selected state by Pitch rotation and Yaw rotation by vertical and horizontal swipe operations (the correspondence between the swipe operation direction and the rotation direction is the same as in the example shown in FIG. 14). Alternatively, as shown in FIG. 17, the virtual object 51b in the selected state may be rolled by performing a rotation operation with two fingers.

FIG. 18 is a diagram explaining selection determination of other operation options for the selected virtual object. As shown in FIG. 18, in the virtual object selection state, options 52a to 52c ("delete", "undo", "copy", etc.) for other operations on the virtual object may be displayed. In this case, the user can point to an arbitrary option with the instruction image L that goes straight from the tip of the operation terminal 20, and can make a decision by a tap operation. Alternatively, as shown in FIG. 18, the focus 53 of options can be switched (up, down, left, and right) by swiping on the operation terminal 20, and can be determined by tapping. Alternatively, these options may be displayed as buttons on the display screen of the location estimation server 30. FIG.

<<5. Supplement >>
Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present technology is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or modifications within the scope of the technical idea described in the claims. are naturally within the technical scope of the present disclosure.

In the above embodiment, AR (Augmented Reality) is assumed as an example, but the present disclosure is not limited to this, and can be applied to VR (Virtual Reality) and MR (Mixed Reality).

Also, the AR display device 10 may be realized by a transmissive or non-transmissive HMD (Head Mounted Display) worn on the head, or may be a mobile terminal held by the user, such as a smartphone or a tablet terminal. It may be various wearable devices worn by the user.

Also, create one or more computer programs for causing the hardware such as the CPU, ROM, and RAM incorporated in the AR display device 10 or the operation terminal 20 described above to exhibit the functions of the AR display device 10 or the operation terminal 20. It is possible. Also provided is a computer-readable storage medium storing the one or more computer programs.

Also, the effects described in this specification are merely descriptive or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

Note that the present technology can also take the following configuration.
(1)
A control unit for controlling display of a virtual object associated with the real space,
The control unit
The virtual space position information of the virtual object that is the target of the user's operation, the geographical coordinate information of the information processing device obtained based on the captured image of the real space, and the virtual space corresponding to the geographical coordinate information An information processing device that converts into geographic coordinate information based on location information.
(2)
The information processing according to (1) above, wherein the control unit performs control to transmit information about the virtual object including at least the transformed geographic coordinate information to an external device that manages geographic information using a geographic coordinate system. Device.
(3)
The information processing apparatus according to (2), wherein the geographic information managed by the external device includes map image data and virtual object information associated with geographic coordinate information.
(4)
The information processing apparatus according to (3), wherein the geographic information managed by the external device further includes POI (point of interest) information.
(5)
The control unit converts virtual space position information of the virtual object whose position has changed as a result of the user's operation into corresponding geographic coordinate information, and stores at least the converted geographic coordinate information as update information of the virtual object. The information processing apparatus according to any one of (2) to (4) above, which controls transmission of information including to the external device.
(6)
The control unit
Acquiring information of one or more virtual objects in a predetermined area including a location indicated by the geographical coordinate information of the information processing device from the external device;
converting the geographical coordinate information of the virtual object included in the acquired information of the virtual object into corresponding virtual space position information based on the geographical coordinate information and the virtual space position information of the information processing device;
The information processing apparatus according to any one of (2) to (5), wherein display control of the obtained virtual object is performed based on the converted virtual space position information.
(7)
The information processing device according to any one of (1) to (6), wherein the captured image is obtained from an imaging unit of the information processing device.
(8)
The information processing device according to any one of (1) to (7) above, wherein the information processing device is a display device worn on a user's head.
(9)
The information processing apparatus according to (8), wherein the captured image is a field-of-view image of the user obtained from an imaging unit of the display device.
(10)
The control unit according to any one of (1) to (9) above, wherein the control unit controls display of the virtual object that is the target of user operation in accordance with information indicating the user's operation obtained from the operation terminal. The information processing device described.
(11)
The information according to any one of (1) to (9), wherein the control unit controls display of the virtual object to be operated by the user according to the position and orientation of the operation terminal and touch operation information. processing equipment.
(12)
The information processing apparatus according to (10) or (11), wherein the operation terminal is a general-purpose communication terminal that is held and operated by a user and provided with a motion sensor and a touch panel display.
(13)
The control unit according to any one of (2) to (12) above, wherein the control unit displays an arrangement of one or more virtual objects on a map image based on geographic information acquired from the external device. The information processing device described.
(14)
The information processing apparatus according to (13), wherein the control unit further displays POI (point of interest) information included in the geographic information on the map image.
(15)
The control unit acquires, every second, information on the geographical coordinate information of the information processing device based on the captured image and information on virtual space position information corresponding to the geographical coordinate information from an external device that performs position estimation, the (1 ) to (14).
(16)
The information processing device according to any one of (1) to (15), wherein the control unit periodically acquires update information from an external device that manages the geographic coordinate information.
(17)
The information processing apparatus according to any one of (1) to (16), wherein the geographical coordinate information is latitude, longitude, and altitude information.
(18)
In the transformation, posture information of the information processing device in a geographic coordinate system is used in addition to the geographical coordinate information. , the information processing apparatus according to any one of (1) to (17) above, wherein posture information of the virtual object in the virtual space is also converted.
(19)
the processor
displaying a virtual object associated with real space;
The virtual space position information of the virtual object that is the target of the user's operation, the geographical coordinate information of the information processing device obtained based on the captured image of the real space, and the virtual space corresponding to the geographical coordinate information converting to geographic coordinate information based on the location information;
A method of processing information, comprising:
(20)
the computer,
Function as a control unit that controls the display of virtual objects associated with the real space,
The control unit
The virtual space position information of the virtual object that is the target of the user's operation, the geographical coordinate information of the information processing device obtained based on the captured image of the real space, and the virtual space corresponding to the geographical coordinate information A program that converts geographic coordinates based on location information.

10 AR display device 121 coordinate information acquisition unit 122 coordinate information conversion unit 123 virtual space processing unit 20 operation terminal 221 coordinate information acquisition unit 222 operation information transmission control unit 223 display control unit 30 position estimation server 301 point cloud database 40 map data server 401 Map image database 402 POI database 403 AR database

Claims

A control unit for controlling display of a virtual object associated with the real space,
The control unit
The virtual space position information of the virtual object that is the target of the user's operation, the geographical coordinate information of the information processing device obtained based on the captured image of the real space, and the virtual space corresponding to the geographical coordinate information Transform into geographic coordinate information based on location information and
Information processing equipment.
The information processing apparatus according to claim 1, wherein said control unit performs control to transmit information about said virtual object including at least said transformed geographical coordinate information to an external device managing geographical information using a geographical coordinate system. .
The information processing apparatus according to claim 2, wherein the geographic information managed by the external device includes map image data and virtual object information associated with geographic coordinate information.
The information processing apparatus according to claim 3, wherein the geographic information managed by the external device further includes POI (point of interest) information.
The control unit converts virtual space position information of the virtual object whose position has changed as a result of the user's operation into corresponding geographic coordinate information, and stores at least the converted geographic coordinate information as update information of the virtual object. 3. The information processing apparatus according to claim 2, which controls transmission of information containing to said external device.
The control unit
Acquiring information of one or more virtual objects in a predetermined area including a location indicated by the geographical coordinate information of the information processing device from the external device;
converting the geographical coordinate information of the virtual object included in the acquired information of the virtual object into corresponding virtual space position information based on the geographical coordinate information and the virtual space position information of the information processing device;
3. The information processing apparatus according to claim 2, wherein display control of said acquired virtual object is performed based on said converted virtual space position information.
The information processing apparatus according to claim 1, wherein the captured image is obtained from an imaging unit of the information processing apparatus.
The information processing device according to claim 1, wherein the information processing device is a display device worn on a user's head.
The information processing apparatus according to claim 8, wherein the captured image is a field-of-view image of the user obtained from an imaging unit of the display device.
The information processing apparatus according to claim 1, wherein the control unit controls display of the virtual object that is the target of the user's operation according to information indicating the user's operation acquired from the operation terminal.
The information processing apparatus according to claim 1, wherein the control unit controls display of the virtual object that is the target of user operation according to the position and orientation of the operation terminal and touch operation information.
The information processing apparatus according to claim 10, wherein the operation terminal is a general-purpose communication terminal that is held and operated by a user and provided with a motion sensor and a touch panel display.
3. The information processing apparatus according to claim 2, wherein the control unit displays a layout of one or more virtual objects on a map image based on geographic information acquired from the external device.
The information processing apparatus according to claim 13, wherein said control unit further displays POI (point of interest) information included in said geographic information on said map image.
2. The control unit acquires, every second from an external device that performs position estimation, information on the geographical coordinate information of the information processing device based on the captured image and information on virtual space position information corresponding to the geographical coordinate information. The information processing device according to .
The information processing apparatus according to claim 1, wherein the control unit periodically acquires update information from an external device that manages the geographic coordinate information.
The information processing apparatus according to claim 1, wherein the geographical coordinate information is latitude, longitude, and altitude information.
In the transformation, posture information of the information processing device in a geographic coordinate system is used in addition to the geographical coordinate information. 2. The information processing apparatus according to claim 1, wherein posture information of said virtual object in said virtual space is also transformed.
the processor
displaying a virtual object associated with real space;
The virtual space position information of the virtual object that is the target of the user's operation, the geographical coordinate information of the information processing device obtained based on the captured image of the real space, and the virtual space corresponding to the geographical coordinate information converting to geographic coordinate information based on the location information;
A method of processing information, comprising:
the computer,
Function as a control unit that controls the display of virtual objects associated with the real space,
The control unit
The virtual space position information of the virtual object that is the target of the user's operation, the geographical coordinate information of the information processing device obtained based on the captured image of the real space, and the virtual space corresponding to the geographical coordinate information A program that converts geographic coordinates based on location information.