JP2022140891A - Terminal device, information processing system, information processing method, and computer program - Google Patents

Abstract

To estimate a position and attitude of a terminal device with high accuracy.SOLUTION: A terminal device that is held by a user and moves with the user includes an imaging unit, a marker information acquisition unit, an image association information acquisition unit, a relative relation determination unit, and a terminal position and attitude estimation unit. The marker information acquisition unit obtains marker position and attitude information indicating a position and attitude of a marker portion of a moving body on a three-dimensional point group map. The image association information acquisition unit obtains image association information for associating an image in which the marker portion is captured with the relative position and relative attitude with respect to the marker portion. The relative relation determination unit determines relative relation information indicating the relative position and relative attitude of the terminal device with respect to the marker portion, on the basis of, both the image of the marker portion which is captured by the imaging unit and the image association information. The terminal position and attitude estimation unit estimates the position and attitude of the terminal device on the three-dimensional point group map, on the basis of, the marker position and attitude information and the relative relation information.SELECTED DRAWING: Figure 6

Description

The technology disclosed in this specification relates to a terminal device, an information processing system, an information processing method, and a computer program.

In recent years, augmented reality (AR) technology, in which a computer augments a real environment perceived by humans, is becoming widespread. For example, in a terminal device (smartphone, etc.) that is held by a user and moves with the user, an AR-type information presentation in which a virtual image (computer graphics) is superimposed on the image of the real environment generated by the imaging unit and displayed on the display unit. are used, for example, in the form of walking navigation, tourist information, maintenance and inspection guides, location games, and the like.

In the AR-type information presentation, based on the position and orientation of the terminal device, information about where and from what distance and angle the imaging unit of the terminal device is viewed is specified, and based on the information, a virtual image to be displayed on the display unit is displayed. The type, size, position, etc. are determined. Therefore, in order to display a virtual image that matches the real environment in AR-type information presentation, it is necessary to estimate the position and orientation of the terminal device (more specifically, the imaging unit of the terminal device; the same shall apply hereinafter) with high accuracy. is required.

Conventionally, there is known a method of estimating the position and attitude of a terminal device by receiving signals from a GNSS (Global Navigation Satellite System) such as GPS, GLONASS, Galileo, and QZSS. . Further, by setting a marker whose position and orientation in the real environment are known and capturing an image of the marker with an imaging unit of the terminal device, information on the relative position and orientation of the terminal device with respect to the marker is specified, and based on the information, A method of estimating the position and orientation of a terminal device in a real environment is known (see Patent Document 1, for example).

Japanese Patent Application Laid-Open No. 2021-4894

In the method of estimating the position and attitude of the terminal device using GNSS, the estimation error is relatively large (for example, the error is about several meters), and the signal cannot be received well indoors. There is a problem that it is difficult to use. Moreover, the method of estimating the position and orientation of a terminal device using markers has the problem that it is necessary to install a large number of markers whose positions and orientations are known in advance.

In addition, the terminal device may be equipped with a sensor (e.g., LiDAR) and a 3D point cloud map database used for self-position and orientation estimation by an autonomous mobile body, and use them to estimate the position and orientation of the terminal device. It is possible. However, when such a sensor or database is installed in a terminal device that is held by a user and moves with the user, there is a problem that the terminal device becomes large and heavy, and the storage capacity of the terminal device becomes insufficient.

In this way, conventionally, while suppressing the increase in size and weight of the terminal device and the lack of storage capacity, and without the need to install a large number of markers in advance, the terminal device can be operated with high accuracy indoors and outdoors. There is a problem that it is difficult to estimate the position and orientation.

Such a problem is not limited to estimating the position and orientation of a terminal device for AR-type information presentation, but is common to estimating the position and orientation of a terminal device for other uses.

This specification discloses a technology capable of solving the above-described problems.

The technology disclosed in this specification can be implemented, for example, in the following forms.

(1) A terminal device disclosed in this specification is a device that is held by a user and moves with the user. The terminal device includes an imaging unit, a marker information acquisition unit, an image correspondence information acquisition unit, a relative relationship identification unit, and a terminal position/orientation estimation unit. The marker information acquisition unit is a moving object that moves while specifying its own position and orientation on a 3D point cloud map composed of a point cloud to which information indicating a 3D absolute position is assigned, and functions as a marker. Marker position and orientation information indicating the position and orientation of the marker section on the three-dimensional point cloud map is acquired for the moving object having the marker section. The image correspondence information acquisition unit acquires image correspondence information that associates an image obtained by capturing the marker portion with a relative position and a relative orientation with respect to the marker portion. A relative relationship specifying unit specifies relative relationship information indicating a relative position and a relative attitude of the terminal device with respect to the marker unit, based on the image of the marker unit captured by the imaging unit and the image correspondence information. A terminal position/orientation estimation unit estimates the position and orientation of the terminal device on the three-dimensional point cloud map based on the marker position/orientation information and the relative relationship information.

As described above, the present terminal device has marker position and orientation information indicating the position and orientation of the marker section of the moving object on the three-dimensional point cloud map, and relative relationship information indicating the relative position and relative orientation of the terminal device with respect to the marker section. , the position and orientation of the terminal device on the 3D point cloud map can be estimated. Therefore, according to the present terminal device, a large number of markers whose positions and orientations are known are installed in advance seamlessly regardless of indoors or outdoors as long as the marker unit of the moving body can be imaged by the imaging unit. Furthermore, the position and orientation of the terminal device can be estimated with high accuracy without installing a sensor (for example, LiDAR) or a 3D point cloud map database other than the imaging unit in the terminal device.

(2) The terminal device further includes a tracking unit that tracks changes in the position and orientation of the terminal device from the point in time when the position and orientation of the terminal device are estimated; and an estimation result updating unit that updates the posture estimation result. According to this terminal device, after once estimating the position and orientation (initial position and initial orientation) of the terminal device on the three-dimensional point cloud map based on the marker position and orientation information and the relative relationship information, the mobile body marker The position and orientation estimation result of the terminal device can be updated by the terminal device alone, regardless of whether or not the unit can be imaged by the imaging unit. At this time, since the estimation accuracy of the initial position and initial orientation is high, the accuracy of estimating the position and orientation of the terminal device is maintained at a certain level or higher even with subsequent updates. Therefore, according to this terminal device, it is possible to continuously and highly accurately estimate the position and attitude of a terminal device that moves independently of a moving body.

(3) In the above terminal device, the tracking unit may be configured to track changes in the position and posture of the terminal device using the image captured by the imaging unit. According to this terminal device, it is possible to continuously estimate the position and orientation of the terminal device with high accuracy without requiring a sensor other than the imaging unit.

(4) In the above terminal device, the display unit further superimposes a virtual image that changes according to the estimation result of the position and orientation of the terminal device on the real image captured by the imaging unit, and the display unit It may be configured to include an augmented reality processing unit for displaying. According to this terminal device, a virtual image can be displayed based on the position and orientation of the terminal device estimated with high accuracy, and highly accurate AR information presentation can be realized.

(5) An information processing system disclosed in this specification includes the terminal device and the moving body. The terminal device further includes an estimation result transmission unit that transmits the estimation result of the position and orientation of the terminal device to the moving body directly or via another device, and the moving body receives the terminal device A configuration including a specific processing execution unit that executes specific processing using the position and orientation estimation results may be provided. According to this information processing system, the position and orientation of the terminal device on the three-dimensional point cloud map estimated with high accuracy can be used to execute specific processing, thereby improving the accuracy of the specific processing. can be made

(6) In the above information processing system, the specific processing may include hazard notification processing for notifying the terminal device of the presence of a hazard in proximity to the terminal device. According to this information processing system, the position and orientation of the terminal device on the three-dimensional point cloud map estimated with high accuracy can be used to notify the terminal device of the presence of a hazard from the mobile object, thereby achieving high accuracy. hazard notification can be realized.

(7) In the above information processing system, the mobile object may be an autonomous mobile object that autonomously moves using surrounding detection information from a sensor and the three-dimensional point cloud map. According to this information processing system, it is possible to estimate the position and orientation of the terminal device with high accuracy over a wide range by using the moving marker unit by moving the marker unit without human driving operation. can do.

(8) The information processing system further includes a server device, wherein the moving body transmits the marker position/orientation information to the server device, and the marker information acquisition unit of the terminal device transmits the marker position and orientation information to the server device. The marker position/orientation information may be obtained by According to this information processing system, even in a terminal device that cannot directly communicate with a mobile object, it is possible to estimate the position and orientation of the terminal device with high accuracy using the marker portion of the mobile object.

The technology disclosed in this specification can be implemented in various forms. For example, terminal devices, information processing systems, information processing methods, computer programs for realizing these methods, It can be realized in the form of a recorded non-transitory recording medium or the like.

1 is an explanatory diagram schematically showing the configuration of an information processing system 10 according to a first embodiment; FIG. 1 is a block diagram schematically showing the configuration of a terminal device 100 according to the first embodiment; FIG. Block diagram schematically showing the configuration of the autonomous mobile body 200 in the first embodiment 1 is a block diagram schematically showing the configuration of a server device 300 according to the first embodiment; FIG. Explanatory diagram showing the flow of AR-type information presentation processing executed in the information processing system 10 of the first embodiment. FIG. 2 is an explanatory diagram conceptually showing AR-type information presentation processing executed in the information processing system 10 of the first embodiment; FIG. 2 is an explanatory diagram conceptually showing AR-type information presentation processing executed in the information processing system 10 of the first embodiment; Explanatory diagram showing how to convert from the user coordinate system to the world coordinate system Explanatory diagram schematically showing the configuration of an information processing system 10a according to the second embodiment. A block diagram schematically showing the configuration of a terminal device 100a according to the second embodiment. Block diagram schematically showing the configuration of an autonomous mobile body 200a in the second embodiment A block diagram schematically showing the configuration of a server device 300a according to the second embodiment. Explanatory diagram showing the flow of hazard notification processing executed in the information processing system 10a of the second embodiment. Explanatory diagram conceptually showing the hazard notification process executed in the information processing system 10a of the second embodiment.

A. First embodiment:
A-1. Configuration of information processing system 10:
FIG. 1 is an explanatory diagram schematically showing the configuration of an information processing system 10 according to the first embodiment. The information processing system 10 of the present embodiment highly accurately estimates the position and orientation (orientation) of the terminal device 100 (more specifically, the imaging unit 154, which will be described later; the same applies hereinafter) on the three-dimensional point cloud map. This is a system for executing AR-type information presentation in a terminal device 100 based on the position and orientation of the terminal device 100. In the present embodiment, an example of AR type information presentation in which guidance is displayed indoors and outdoors in a shopping mall will be described.

The information processing system 10 includes a terminal device 100 used by each of a plurality of users U, a plurality of autonomous mobile bodies 200 and a server device 300 . Each device that configures the information processing system 10 is communicably connected to each other via a communication network NET.

The terminal device 100 is a device that is held by the user U and moves with the user U, and is, for example, a handheld device such as a smart phone or a tablet terminal, or a head mounted display (HMD). An example using a smart phone as the terminal device 100 will be described below.

FIG. 2 is a block diagram schematically showing the configuration of the terminal device 100 according to the first embodiment. The terminal device 100 includes a control unit 110 , a storage unit 120 , a display unit 151 , an operation input unit 152 , an audio output unit 153 , an imaging unit 154 and a communication unit 155 . These units are communicably connected to each other via a bus 190 .

The display unit 151 of the terminal device 100 is configured by, for example, a liquid crystal display or an organic EL display, and displays various images and information. The operation input unit 152 is composed of, for example, buttons, a microphone, and the like, and receives user U's operations and instructions. Note that the display unit 151 may function as the operation input unit 152 by including a touch panel. The audio output unit 153 is configured by, for example, a speaker, and outputs (plays back) audio. The imaging unit 154 is configured by, for example, a camera, and images a subject to generate a still image or a moving image representing the subject. In this embodiment, the imaging unit 154 is a camera (rear camera) having a lens on the surface (rear surface) opposite to the display surface of the display unit 151 in the terminal device 100 . The communication unit 155 is a communication interface that communicates with other devices using a predetermined communication method via the communication network NET.

The storage unit 120 of the terminal device 100 is configured by, for example, ROM, RAM, etc., stores various programs and data, and is used as a work area when executing various programs and a temporary storage area for data. do. For example, the storage unit 120 stores an augmented reality processing program CP11 for performing AR-type information presentation processing, which will be described later. The augmented reality processing program CP11 is, for example, provided in a state stored in a computer-readable recording medium (not shown) such as a CD-ROM, DVD-ROM, or USB memory, or transmitted from an external device via the communication network NET. It is stored in the storage unit 120 by being downloaded and installed in the terminal device 100 .

Further, the storage unit 120 stores marker image information MI, marker position/orientation information LI, and image correspondence information II when installing the augmented reality processing program CP11 or when performing an AR-type information presentation process, which will be described later. , relative relationship information CI are stored. The contents of these pieces of information will be described in conjunction with the description of the AR-type information presentation process that will be described later.

The control unit 110 of the terminal device 100 is configured by, for example, a CPU, etc., and controls the operation of the terminal device 100 by executing a computer program read from the storage unit 120 . For example, the control unit 110 reads the augmented reality processing program CP11 from the storage unit 120 and executes it, thereby functioning as the augmented reality processing unit 111 that performs AR information presentation processing. Augmented reality processing unit 111 includes terminal position/orientation estimation unit 112, and terminal position/orientation estimation unit 112 includes marker information acquisition unit 113, image correspondence information acquisition unit 114, relative relationship identification unit 115, and tracking unit 116. , and an estimation result updating unit 117 . The functions of these units will be described together with the description of the AR information presentation process described later.

The autonomous mobile body 200 is a mobile body that autonomously travels without the need for human operation. The autonomous mobile body 200 is, for example, an autonomous vehicle or an autonomous mobile robot. In this embodiment, as the autonomous mobile body 200, an example using an autonomous mobile robot that carries packages in a shopping mall will be described. The autonomous mobile body 200 is an example of a mobile body in the scope of claims.

FIG. 3 is a block diagram schematically showing the configuration of the autonomous mobile body 200 in the first embodiment. The autonomous mobile body 200 includes a control section 210 , a storage section 220 , a sensor 230 , a drive section 250 , a communication section 255 and a marker section 260 .

The sensor 230 of the autonomous mobile body 200 is composed of, for example, LiDAR, RADAR, a visible light camera, a far infrared camera, an ultrasonic sensor, etc., and detects the surrounding environment of the autonomous mobile body 200 (other vehicles, pedestrians, bicycles, road obstacles objects, roadside trees, road steps, etc.) and the state of the autonomous mobile body 200 itself (position, posture, shape, movement speed, etc.). The driving unit 250 is configured by, for example, a motor, an engine, a steering wheel, a brake, etc., and performs running operations (acceleration, steering, braking, etc.) of the autonomous mobile body 200 . The communication unit 255 is a communication interface that communicates with other devices using a predetermined communication method via the communication network NET. Also, the marker part 260 is a part that functions as a marker. In this embodiment, the marker part 260 is a predetermined planar image attached to the side surface of the autonomous mobile body 200 .

The storage unit 220 of the autonomous mobile body 200 is configured by, for example, a ROM, a RAM, a hard disk drive (HDD), etc., and stores various programs and data, and temporarily stores work areas and data when executing various programs. used as a storage area. For example, the storage unit 220 stores an autonomous movement control program CP21, a position and orientation information providing program CP22, and a 3D point cloud map database MD. The 3D point cloud map database MD contains high-precision point clouds that are assigned information indicating the 3D absolute positions (latitude, longitude, altitude) of roads, passages, roadside trees, guardrails, buildings, etc. A 3D point cloud map is included. The 3D point cloud map contained in the 3D point cloud map database MD covers the entire movable range of the autonomous mobile body 200 . In the following description, the coordinate system of the 3D point cloud map is also referred to as "world coordinate system", and the position and orientation on the 3D point cloud map are also referred to as "absolute position" and "absolute orientation".

The control unit 210 of the autonomous mobile body 200 is configured by, for example, a CPU, etc., and controls the operation of the autonomous mobile body 200 by executing a computer program read from the storage unit 220 . For example, the control unit 210 functions as an autonomous movement control unit 212 that controls autonomous movement of the autonomous mobile body 200 by reading the autonomous movement control program CP21 from the storage unit 220 and executing it. The autonomous movement control unit 212 sets a movement route based on a command issued by the administrator. Further, the autonomous movement control unit 212 performs matching between various signals output from the sensor 230 and the 3D point cloud map included in the 3D point cloud map database MD, thereby enabling autonomous movement on the 3D point cloud map. The position (absolute position) and orientation (absolute orientation) of the body 200 (more specifically, the sensor 230 of the autonomous mobile body 200, hereinafter the same) are estimated. Furthermore, the autonomous movement control unit 212 detects and recognizes the state of the surrounding environment based on various signals output from the sensor 230 . The autonomous movement control unit 212 controls the traveling operation of the drive unit 250 based on these pieces of information, thereby causing the autonomous mobile body 200 to travel along the set movement route while avoiding collisions with surrounding objects.

Also, the control unit 210 of the autonomous mobile body 200 functions as a position/attitude information providing unit 214 by reading out the position/attitude information providing program CP22 from the storage unit 220 and executing it. The function of the position/orientation information providing unit 214 will be described together with the description of the AR-type information presentation process described later.

The server device 300 is a device for acquiring information from the terminal device 100 and the autonomous mobile body 200 and for providing information to the terminal device 100 and the autonomous mobile body 200 . FIG. 4 is a block diagram schematically showing the configuration of the server device 300 according to the first embodiment. The server device 300 includes a control unit 310 , a storage unit 320 , a display unit 351 , an operation input unit 352 and a communication unit 355 . These units are communicatively connected to each other via a bus 390 .

A display unit 351 of the server device 300 is configured by, for example, a liquid crystal display or an organic EL display, and displays various images and information. The operation input unit 352 is composed of, for example, a keyboard, mouse, buttons, microphone, etc., and receives operations and instructions from the administrator. The communication unit 355 is a communication interface that communicates with other devices using a predetermined communication method via the communication network NET.

The storage unit 320 of the server device 300 is composed of, for example, a ROM, a RAM, a hard disk drive (HDD), etc., and stores various programs and data, and temporarily stores work areas and data when executing various programs. It is also used as a storage area. For example, the storage unit 320 stores an augmented reality server program CP31 for performing AR-type information presentation processing, which will be described later. The augmented reality server program CP31 is provided, for example, in a state stored in a computer-readable recording medium (not shown) such as a CD-ROM, DVD-ROM, or USB memory, or received from an external device via the communication network NET. It is stored in the storage unit 320 by being downloaded and installed in the server device 300 .

In addition, the storage unit 320 of the server device 300 stores the virtual image database ID and the marker position/orientation information LI when the augmented reality server program CP31 is installed or when the AR type information presentation process described later is performed. be done. In the virtual image database ID, a plurality of virtual images to be provided to the terminal device 100 in AR-type information presentation processing, which will be described later, are set in association with the position and orientation of the terminal device 100 . In this embodiment, the virtual image is an image for displaying guidance in the shopping mall (see FIGS. 6 and 7, which will be described later). Further, the contents of the marker position/orientation information LI will be described together with the description of the AR-type information presentation process that will be described later.

The control unit 310 of the server device 300 is configured by, for example, a CPU, etc., and controls the operation of the server device 300 by executing a computer program read from the storage unit 320 . For example, the control unit 310 reads the augmented reality server program CP31 from the storage unit 320 and executes it, thereby functioning as an augmented reality server processing unit 312 that performs AR information presentation processing. The augmented reality server processing unit 312 includes a position and orientation server processing unit 314 . The functions of these units will be described together with the description of the AR information presentation process described later.

A-2. AR-type information presentation processing:
Next, AR type information presentation processing executed in the information processing system 10 of the first embodiment will be described. The AR-type information presentation process of the present embodiment is a process of superimposing a virtual image on the image of the real environment captured by the imaging unit 154 and displaying it on the display unit 151 in the terminal device 100 . In the present embodiment, an example of AR type information presentation in which guidance is displayed indoors and outdoors in a shopping mall will be described.

FIG. 5 is an explanatory diagram showing the flow of AR-type information presentation processing executed in the information processing system 10 of the first embodiment. 6 and 7 are explanatory diagrams conceptually showing the AR-type information presentation processing executed in the information processing system 10 of the first embodiment. FIG. 6 shows a scene in which AR information is presented outdoors in a shopping mall, and FIG. 7 shows a scene in which AR information is presented indoors in a shopping mall.

As shown in FIG. 5, when the process is started, the autonomous mobile body 200 starts autonomous movement outdoors and indoors of the shopping mall (S210). That is, the autonomous movement control unit 212 (FIG. 3) of the autonomous mobile body 200 controls the position (absolute position) and attitude (absolute attitude) of the autonomous mobile body 200 on the three-dimensional point cloud map included in the three-dimensional point cloud map database MD. ), the autonomous mobile body 200 is made to run autonomously. The autonomous movement control unit 212 estimates the position and orientation of the autonomous moving body 200 on the 3D point cloud map by matching various signals output from the sensor 230 with the 3D point cloud map. , and the estimation accuracy is very high, for example, with an error of about several centimeters.

When the autonomous mobile body 200 starts to move autonomously, the position and orientation information providing unit 214 (FIG. 3) of the autonomous mobile body 200 provides marker position and orientation information LI indicating the position and orientation of the marker unit 260 on the three-dimensional point cloud map. , to the server device 300 (S220). In the autonomous mobile body 200, information indicating the correspondence relationship between the position and orientation of (the sensor 230 of) the autonomous mobile body 200 grasped by the autonomous movement control unit 212 and the position and orientation of the marker unit 260 is stored in the storage unit 220 in advance. stored in The position and orientation information providing unit 214 refers to the correspondence relationship and converts the position and orientation of the autonomous mobile body 200 ascertained by the autonomous movement control unit 212 into the position and orientation of the marker unit 260 to obtain the marker position and orientation. Information LI is generated, and the marker position/orientation information LI is transmitted to the server device 300 . The marker position/orientation information LI includes a mobile ID for identifying the autonomous mobile body 200, a time stamp indicating the estimated time of the position and orientation, and information specifying the position and orientation (see FIG. 4). .

In addition, the position and orientation information providing unit 214 of the autonomous mobile body 200 determines whether or not the autonomous movement of the autonomous mobile body 200 has ended (S230), and it is determined that the autonomous movement of the autonomous mobile body 200 has not ended. If so (S230: NO), the process of transmitting the marker position/orientation information LI (S220) is executed again. That is, the position/posture information providing unit 214 repeatedly performs the process of transmitting the marker position/posture information LI (S220) periodically (for example, at about 1 Hz to 100 Hz) while the autonomous mobile body 200 is moving autonomously. In this embodiment, many autonomous mobile bodies 200 are performing such processing indoors and outdoors in a shopping mall. When it is determined that the autonomous movement of the autonomous mobile body 200 has ended (S230: YES), the autonomous mobile body 200 ends its operation.

Also, the position/orientation server processing unit 314 ( FIG. 4 ) of the server device 300 receives the marker position/orientation information LI transmitted from the autonomous mobile body 200 in S 220 and stores the received marker position/orientation information LI in the storage unit 320 . (S310). The position/orientation server processing unit 314 accumulates in the storage unit 320 the marker position/orientation information LI that is constantly transmitted from the plurality of autonomous mobile bodies 200 .

In addition, the user U of the terminal device 100 instructs the execution of an application program (augmented reality processing program CP11) for presenting AR information indoors and outdoors in a shopping mall, searches for a nearby autonomous mobile body 200, and takes an image. The terminal device 100 is set so that the section 154 faces the direction of the marker section 260 of the autonomous mobile body 200 (see FIGS. 6 and 7). Triggered by such an operation by the user U, the terminal position/orientation estimation unit 112 (FIG. 2) of the terminal device 100 detects the image of the marker unit 260 from the image generated by the imaging unit 154 (S110). As described above, the storage section 120 of the control section 110 stores the marker image information MI. The marker image information MI is information specifying an image used as the marker section 260 . The terminal position/orientation estimation unit 112 refers to the marker image information MI and detects the image of the marker unit 260 from the captured image by a known method such as pattern matching. In the example shown in FIG. 6, the marker unit 260 of the autonomous mobile body 200 moving near the entrance of the outdoor shopping mall is imaged by the imaging unit 154 of the terminal device 100 held by the user U and displayed on the display unit 151. . Also, in the example shown in FIG. 7, the marker unit 260 of the autonomous mobile body 200 moving inside the shopping mall is imaged by the imaging unit 154 of the terminal device 100 held by the user U and displayed on the display unit 151. .

Further, the image correspondence information acquisition unit 114 of the terminal device 100 acquires the image correspondence information II from the storage unit 120, and the relative relationship identification unit 115 of the terminal device 100 acquires the image of the marker unit 260 captured by the imaging unit 154. , and the image correspondence information II, relative relationship information CI indicating the relative position and relative orientation of (the imaging unit 154 of) the terminal device 100 with respect to the marker unit 260 is specified (S120).

More specifically, the image correspondence information II is information that associates an image captured by the marker unit 260 with a relative position and a relative orientation with respect to the marker unit 260 . The appearance of the marker section 260 in the image captured by the imaging section 154 changes depending on the relative position and orientation of the imaging section 154 of the terminal device 100 with respect to the marker section 260 . For example, if the distance between the imaging unit 154 and the marker unit 260 is relatively short, the marker unit 260 appears relatively large in the captured image. If it is far away, the marker portion 260 appears relatively small in the captured image. The appearance of the marker section 260 in the captured image differs between the state in which the imaging section 154 faces the marker section 260 and the state in which the imaging section 154 is positioned obliquely to the marker section 260 . The image correspondence information II is information that associates the relative position and orientation of the imaging unit 154 of the terminal device 100 with respect to the marker unit 260 and how the marker unit 260 appears in the captured image. Therefore, the relative relationship identifying section 115 can identify the relative position and orientation of (the imaging section 154 of) the terminal device 100 with respect to the marker section 260 based on the image of the marker section 260 and the image correspondence information II. Note that in the present embodiment, the relative position and orientation of the terminal device 100 with respect to the marker unit 260 are defined as the position and orientation of the marker unit 260 in the coordinate system used by the terminal device 100 (hereinafter referred to as the “user coordinate system”). expressed. The characterized relative relationship information CI is stored in the storage unit 120 .

Further, the marker information acquisition unit 113 (FIG. 1) of the terminal device 100 requests the server device 300 for the marker position and orientation information LI, and acquires the marker position and orientation information LI from the server device 300 (S130). In this embodiment, the information (mobile body ID) for identifying the autonomous mobile body 200 is included in the marker section 260 of the autonomous mobile body 200 . Therefore, the marker information acquisition unit 113 designates the moving body ID of the autonomous moving body 200 whose relative position and relative orientation have been identified in S120, and also designates the specific time of the relative position and relative orientation, and the server device 300 is requested for marker position and orientation information LI. The position/orientation server processing unit 314 (FIG. 4) of the server device 300 selects the marker position/orientation information LI corresponding to the specification of the mobile ID and the specification of the time included in the request from the terminal device 100, and sends the marker position/orientation information LI to the terminal device 100. (S320). The marker position/orientation information LI acquired by the terminal device 100 is stored in the storage unit 120 .

Next, the terminal position/orientation estimation unit 112 (FIG. 2) of the terminal device 100 performs The position (absolute position) and orientation (absolute orientation) of (the imaging unit 154 of) the terminal device 100 are estimated (S140). As described above, the marker position/orientation information LI is information indicating the position (absolute position) and orientation (absolute orientation) of the marker section 260 on the three-dimensional point cloud map. It is information indicating the relative position and relative attitude of the terminal device 100 . Therefore, terminal position/orientation estimation section 112 performs coordinate conversion (conversion from the user coordinate system to the world coordinate system) using both pieces of information, thereby estimating the position and orientation of terminal device 100 on the three-dimensional point cloud map. can do. The coordinate transformation used at this time will be described in detail later. Hereinafter, the time point at which the estimation in S140 is performed will be referred to as a "tracking start point", and the position and orientation of the terminal device 100 estimated in S140 will be referred to as an "initial position" and an "initial orientation".

After that, tracking section 116 ( FIG. 2 ) of terminal device 100 tracks changes in the position and orientation of terminal device 100 from the initial position and initial orientation at the tracking start point, and estimation result updating section 117 updates based on the changes. , update the estimation result of the position and orientation of the terminal device 100 (S150). Tracking of changes in the position and orientation of the terminal device 100 by the tracking unit 116 (hereinafter also referred to as “tracking”) is realized by a known technique. For example, the tracking unit 116 uses Visual-SLAM, which is one of SLAM (Simultaneous Localization and Mapping) techniques for simultaneously estimating the position of a mobile object and creating an environment map, to determine the position of the terminal device 100 and Track posture changes. That is, the tracking unit 116 creates a map of the surrounding environment (map of the user coordinate system) based on the image captured by the imaging unit 154, and based on the transition of the feature amount in the image captured by the imaging unit 154, The movement amount and attitude change amount of the terminal device 100 in the user coordinate system are estimated, and the position and attitude of the imaging unit 154 are tracked on the surrounding environment map. Thus, tracking by the tracking unit 116 can be continuously performed even if the marker unit 260 of the autonomous mobile body 200 does not exist within the imaging range of the imaging unit 154 . In addition, the estimation result update unit 117 performs coordinate transformation (conversion from the user coordinate system to the world coordinate system) with respect to the position and orientation of the imaging unit 154 on the surrounding environment map, so that update the estimation result of the position and orientation of the imaging unit 154 of the terminal device 100 in .

Transformation from the user coordinate system to the world coordinate system will now be described. FIG. 8 is an explanatory diagram showing a conversion method from the user coordinate system to the world coordinate system. As described above, the world coordinate system is the coordinate system of the 3D point cloud map, and the user coordinate system is the coordinate system used by the terminal device 100 . Further, hereinafter, the coordinate system whose origin is the position of the autonomous mobile body 200 is referred to as the "moving body coordinate system", and the coordinate system whose origin is the position of the imaging unit 154 of the terminal device 100 is referred to as the "imaging unit coordinate system". .

Column A in FIG. 8 shows the position of the origin O _V ^W of the mobile body coordinate system in the world coordinate system (origin: O ^w ) and the orientation of the mobile body coordinate system, and column B in FIG. The position of the origin ^OCU of the imaging unit coordinate system and the direction of the imaging unit coordinate system in the user coordinate system (origin: _O ^U ) and the position of the ^origin _OvU of the moving body coordinate system and the direction of the moving body coordinate system are 8, column C in FIG. 8 shows the position of the origin O _U ^W of the user coordinate system in the world coordinate system and the direction of the user coordinate system, the position of the origin O _C ^W of the imaging unit coordinate system and the imaging unit coordinates. The orientation of the system, the position of the origin O _V ^W of the mover coordinate system and the orientation of the mover coordinate system are indicated. The origin ^OU of the user coordinate system is the position of the imaging unit 154 at the tracking start point estimated in S140. Therefore, at the tracking start point, the imaging unit coordinate system matches the user coordinate system. After the tracking start point, as the position and orientation of the imaging unit 154 of the terminal device 100 change (indicated by thin dashed lines in FIG. 8), the origin position of the imaging unit coordinate system and the orientation of the imaging unit coordinate system change from those of the user coordinate system. It varies from the position of the origin and the orientation of the user coordinate system.

As shown in the following formula (1), the linear transformation matrix T _OU ^w of a point in the user coordinate system to the world coordinate system is a vector from the origin O ^w of the world coordinate system to the origin O _V ^W of the mobile body coordinate system It can be _obtained by subtracting from T _OV ^W a vector T _O ^V from the origin O _U ^W of the user coordinate system in the world coordinate system to the origin O _V ^W of the mobile body coordinate system.

Further, as shown in the following formula (2), the rotation transformation matrix R _OU ^w of the point in the user coordinate system to the world ^coordinate system is the origin O It can be obtained from the quaternion product of the amount of rotation of _VW and the inverse ^quaternion of the amount of rotation of the moving body coordinate system viewed from the user coordinate system.

By using these transformation matrices, the terminal position/orientation estimation unit 112 can estimate the position and orientation of the terminal device 100 on the 3D point cloud map at the tracking start point (S140). After the point, the estimation result updating unit 117 can update the estimation result of the position and orientation of the terminal device 100 on the 3D point cloud map (S150). As a result, the terminal device 100 can continuously grasp the position and orientation of the terminal device 100 on the three-dimensional point cloud map (that is, the absolute position and absolute orientation in the world coordinate system) after the tracking start point. .

As shown in FIG. 5, after the tracking start point, the augmented reality processing unit 111 (FIG. 2) of the terminal device 100 instructs the server device 300 to A request is made for the virtual image VI, and the virtual image VI is obtained from the server device 300 (S160). That is, the augmented reality processing unit 111 specifies the position and orientation of the terminal device 100 on the three-dimensional point cloud map, and requests the server device 300 for the virtual image VI. As described above, in the server device 300, a virtual image database ID (FIG. 4) is constructed in which a plurality of virtual images VI are set in association with the position and orientation of the terminal device 100. FIG. The augmented reality server processing unit 312 of the server device 300 selects a virtual image VI corresponding to the position and orientation of the terminal device 100 included in the request from the terminal device 100 from the virtual image database ID, and sends the virtual image VI to the terminal device 100. It transmits to the device 100 (S330).

The augmented reality processing unit 111 of the terminal device 100 superimposes the virtual image VI acquired in S160 on the real image RI captured by the imaging unit 154, and causes the display unit 151 to display the superimposed image (S170). For example, in the example shown in FIG. 6, a virtual image VI (“ Main Entrance (main entrance," "Parking," "Cafe," etc.) are displayed.In addition, in the example shown in Fig. 7, the display unit 151 of the terminal device 100 held by the user U is displayed. Then, a virtual image VI for guidance display ("Downstair", "Event Space", "Food Court") is superimposed on the real image RI representing the inside of the building of the shopping mall. )" etc.) are displayed. The user U can grasp the position of each facility in the shopping mall by referring to the virtual image VI displayed on the display unit 151 .

After the tracking start point, the augmented reality processing unit 111 (FIG. 2) of the terminal device 100 monitors whether or not the image of the marker unit 260 is detected from the image generated by the imaging unit 154 (S180). It is monitored whether or not there is an instruction to end the information presentation process (S190). When it is determined that the image of the marker unit 260 is not detected (S180: NO) and that there is no end instruction (S190: NO), the terminal device 100 repeats the processing from S150 to S170 described above. . That is, the position and orientation of the terminal device 100 change from moment to moment as the user U moves and orientation changes. The VI request/acquisition process (S160) is executed again, and the displayed virtual image VI is updated (S170). As a result, even if the user U moves or changes posture, the appropriate virtual image VI that matches the real image RI is displayed on the display unit 151 of the terminal device 100 .

Further, when it is determined that the image of the marker unit 260 has been detected in the image captured by the imaging unit 154 (S180: YES), the terminal device 100 executes the above-described processing from S120 onwards again. That is, based on the image of the marker section 260 and the image correspondence information II, the relative position and orientation of the terminal device 100 with respect to the marker section 260 are specified (S120), and the position and orientation of the marker section 260 on the three-dimensional point cloud map are identified. is obtained (S130), and the position and orientation of the terminal device 100 on the three-dimensional point cloud map are estimated (S140). That is, the initial position and initial attitude of the terminal device 100 are updated. Tracking by the terminal device 100 after the tracking start point may accumulate estimation errors in the position and orientation of the terminal device 100, but updating the initial position and initial orientation of the terminal device 100 reduces the errors. Since it is eliminated, continuous highly accurate estimation of the position and orientation of the terminal device 100 is realized. Note that even after the autonomous movement of the autonomous mobile body 200 has ended (S230: YES), the marker position/orientation information LI transmitted to the server device 300 immediately before the end of the autonomous movement can be used. , the initial position and initial orientation of the terminal device 100 can be estimated or updated using the marker position and orientation information LI. In addition, when the terminal device 100 fails in tracking or becomes unable to track on the way, the server device 300 presents the position of the autonomous mobile body 200 near the terminal device 100, or the position of the autonomous mobile body 200 may be changed to move closer to the terminal device 100 . After that, the processing (tracking, etc.) after S150 is similarly executed.

If it is determined that there is an instruction to end the AR information presentation process while the terminal device 100 is executing the above-described processing (S190: YES), the augmented reality processing unit 111 of the terminal device 100 displays the AR information. End the presentation process. Similarly, in the server device 300, when it is determined that there is an instruction to end the AR-type information presentation process (S340: YES), the augmented reality server processing unit 312 of the server device 300 performs the AR-type information presentation process. End the process.

A-3. Effect of the first embodiment:
As described above, the terminal device 100 that constitutes the information processing system 10 of the first embodiment is a device that is held by the user U and moves with the user U, and includes the imaging unit 154, the marker information acquisition unit 113, It includes an image correspondence information acquisition unit 114 , a relative relationship identification unit 115 , and a terminal position/orientation estimation unit 112 . The marker information acquisition unit 113 is an autonomous mobile body 200 that moves while specifying its own position and orientation on a three-dimensional point cloud map configured from a point cloud to which information indicating a three-dimensional absolute position is assigned, For the autonomous mobile body 200 having the marker section 260 functioning as a marker, the marker position/orientation information LI indicating the position and orientation of the marker section 260 on the three-dimensional point cloud map is acquired. The image correspondence information acquisition unit 114 acquires image correspondence information II that associates the image obtained by capturing the marker unit 260 with the relative position and relative orientation with respect to the marker unit 260 . The relative relationship specifying unit 115 specifies relative relationship information CI indicating the relative position and orientation of the terminal device 100 with respect to the marker unit 260 based on the image of the marker unit 260 captured by the imaging unit 154 and the image correspondence information II. . The terminal position/orientation estimation unit 112 estimates the position and orientation of the terminal device 100 on the three-dimensional point cloud map based on the marker position/orientation information LI and the relative relationship information CI.

Thus, in the terminal device 100 of the present embodiment, the marker position/orientation information LI indicating the position and orientation of the marker unit 260 of the autonomous mobile body 200 on the three-dimensional point cloud map, and the relative position of the terminal device 100 to the marker unit 260 Based on the relative relationship information CI indicating the position and relative orientation, the position and orientation of the terminal device 100 on the three-dimensional point cloud map can be estimated. Therefore, according to the terminal device 100 of the present embodiment, as long as the marker unit 260 of the autonomous mobile body 200 can be imaged by the imaging unit 154, it can be performed seamlessly regardless of whether it is indoors or outdoors, and the position and orientation are known. It is not necessary to install a large number of markers in advance, and the position of the terminal device 100 can be determined with high accuracy without installing a sensor (for example, LiDAR) other than the imaging unit 154 or a 3D point cloud map database in the terminal device 100. and pose can be estimated.

Moreover, the terminal device 100 of the present embodiment further includes a tracking section 116 and an estimation result updating section 117 . The tracking unit 116 tracks changes in the position and orientation of the terminal device 100 from when the position and orientation of the terminal device 100 are estimated (tracking start point). The estimation result update unit 117 updates the estimation result of the position and orientation of the terminal device 100 based on the change. Therefore, according to the terminal device 100 of the present embodiment, the position and orientation (initial position and initial orientation) of the terminal device 100 on the three-dimensional point cloud map are once determined based on the marker position and orientation information LI and the relative relationship information CI. After the estimation, regardless of whether or not the marker unit 260 of the autonomous mobile body 200 can be imaged by the imaging unit 154, the terminal device 100 alone can update the estimation result of the position and orientation of the terminal device 100. can. At this time, since the estimation accuracy of the initial position and initial orientation is high, the estimation accuracy of the position and orientation of the terminal device 100 is maintained at a certain level or higher even after subsequent updates. Therefore, according to the terminal device 100 of the present embodiment, the position and orientation of the terminal device 100 that moves independently of the autonomous mobile body 200 can be continuously estimated with high accuracy.

In addition, in the terminal device 100 of the present embodiment, the tracking unit 116 tracks changes in the position and orientation of the terminal device 100 using the image captured by the imaging unit 154 . Therefore, in the terminal device 100 of this embodiment, the position and orientation of the terminal device 100 can be continuously estimated with high accuracy without requiring a sensor other than the imaging unit 154 .

Moreover, the terminal device 100 of the present embodiment further includes a display unit 151 and an augmented reality processing unit 111 . The augmented reality processing unit 111 causes the display unit 151 to display the virtual image VI, which changes according to the estimation result of the position and orientation of the terminal device 100 , superimposed on the real image RI captured by the imaging unit 154 . Therefore, according to the terminal device 100 of the present embodiment, it is possible to display the virtual image VI based on the position and orientation of the terminal device 100 estimated with high accuracy, thereby realizing highly accurate AR information presentation. can be done.

Also, the information processing system 10 of the present embodiment includes a terminal device 100 and an autonomous mobile body 200 . The autonomous mobile body 200 is a mobile body that autonomously moves using peripheral detection information from the sensor 230 and a three-dimensional point cloud map. Therefore, according to the information processing system 10 of the present embodiment, by moving the marker unit 260 without requiring a human driving operation, the terminal device can be operated in a wide range with high accuracy using the moving marker unit 260. 100 position and pose estimates can be implemented.

Moreover, the information processing system 10 of the present embodiment further includes a server device 300 . The autonomous mobile body 200 transmits the marker position/posture information LI to the server device 300 , and the marker information acquisition unit 113 of the terminal device 100 acquires the marker position/posture information LI via the server device 300 . Therefore, according to the information processing system 10 of the present embodiment, even in the terminal device 100 that cannot directly communicate with the autonomous mobile body 200, the terminal device 100 can be detected with high accuracy using the marker unit 260 of the autonomous mobile body 200. Position and pose estimation can be implemented.

B. Second embodiment:
FIG. 9 is an explanatory diagram schematically showing the configuration of an information processing system 10a according to the second embodiment, and FIG. 10 is a block diagram schematically showing the configuration of a terminal device 100a according to the second embodiment. 11 is a block diagram schematically showing the configuration of an autonomous mobile body 200a in the second embodiment, and FIG. 12 is a block diagram schematically showing the configuration of a server device 300a in the second embodiment. Below, among the configurations of the information processing system 10a, the terminal device 100a, the autonomous mobile body 200a, and the server device 300a of the second embodiment and the processing content of each device, the same configuration and processing content as in the first embodiment described above will be described. are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

The information processing system 10a of the second embodiment, like the information processing system 10 of the first embodiment, is a system that highly accurately estimates the position and orientation of the terminal device 100a on the three-dimensional point cloud map. A system for performing hazard notification, rather than performing AR-type information presentation, using estimated position and attitude. In this embodiment, a hazard notification that notifies the presence of a hazard approaching the terminal device 100a at an intersection or the like on a public road with poor visibility will be described as an example.

As shown in FIG. 9, the information processing system 10a includes a plurality of terminal devices 100a, a plurality of autonomous mobile bodies 200a, and a server device 300a. In the second embodiment, the autonomous mobile body 200a is an autonomous vehicle that carries a passenger and autonomously travels on public roads.

As shown in FIG. 10, the storage unit 120 of the terminal device 100a in the second embodiment stores a hazard notification program CP12 for performing hazard notification processing, which will be described later. The control unit 110 of the terminal device 100a reads out the hazard notification program CP12 from the storage unit 120 and executes it, thereby functioning as a hazard notification unit 119 that performs hazard notification processing, which will be described later. The hazard notification unit 119 includes the terminal position/orientation estimation unit 112a similar to that of the first embodiment. However, the terminal position/orientation estimator 112 a of the second embodiment includes an estimation result transmitter 118 . Further, the storage unit 120 of the terminal device 100a stores hazard information HI during hazard notification processing, which will be described later.

As shown in FIG. 11, the storage unit 220 of the autonomous mobile body 200a in the second embodiment stores a hazard notification program CP23 for performing hazard notification processing, which will be described later. The control unit 210 of the autonomous mobile body 200a reads out the hazard notification program CP23 from the storage unit 220 and executes it, thereby functioning as a hazard notification unit 216 that performs hazard notification processing, which will be described later. In addition, terminal position and orientation information TI is stored in the storage unit 220 of the autonomous mobile body 200a at the time of hazard notification processing, which will be described later. The hazard notification process is an example of a specific process in the scope of claims, and the hazard notification unit 216 is an example of a specific process execution unit in the scope of claims.

As shown in FIG. 12, the storage unit 320 of the server device 300a in the second embodiment stores a hazard notification server program CP32 for performing hazard notification processing, which will be described later. The control unit 310 of the server device 300a reads out the hazard notification server program CP32 from the storage unit 320 and executes it, thereby functioning as a hazard notification server processing unit 318 that performs hazard notification processing, which will be described later. A hazard notification server processing unit 318 includes a position/orientation server processing unit 314 similar to that of the first embodiment. Further, the storage unit 320 of the server device 300a stores the terminal position/orientation information TI and the hazard information HI at the time of the hazard notification process, which will be described later.

Next, hazard notification processing executed in the information processing system 10a of the second embodiment will be described. The hazard notification process of the present embodiment is a process of notifying the terminal device 100a of the existence of a hazard approaching the terminal device 100a recognized by the autonomous mobile body 200a and notifying it by the voice output unit 153 of the terminal device 100a. In this embodiment, hazard notification processing for notifying the presence of a hazard approaching the terminal device 100a at an intersection or the like on a public road with poor visibility will be described as an example.

FIG. 13 is an explanatory diagram showing the flow of hazard notification processing executed in the information processing system 10a of the second embodiment. FIG. 14 is an explanatory diagram conceptually showing the hazard notification process executed in the information processing system 10a of the second embodiment. In the example shown in FIG. 14, an autonomous mobile body 200a is traveling within the imaging range of the imaging unit 154 of the terminal device 100a held by the user U at an intersection with poor visibility on a public road. In addition, behind the building Bu, there is a bicycle as a hazard Ha that cannot be visually recognized by the user U but can be recognized by the sensor 230 of the autonomous mobile body 200a. and

In the hazard notification process shown in FIG. 13, the processing of S210 and S220 by the autonomous mobile body 200a, the processing of S310 and S320 by the server device 300a, and the processing contents of S110 to S140 by the terminal device 100a are the first embodiment shown in FIG. are the same as the processing contents of each step in the AR type information presentation processing of . Through these processes, the terminal device 100a can highly accurately estimate the position (initial position) and orientation (initial accuracy) of (the imaging unit 154 of) the terminal device 100a on the three-dimensional point cloud map.

After that, the estimation result transmission unit 118 (FIG. 10) of the terminal device 100a transmits the terminal position and orientation information TI indicating the estimation result of the position and orientation of the terminal device 100a on the three-dimensional point cloud map via the server device 300a. It transmits to the autonomous mobile body 200a. That is, estimation result transmitting section 118 transmits terminal position and orientation information TI to server device 300a (S142), and position and orientation server processing section 314 (FIG. 12) of server device 300a transmits received terminal position and orientation information TI. toward the autonomous mobile body 200a (S322), and the hazard notification unit 216 (FIG. 11) of the autonomous mobile body 200a receives the terminal position and orientation information TI (S222). The terminal position and orientation information TI includes a terminal ID for identifying the terminal device 100a, a time stamp indicating the estimated time of the position and orientation, and information specifying the position and orientation (see FIG. 12). In the present embodiment, the terminal position/attitude information TI is transmitted to the autonomous mobile body 200a existing within a predetermined distance from the estimated position of the terminal device 100a (that is, existing near the terminal device 100a). be done.

After transmitting the terminal position and orientation information TI, the terminal device 100a estimates the position and orientation of the terminal device 100a based on changes in the position and orientation of the terminal device 100a from the initial position and initial orientation, as in the first embodiment. is updated (S150), it is determined whether or not hazard information HI has been received (S152).

The hazard notification unit 216 (FIG. 11) of the autonomous mobile body 200a that has received the terminal position and orientation information TI identifies the position and orientation of the terminal device 100a on the three-dimensional point cloud map based on the terminal position and orientation information TI, and the terminal A hazard Ha around the device 100a is searched (S224). That is, the hazard notification unit 216 determines whether or not there is a peripheral object that can be a hazard Ha for the terminal device 100a based on the relative positional relationship and relative speed relationship between the peripheral object recognized by the sensor 230 and the terminal device 100a.

When the hazard notification unit 216 of the autonomous mobile body 200a detects a hazard Ha around the terminal device 100a (S224: YES), it creates hazard information HI specifying the detected hazard Ha, and sends the hazard information HI. , to the target autonomous mobile body 200a via the server device 300a. That is, the hazard notification unit 216 transmits the hazard information HI to the server device 300a (S226), and the hazard notification server processing unit 318 (FIG. 12) of the server device 300a transmits the received hazard information HI to the autonomous mobile body 200a. (S324), and the hazard notification unit 119 (FIG. 10) of the terminal device 100a receives the hazard information HI (S152: YES). The hazard information HI is, for example, information indicating the position, speed, name, etc. of the hazard Ha. In addition, when the hazard Ha is not detected in the autonomous mobile body 200a (S224:NO), the process of S226 is skipped.

The hazard notification unit 119 of the terminal device 100a that has received the hazard information HI executes notification processing to notify the user U of the presence of the hazard Ha (S154). For example, in the example of FIG. 14, a voice "attention!" is output from the voice output unit 153 of the terminal device 100a. By the notification process, the user U of the terminal device 100a can recognize the existence of the hazard Ha recognized by the autonomous mobile body 200a, although the user U of the terminal device 100a cannot visually recognize the hazard Ha, for example. In addition, in the notification process, the direction of the hazard Ha as seen from the terminal device 100a (for example, front) or the name of the hazard Ha (for example, bicycle) may be notified. Further, in place of the notification by sound, or in addition to the notification by sound, notification by image or notification by vibration may be executed. If the terminal device 100a does not receive the hazard information HI (S152: NO), the process of S154 is skipped. Subsequent processing is the same as in the first embodiment.

As described above, the information processing system 10a of the second embodiment includes the terminal device 100a and the autonomous mobile body 200a. The terminal device 100a includes an estimation result transmission unit 118 that transmits the estimation result of the position and orientation of the terminal device 100a to the autonomous mobile body 200a directly or via another device. The autonomous mobile body 200a includes a hazard notification unit 216 that performs hazard notification processing for notifying the terminal device 100a of the presence of a hazard Ha that is close to the terminal device 100a using the position and posture estimation results of the terminal device 100a. Therefore, according to the information processing system 10a of the second embodiment, using the position and orientation of the terminal device 100a on the three-dimensional point cloud map estimated with high accuracy, the hazard from the autonomous mobile body 200a to the terminal device 100a The presence of Ha can be notified, and highly accurate hazard notification can be realized.

C. Variant:
The technology disclosed in this specification is not limited to the above-described embodiments, and can be modified in various forms without departing from the scope of the invention. For example, the following modifications are possible.

The configuration of the information processing system 10 in the above embodiment is merely an example, and various modifications are possible. For example, in the above-described embodiment, the moving object having the marker unit 260 is the autonomous moving object 200 that autonomously moves using surrounding detection information from the sensor 230 and a 3D point cloud map. As long as the mobile body moves while specifying its own position and posture, it does not necessarily have to be an autonomous mobile body, and may be a mobile body that moves according to a driving operation by a person. Further, in the above embodiment, the autonomous mobile body 200 estimates its own position and orientation by matching the peripheral recognition result of the sensor 230 and the 3D point cloud map. or a method using GNSS) may be combined to estimate the position and orientation of the self with higher accuracy.

In the above-described embodiment, the autonomous mobile body 200 has the image attached to the side surface of the vehicle body as the marker portion 260, but the aspect of the marker portion 260 is not limited to this. For example, the marker section 260 may be a three-dimensional object, and the shape of the autonomous mobile body 200 itself may function as the marker section 260 .

The contents of the AR-type information presentation process and the hazard notification process in the above embodiment are only examples, and can be changed in various ways. For example, in the above embodiment, the position and orientation information providing unit 214 of the autonomous mobile body 200 transmits the marker position and orientation information LI indicating the position and orientation of the marker unit 260 on the three-dimensional point cloud map to the server device 300. (S220 in FIG. 5), the position and orientation server processing unit 314 of the server device 300 is transmitting the marker position and orientation information LI to the terminal device 100 (S320). Information indicating the position and orientation of (the sensor 230 of) the autonomous mobile body 200 on the three-dimensional point cloud map is transmitted to the server device 300 instead of the marker position and orientation information LI, and the position and orientation server processing unit of the server device 300 314 may generate marker position and orientation information LI indicating the position and orientation of the marker unit 260 based on the information, and transmit the information to the terminal device 100 . Further, in the above-described embodiment, exchange of information such as the marker position/orientation information LI between the autonomous mobile body 200 and the terminal device 100 is performed via the server device 300, but the autonomous mobile body 200 and the terminal device Information may be exchanged directly with 100 .

In the above embodiment, the tracking of changes in the position and orientation of the terminal device 100 after the tracking start point is performed using Visual-SLAM. or other known techniques such as techniques that utilize GNSS.

In the above-described first embodiment, the AR-type information presentation that performs guidance display indoors and outdoors in a shopping mall has been described as an example. It is also applicable to AR-type information presentation for sightseeing guides, maintenance guides, location games, etc.).

In the second embodiment, the autonomous mobile body 200a that has received the terminal position and orientation information TI uses the estimation result of the position and orientation of the terminal device 100a to inform the terminal device 100a of the existence of the hazard Ha that is close to the terminal device 100a. Although the hazard notification process to notify is being executed, the autonomous mobile body 200a that has received the terminal position and attitude information TI uses the estimation result of the position and attitude of the terminal device 100a to perform other processing (for example, the precision of the mobile body). guidance process) may be executed. Conversely, when a hazard to the autonomous mobile body 200a is recognized by the imaging unit 154 of the terminal device 100a, the terminal device 100a transmits information on the hazard together with the terminal position and orientation information TI to the autonomous mobile body 200a. You may notify. As a result, the terminal device 100a can compensate for the recognition of the hazard by the autonomous mobile body 200a, and the safety of the autonomous mobile body 200a can be improved.

In the above embodiments, part of the configuration implemented by hardware may be replaced with software, and conversely, part of the configuration implemented by software may be replaced with hardware.

10: Information processing system 100: Terminal device 110: Control unit 111: Augmented reality processing unit 112: Terminal position/orientation estimation unit 113: Marker information acquisition unit 114: Image correspondence information acquisition unit 115: Relative relationship identification unit 116: Tracking unit 117 : estimation result update unit 118: estimation result transmission unit 119: hazard notification unit 120: storage unit 151: display unit 152: operation input unit 153: audio output unit 154: imaging unit 155: communication unit 190: bus 200: autonomous mobile body 210: Control unit 212: Autonomous movement control unit 214: Position/orientation information providing unit 216: Hazard notification unit 220: Storage unit 230: Sensor 250: Driving unit 255: Communication unit 260: Marker unit 300: Server device 310: Control unit 312 : Augmented reality server processing unit 314: Position and orientation server processing unit 318: Hazard notification server processing unit 320: Storage unit 351: Display unit 352: Operation input unit 355: Communication unit 390: Bus Ha: Hazard RI: Real image U: User VI: virtual image

Claims (10)

A terminal device that is held by a user and moves with the user,
an imaging unit;
A moving body that moves while specifying its own position and orientation on a three-dimensional point cloud map composed of a point cloud to which information indicating a three-dimensional absolute position is assigned, the moving body having a marker portion functioning as a marker. a marker information acquisition unit that acquires marker position and orientation information indicating the position and orientation of the marker unit on the three-dimensional point cloud map for the moving object;
an image correspondence information acquisition unit that acquires image correspondence information that associates an image obtained by capturing the marker unit with a relative position and a relative orientation with respect to the marker unit;
a relative relationship identifying unit that identifies relative relationship information indicating the relative position and relative orientation of the terminal device with respect to the marker unit based on the image of the marker unit captured by the imaging unit and the image correspondence information;
a terminal position/orientation estimation unit that estimates the position and orientation of the terminal device on the three-dimensional point cloud map based on the marker position/orientation information and the relative relationship information;
A terminal device. The terminal device according to claim 1, further comprising:
a tracking unit that tracks changes in the position and orientation of the terminal device from when the position and orientation of the terminal device were estimated;
an estimation result updating unit that updates an estimation result of the position and orientation of the terminal device based on the change;
A terminal device. The terminal device according to claim 2,
The terminal device, wherein the tracking unit tracks changes in the position and orientation of the terminal device using the image captured by the imaging unit. The terminal device according to any one of claims 1 to 3, further comprising:
a display unit;
an augmented reality processing unit that superimposes a virtual image that changes according to the estimation result of the position and orientation of the terminal device on the real image captured by the imaging unit and displays the virtual image on the display unit;
A terminal device. a terminal device according to any one of claims 1 to 4;
the moving body;
with
The terminal device further comprises an estimation result transmission unit that transmits the estimation result of the position and orientation of the terminal device to the moving body directly or via another device,
The information processing system, wherein the moving body includes a specific processing execution unit that executes specific processing using the estimation result of the position and orientation of the terminal device. The information processing system according to claim 5,
The information processing system, wherein the specific processing includes hazard notification processing for notifying the terminal device of existence of a hazard in proximity to the terminal device. The information processing system according to claim 5 or claim 6,
The information processing system, wherein the moving object is an autonomous moving object that autonomously moves using peripheral detection information from a sensor and the three-dimensional point group map. The information processing system according to any one of claims 5 to 7, further comprising:
Equipped with a server device,
the moving object transmits the marker position/orientation information to the server device;
The information processing system, wherein the marker information acquisition unit of the terminal device acquires the marker position and orientation information via the server device. An information processing method for estimating the position and orientation of a terminal device that has an imaging unit and is held by a user and moves with the user,
A moving body that moves while specifying its own position and orientation on a three-dimensional point cloud map composed of a point cloud to which information indicating a three-dimensional absolute position is assigned, the moving body having a marker portion functioning as a marker. Acquiring marker position and orientation information indicating the position and orientation of the marker unit on the three-dimensional point cloud map for the moving object;
a step of obtaining image correspondence information that associates an image obtained by imaging the marker section with a relative position and a relative orientation with respect to the marker section;
identifying relative relationship information indicating the relative position and orientation of the terminal device with respect to the marker unit based on the image of the marker unit captured by the imaging unit and the image correspondence information;
estimating the position and orientation of the terminal device on the three-dimensional point cloud map based on the marker position and orientation information and the relative relationship information;
A method of processing information, comprising: A computer program for estimating the position and orientation of a terminal device that has an imaging unit and is held by a user and moves with the user,
to the terminal device,
A moving body that moves while specifying its own position and orientation on a three-dimensional point cloud map composed of a point cloud to which information indicating a three-dimensional absolute position is assigned, the moving body having a marker portion functioning as a marker. a process of acquiring marker position and orientation information indicating the position and orientation of the marker unit on the three-dimensional point cloud map for the moving object;
a process of acquiring image correspondence information that associates an image obtained by capturing the marker section with a relative position and a relative orientation with respect to the marker section;
a process of identifying relative relationship information indicating a relative position and relative orientation of the terminal device with respect to the marker unit based on the image of the marker unit captured by the imaging unit and the image correspondence information;
a process of estimating the position and orientation of the terminal device on the three-dimensional point cloud map based on the marker position and orientation information and the relative relationship information;
computer program that causes the

Images