WO2023149376A1

WO2023149376A1 - Control system, control method, and storage medium

Info

Publication number: WO2023149376A1
Application number: PCT/JP2023/002676
Authority: WO
Inventors: 翼仲谷; 洋平佐藤
Original assignee: キヤノン株式会社
Priority date: 2022-02-01
Filing date: 2023-01-27
Publication date: 2023-08-10

Abstract

Provided is a control system capable of stably detecting its own position, wherein: a unique identifier is imparted to a three-dimensional space defined by a prescribed coordinate system such as latitude/longitude/height, for example; included is a formatting means that associates, with the unique identifier, space information pertaining to the time and the state of an object present in the space, and that formats and saves the same; and the formatting means makes it possible to register key frame information as the space information.

Description

Control system, control method, and storage medium

　The present invention relates to a control system, a control method, a storage medium, etc. related to a three-dimensional space.

In recent years, along with technological innovations such as autonomous driving mobility and spatial recognition systems, the development of an overall picture (hereinafter referred to as digital architecture) that connects data and systems between different organizations and members of society is progressing.

In Patent Document 1, a single processor divides a spatio-temporal area in time and space according to spatio-temporal management data provided by a user to generate a plurality of spatio-temporal divided areas. Also, in consideration of the temporal and spatial proximity of the spatio-temporal segments, an identifier expressed by a one-dimensional integer value is assigned to uniquely identify each of the plurality of spatio-temporal segments.

Then, a spatio-temporal data management system is disclosed that determines the arrangement of time-series data so that data in spatio-temporal divided areas with similar identifiers are arranged closely on the storage device.

JP 2014-002519 A

However, in Patent Document 1, it is only within the processor that generated the data that the data regarding the generated area can be grasped by the identifier. Therefore, users of different systems cannot utilize the information of the spatial division area. In addition, in order for the system user to use the information of the spatio-temporal segmented area, means for accurately detecting his/her own position was required.

Therefore, the present invention provides a control system that can stably detect its own position.

A control system according to the present invention includes:
A formatting means for assigning a unique identifier to a three-dimensional space defined by a predetermined coordinate system, and for formatting and storing spatial information relating to the state and time of an object existing in the space in association with the unique identifier. ,
The formatting means registers key frame information as the spatial information.

According to the present invention, it is possible to provide a control system that can stably detect its own position.

1 is a diagram showing an overall configuration example of an autonomous mobile body control system according to a first embodiment; FIG. (A) is a diagram showing an example of an input screen when a user inputs position information, and (B) is a diagram showing an example of a selection screen for selecting an autonomous mobile body to be used. (A) is a diagram showing an example of a screen for confirming the current position of an autonomous mobile body, and (B) is a diagram showing an example of a map display screen when confirming the current position of an autonomous mobile body. 2 is a functional block diagram showing an internal configuration example of 10 to 15 in FIG. 1; FIG. (A) is a diagram showing the spatial positional relationship between the autonomous mobile body 12 in the real world and the pillar 99 that exists as feature information around it, and (B) shows the autonomous mobile body 12 and the pillar 99 with P0 as the origin. It is a diagram showing a state of mapping in an arbitrary XYZ coordinate system space. 1 is a perspective view showing a mechanical configuration example of an autonomous mobile body 12 according to Embodiment 1. FIG. 3 is a block diagram showing a specific hardware configuration example of a control unit 10-2, a control unit 11-2, a control unit 12-2, a control unit 13-2, a control unit 14-3, and a control unit 15-2; FIG. 4 is a sequence diagram illustrating processing executed by the autonomous mobile body control system according to the first embodiment; FIG. FIG. 9 is a sequence diagram continued from FIG. 8; FIG. 10 is a sequence diagram continued from FIG. 9; (A) is a diagram showing latitude/longitude information of the earth, and (B) is a perspective view showing the predetermined space 100 of (A). 4 is a diagram schematically showing spatial information in space 100. FIG. (A) is a diagram showing route information using map information, (B) is a diagram showing route information using position point cloud data using map information, and (C) is a map showing route information using unique identifiers. It is the displayed figure. FIG. 11 is a functional block diagram related to key frame information storage processing according to the second embodiment; FIG. 12 is a sequence diagram for explaining key frame information storage processing according to the second embodiment; FIG. 10 is an image diagram of format route information 900 of an autonomous mobile body 12 according to Embodiment 2; FIG. 17 is a sequence diagram illustrating the operation of an autonomous mobile body 12 that moves using the format route information 900 of FIG. 16 and the system control device 10 that controls the autonomous mobile body 12;

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are denoted by the same reference numerals, and overlapping descriptions are omitted or simplified.

In addition, although an example applied to control of an autonomous mobile body will be described in the embodiment, the mobile body may be one in which the user can operate at least a part of the movement of the mobile body. That is, for example, various displays related to the moving route and the like may be displayed to the user, and the user may perform a part of the driving operation of the moving body with reference to the display.

<Embodiment 1>
FIG. 1 is a diagram showing an overall configuration example of an autonomous mobile body control system according to Embodiment 1 of the present invention. As shown in FIG. 1, the autonomous mobile body control system (also abbreviated as control system) of the present embodiment includes a system control device 10, a user interface 11, an autonomous mobile body 12, a route determination device 13, conversion information holding It includes a device 14, a sensor node 15, and the like. Here, the user interface 11 means a user terminal device.

In this embodiment, each device shown in FIG. 1 is connected via the Internet 16 by respective network connection units, which will be described later. However, other network systems such as LAN (Local Area Network) may be used.

Also, part of the system control device 10, the user interface 11, the route determining device 13, the conversion information holding device 14, etc. may be configured as the same device.

The system control device 10, the user interface 11, the autonomous mobile body 12, the route determination device 13, the conversion information holding device 14, and the sensor node 15 each contain information such as a CPU as a computer and ROM, RAM, HDD, etc. as storage media. Contains processing equipment. Details of the function and internal configuration of each device will be described later.

Next, the service application software (hereinafter abbreviated as application) provided by the autonomous mobile control system will be explained. In the explanation, first, screen images displayed on the user interface 11 when the user inputs position information will be explained with reference to FIGS. 2(A) and 2(B).

Next, screen images displayed on the user interface 11 when the user browses the current position of the autonomous mobile body 12 will be described with reference to FIGS. 3(A) and 3(B). Also, how the user operates the application in the autonomous mobile body control system will be explained using an example.

In this description, the map display will be described on a two-dimensional plane for the sake of convenience. You can also enter information. That is, in this embodiment, the application can display a 3D map.

　Fig. 2(A) is a diagram showing an example of an input screen when a user inputs position information, and Fig. 2(B) is a diagram showing an example of a selection screen for selecting an autonomous mobile body to be used. When the user operates the display screen of the user interface 11 to access the Internet 16 and select, for example, a route setting application of the autonomous mobile control system, the WEB page of the system control device 10 is displayed.

What is first displayed on the WEB page is an input screen 40 for the departure point, transit point, and arrival point for setting the departure point, transit point, and arrival point when moving the autonomous mobile body 12 . The input screen 40 has a list display button 48 for displaying a list of autonomous mobile bodies to be used (for example, mobilities capable of automatic operation). When the user presses the list display button 48, a mobility list display screen 47 is displayed as shown in FIG. 2(B).

The user first selects the autonomous mobile body to be used (for example, mobility capable of automatic operation) on the list display screen 47 . In the list display screen 47, for example, mobilities M1 to M3 are displayed in a selectable manner, but the number is not limited to this.

When the user selects one of the mobilities M1 to M3 by touch operation, click operation, etc., the screen automatically returns to the input screen 40 of FIG. 2(A). Also, the selected mobility name is displayed on the list display button 48 . After that, the user inputs the location to be set as the starting point in the input field 41 of "starting point".

In addition, the user inputs the location to be set as a transit point in the input field 42 of "transit point 1". It is possible to add a waypoint, and when the add waypoint button 44 is pressed once, an input field 46 for "waypoint 2" is additionally displayed, and the waypoint to be added can be input. .

Every time the button 44 for adding a waypoint is pressed, additional input fields 46 are displayed, such as "waypoint 3" and "waypoint 4", and a plurality of additional waypoints can be input. can. Also, the user inputs a place to be set as the arrival point in the input field 43 of "arrival point". Although not shown in the figure, when the input fields 41 to 43, 46, etc. are clicked, a keyboard or the like for inputting characters is temporarily displayed.

Then, the user can set the movement route of the autonomous mobile body 12 by pressing the decision button 45 . In the example of FIG. 2, "AAA" is set as the departure point, "BBB" is set as the transit point 1, and "CCC" is set as the arrival point.

The text to be entered in the input field may be, for example, an address, latitude information and longitude information (hereinafter also referred to as latitude/longitude information), store name, telephone number, etc., to indicate a specific location. You may enable it to input information.

FIG. 3A is a diagram showing an example of a screen for confirming the current position of an autonomous mobile body, and FIG. 3B is a diagram showing an example of a map display screen when confirming the current position of an autonomous mobile body. be. Reference numeral 50 in FIG. 3(A) denotes a confirmation screen, which is displayed when the user operates an operation button (not shown) after setting the movement route of the autonomous mobile body 12 on the screen as shown in FIG. 2(A). be.

On the confirmation screen 50, the current position of the autonomous mobile body 12 is displayed on the WEB page of the user interface 11, like the current position 56, for example. Therefore, the user can easily grasp the current position.

Also, by pressing the update button 57, the user can update the screen display information to display the latest state. Also, the user can change the place of departure, the waypoint, or the place of arrival by pressing the change waypoint/arrival place button 54 . That is, the user can change the location by inputting the location to be reset in the input field 51 of "departure point", the input field 52 of "route point 1", and the input field 53 of "arrival point".

FIG. 3(B) shows an example of a map display screen 60 that switches from the confirmation screen 50 when the map display button 55 of FIG. 3(A) is pressed. On the map display screen 60, the current location of the autonomous mobile body 12 can be confirmed more easily by displaying the current location 62 on the map. Also, when the user presses the return button 61, the display screen can be returned to the confirmation screen 50 of FIG. 3(A).

As described above, by operating the user interface 11, the user can easily set a movement route for moving the autonomous mobile body 12 from a predetermined location to a predetermined location. Note that such a route setting application can also be applied to, for example, a taxi dispatch service, a drone home delivery service, and the like.

Next, the configuration example and function example of 10 to 15 in FIG. 1 will be explained in detail using FIG. FIG. 4 is a functional block diagram showing an internal configuration example of 10 to 15 in FIG. Some of the functional blocks shown in FIG. 4 are realized by causing a computer (not shown) included in each device to execute a computer program stored in a memory (not shown) as a storage medium.

However, some or all of them may be realized by hardware. As hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like can be used.

Also, each functional block shown in FIG. 4 may not be built in the same housing, and may be configured by separate devices connected to each other via signal paths.

4, the user interface 11 includes an operation unit 11-1, a control unit 11-2, a display unit 11-3, an information storage unit (memory/HDD) 11-4, and a network connection unit 11-5. The user interface 11 is, for example, an information processing device such as a smart phone, a tablet terminal, or a smart watch.

The operation unit 11-1 is composed of a touch panel, key buttons, etc., and is used for data input. The display unit 11-3 is, for example, a liquid crystal screen, and is used to display route information and other data.

The display screen of the user interface 11 shown in FIGS. 2 and 3 is displayed on the display unit 11-3. Using the menu displayed on the display unit 11-3, the user can select a moving route, input information, confirm information, and the like.

In other words, the operation unit 11-1 and the display unit 11-3 provide an operation interface for the user to actually operate. Instead of separately providing the operation section 11-1 and the display section 11-3, a touch panel may be used as both the operation section and the display section.

The control unit 11-2 incorporates a CPU as a computer, manages various applications in the user interface 11, manages modes such as information input and information confirmation, and controls communication processing. Also, it controls the processing in each part in the system controller.

The information storage unit (memory/HDD) 11-4 is a recording medium for holding necessary information such as computer programs to be executed by the CPU. A network connection unit 11-5 controls communication performed via the Internet, LAN, wireless LAN, or the like.

Thus, the user interface 11 of the present embodiment displays the departure point, waypoint, and arrival point input screen 40 on the browser screen of the system control device 10 . In addition, it is possible to accept input of positional information such as a departure point, a waypoint, and an arrival point by the user. The user interface 11 can display the current position of the autonomous mobile body 12 by displaying the confirmation screen 50 and the map display screen 60 on the browser screen.

4, the route determination device 13 includes a map information management unit 13-1, a control unit 13-2, a position/route information management unit 13-3, an information storage unit (memory/HDD) 13-4, and a network connection unit 13. -5. The map information management unit 13-1 holds wide-area map information, searches for route information indicating a route on the map based on designated predetermined position information, and uses the route information of the search result as a position/ It is transmitted to the route information management section 13-3.

In this embodiment, the map information is three-dimensional spatial map information including information such as topography and latitude/longitude/altitude. The map information also includes roadways, sidewalks, direction of travel, and regulation information related to road traffic laws such as traffic regulations.

For example, the map information also includes time-varying traffic regulation information, such as one-way streets depending on the time of day and pedestrian-only roads depending on the time of day. The control unit 13-2 incorporates a CPU as a computer, and controls processing in each unit within the route determination device 13. FIG.

The position/route information management unit 13-3 manages the position information of the autonomous mobile body acquired via the network connection unit 13-5, and transmits the position information to the map information management unit 13-1. Manage the route information as the search result obtained from 13-1. The control unit 13-2 converts the route information managed by the position/route information management unit 13-3 into a predetermined data format according to a request from the external system, and transmits the converted data to the external system.

As described above, in the present embodiment, the route determination device 13 is configured to search for a route in compliance with the Road Traffic Law or the like based on designated position information, and to output the route information in a predetermined data format. It is

The conversion information holding device 14 in FIG. -5 and a network connection unit 14-6.

Further, the conversion information holding device 14 assigns a unique identifier to a three-dimensional space defined by latitude/longitude/height, and associates spatial information about the state and time of objects existing in the space with the unique identifier. It can function as a formatting means to format and save.

The position/route information management unit 14-1 manages predetermined position information acquired through the network connection unit 14-6, and transmits the position information to the control unit 14-3 in accordance with the request of the control unit 14-3. The control unit 14-3 incorporates a CPU as a computer, and controls processing in each unit within the conversion information holding device 14. FIG.

Based on the position information acquired from the position/route information management unit 14-1 and the format information managed by the format database 14-4, the control unit 14-3 converts the position information into a unique identifier defined by the format. Convert to Then, it is transmitted to the unique identifier management section 14-2.

　The format will be explained in detail later, but an identifier (hereinafter referred to as a unique identifier) is assigned to the space starting from a predetermined position, and the space is managed by the unique identifier. In this embodiment, it is possible to acquire the corresponding unique identifier and information in the space based on the predetermined position information.

The unique identifier management unit 14-2 manages the unique identifier converted by the control unit 14-3 and transmits it through the network connection unit 14-6. The format database 14-4 manages format information and transmits the format information to the control unit 14-3 in accordance with a request from the control unit 14-3.

Also, the information in the space acquired through the network connection unit 14-6 is managed using a format. The conversion information holding device 14 manages information related to space acquired by external devices, devices, and networks in association with unique identifiers. In addition, it provides information on unique identifiers and associated spaces to external devices, devices, and networks.

As described above, the conversion information holding device 14 acquires the unique identifier and the information in the space based on the predetermined position information, and can share the information with external devices, devices, and networks connected to itself. managed and provided to Further, the conversion information holding device 14 converts the position information specified by the system control device 10 into a unique identifier and provides the system control device 10 with the unique identifier.

4, the system control device 10 includes a unique identifier management section 10-1, a control section 10-2, a position/path information management section 10-3, an information storage section (memory/HDD) 10-4, and a network connection section 10-. 5. The position/route information management unit 10-3 holds simple map information that associates terrain information with latitude/longitude information, and stores predetermined position information and route information obtained through the network connection unit 10-5. to manage.

The position/route information management unit 10-3 can also divide the route information at predetermined intervals and generate position information such as the latitude/longitude of the divided locations. The unique identifier management unit 10-1 manages information obtained by converting position information and route information into unique identifiers.

The control unit 10-2 has a CPU as a computer, controls the position information, route information, and unique identifier communication functions of the system control device 10, and controls processing in each component of the system control device 10.

In addition, the control unit 10 - 2 provides the user interface 11 with the WEB page and transmits predetermined position information acquired from the WEB page to the route determination device 13 . Further, it acquires predetermined route information from the route determination device 13 and transmits each position information of the route information to the conversion information holding device 14 . Then, the route information converted into the unique identifier acquired from the conversion information holding device 14 is transmitted to the autonomous mobile body 12 .

As described above, the system control device 10 is configured to acquire predetermined position information designated by the user, transmit and receive position information and route information, generate position information, and transmit and receive route information using unique identifiers. there is

In addition, based on the position information input to the user interface 11, the system control device 10 collects route information necessary for the autonomous mobile body 12 to move autonomously, and uses a unique identifier for the autonomous mobile body 12. provide route information. In this embodiment, the system control device 10, the route determination device 13, and the conversion information holding device 14 function as servers, for example.

In FIG. 4, the autonomous moving body 12 includes a detection unit 12-1, a control unit 12-2, a direction control unit 12-3, an information storage unit (memory/HDD) 12-4, a network connection unit 12-5, and a drive unit 12. -6. The detection unit 12-1 has, for example, a plurality of imaging elements, and has a function of performing distance measurement based on phase differences between a plurality of imaging signals obtained from the plurality of imaging elements.

In addition, it has a self-position estimation function that acquires detection information (hereinafter referred to as detection information) such as obstacles such as surrounding terrain and building walls, and estimates its own position based on the detection information and map information.

The detection unit 12-1 also has a self-position detection function such as GPS (Global Positioning System) and a direction detection function such as a geomagnetic sensor. Furthermore, based on the acquired detection information, self-position estimation information, and direction detection information, the control unit 12-2 can generate a three-dimensional map of cyberspace.

Here, a 3D map of cyberspace is one that can express spatial information equivalent to the position of features in the real world as digital data. In this three-dimensional map of cyberspace, the autonomous mobile body 12 that exists in the real world and information on features around it are held as spatially equivalent information as digital data. Therefore, by using this digital data, efficient movement is possible.

The three-dimensional map of cyberspace used in this embodiment will be described below using FIG. 5 as an example. FIG. 5A is a diagram showing the spatial positional relationship between the autonomous mobile body 12 in the real world and a pillar 99 that exists as feature information around it. FIG. 5B shows the autonomous mobile body 12 and the pillar 99. , is a diagram showing a state of mapping in an arbitrary XYZ coordinate system space with the position P0 as the origin.

In FIGS. 5A and 5B, the position of the autonomous mobile body 12 is determined from the latitude and longitude position information acquired by GPS or the like (not shown) mounted on the autonomous mobile body 12. identified as α0. Also, the orientation of the autonomous mobile body 12 is specified by the difference between the orientation αY acquired by an electronic compass (not shown) or the like and the moving direction 12Y of the autonomous mobile body 12 .

Also, the position of the pillar 99 is specified as the position of the vertex 99-1 from position information measured in advance. Also, the distance measurement function of the autonomous mobile body 12 makes it possible to acquire the distance from α0 of the autonomous mobile body 12 to the vertex 99-1. In FIG. 5A, when the moving direction 12Y is the axis of the XYZ coordinate system and α0 is the origin, the coordinates (Wx, Wy, Wz) of the vertex 99-1 are shown.

In the three-dimensional map of cyberspace, the information obtained in this way is managed as digital data, and can be reconstructed as spatial information as shown in FIG. is. FIG. 5B shows a state in which the autonomous mobile body 12 and the pillar 99 are mapped in an arbitrary XYZ coordinate system space with P0 as the origin.

By setting P0 to a predetermined latitude and longitude in the real world and taking the azimuth north of the real world in the Y-axis direction, the autonomous mobile body 12 is expressed as P1 and the pillar 99 as P2 in this arbitrary XYZ coordinate system space. be able to.

Specifically, the position P1 of α0 in this space can be calculated from the latitude and longitude of α0 and the latitude and longitude of P0. Similarly, the column 99 can be calculated as P2. In this example, two of the autonomous mobile body 12 and the pillar 99 are represented by a three-dimensional map of cyber space, but of course, even if there are more, it is possible to treat them in the same way. As described above, a three-dimensional map is a mapping of the self-position and objects in the real world in a three-dimensional space.

Returning to FIG. 4, the autonomous mobile body 12 stores learning result data of object detection that has been machine-learned, for example, in an information storage unit (memory/HDD) 12-4. Objects can be detected. The detection information can also be acquired from an external system via the network connection unit 12-5 and reflected on the three-dimensional map.

The control unit 12-2 has a CPU as a computer, controls movement, direction change, and autonomous running functions of the autonomous mobile body 12, and controls processing in each component of the autonomous mobile body 12.

The direction control unit 12-3 changes the moving direction of the autonomous mobile body 12 by changing the driving direction of the driving unit 12-6. The driving unit 12-6 is composed of a driving device such as a motor, and generates a propulsion force for the autonomous mobile body 12. FIG. The autonomous mobile body 12 reflects its own position, detection information, and object detection information in a three-dimensional map, generates a route that maintains a certain distance from the surrounding terrain, buildings, obstacles, and objects, and performs autonomous driving. can be done.

It should be noted that the route determination device 13 generates routes in consideration of, for example, regulatory information related to the Road Traffic Act. On the other hand, the autonomous mobile body 12 more accurately detects the positions of surrounding obstacles on the route determined by the route determination device 13, and generates a route based on its own size so as to move without touching them.

Also, the information storage unit (memory/HDD) 12-4 of the autonomous mobile body 12 can store the mobility type of the autonomous mobile body itself. The type of mobility is the type of mobile object, such as automobiles, bicycles, and drones. Formatted route information, which will be described later, can be generated based on this mobility format.

Here, an example of the main body configuration of the autonomous mobile body 12 in this embodiment will be described using FIG. FIG. 6 is a perspective view showing a mechanical configuration example of the autonomous mobile body 12 according to the first embodiment. In this embodiment, the autonomous mobile body 12 will be described as an example of a traveling body having wheels, but is not limited to this, and may be a flying body such as a drone.

In FIG. 6, the autonomous moving body 12 includes a detection unit 12-1, a control unit 12-2, a direction control unit 12-3, an information storage unit (memory/HDD) 12-4, a network connection unit 12-5, a drive unit 12-6 are mounted and each component is electrically connected to each other. At least two drive units 12-6 and direction control units 12-3 are provided in the autonomous mobile body 12. FIG.

The direction control unit 12-3 changes the moving direction of the autonomous mobile body 12 by changing the direction of the driving unit 12-6 by rotating the shaft, and the driving unit 12-6 rotates the autonomous mobile body by rotating the shaft. Perform 12 forwards and backwards. The configuration described with reference to FIG. 6 is an example, and the present invention is not limited to this. For example, an omniwheel or the like may be used to change the movement direction.

The autonomous mobile body 12 is, for example, a mobile body using SLAM (Simultaneous Localization and Mapping) technology. Further, based on the detection information detected by the detection unit 12-1 or the like and the detection information of the external system acquired via the Internet 16, it is configured so that it can autonomously move along a designated predetermined route.

The autonomous mobile body 12 can perform trace movement by tracing finely specified points, and can also generate route information by itself in the space between them while passing through roughly set points and move. It is possible. As described above, the autonomous mobile body 12 of this embodiment can autonomously move based on the route information using the unique identifier provided by the system control device 10 .

Returning to FIG. 4, the sensor node 15 is an external system such as a video surveillance system such as a roadside camera unit. , and a network connection unit 15-4. The detection unit 15-1 is an imaging unit composed of, for example, a camera, and acquires detection information of an area in which the detection unit 15-1 can detect itself, and has an object detection function and a distance measurement function.

The control unit 15-2 incorporates a CPU as a computer, controls the detection of the sensor node 15, data storage, and data transmission functions, and controls processing in each unit within the sensor node 15. Further, the detection information acquired by the detection unit 15-1 is stored in the information storage unit (memory/HDD) 15-3 and transmitted to the conversion information holding device 14 through the network connection unit 15-4.

As described above, the sensor node 15 is configured so that detection information such as image information detected by the detection unit 15-1, feature point information of a detected object, and position information can be stored in the information storage unit 15-3 and communicated. It is Also, the sensor node 15 provides the conversion information holding device 14 with detection information of the area detectable by itself.

Next, the specific hardware configuration of each control unit in FIG. 4 will be described. FIG. 7 is a block diagram showing a specific hardware configuration example of the control unit 10-2, the control unit 11-2, the control unit 12-2, the control unit 13-2, the control unit 14-3, and the control unit 15-2. It is a diagram. Note that the hardware configuration is not limited to that shown in FIG. Moreover, it is not necessary to have all the blocks shown in FIG.

In FIG. 7, 21 is a CPU as a computer that manages the calculation and control of the information processing device. The RAM 22 is a recording medium that functions as a main memory of the CPU 21, an execution program area, an execution area for the program, and a data area. The ROM 23 is a recording medium in which an operation processing procedure (program) of the CPU 21 is recorded.

The ROM 23 includes a program ROM that records basic software (OS), which is a system program for controlling the information processing device, and a data ROM that records information necessary for operating the system. Note that an HDD 29, which will be described later, may be used instead of the ROM 23. FIG.

The network I/F 24 is a network interface (NETIF) that controls data transfer between information processing devices via the Internet 16 and diagnoses the connection status. A video RAM (VRAM) 25 develops an image to be displayed on the screen of the LCD 26 and controls the display. The LCD 26 is a display device such as a display (hereinafter referred to as LCD).

The controller 27 is a controller (hereinafter referred to as KBC) for controlling input signals from the external input device 28 . The external input device 28 is an external input device (hereinafter referred to as KB) for receiving operations performed by the user, and a pointing device such as a keyboard or mouse is used, for example.

The HDD 29 is a hard disk drive (hereinafter referred to as HDD) and is used for storing application programs and various data. The application program in this embodiment is a software program or the like that executes various processing functions in this embodiment.

The CDD 30 is an external input/output device (hereinafter referred to as CDD). For example, it is for inputting/outputting data from/to a removable medium 31 as a removable data recording medium such as a CDROM drive, a DVD drive, a Blu-Ray (registered trademark) disk drive, and the like.

The CDD 30 is used, for example, when reading the above application program from removable media. 31 is a removable medium such as a CDROM disk, DVD, Blu-Ray disk, etc., which is read by the CDD 30 .

The removable medium may be a magneto-optical recording medium (eg, MO), a semiconductor recording medium (eg, memory card), or the like. It is also possible to store the application programs and data stored in the HDD 29 in the removable medium 31 and use them. Reference numeral 20 denotes a transmission bus (address bus, data bus, input/output bus, and control bus) for connecting the units described above.

　Next, the details of the control operation in the autonomous mobile body control system for realizing the applications described in FIGS. 2 and 3 will be described using FIGS. 8 to 10. FIG. FIG. 8 is a sequence diagram illustrating processing executed by the autonomous mobile body control system according to the first embodiment, FIG. 9 is a sequence diagram following FIG. 8, and FIG. 10 is a sequence diagram following FIG. is.

8 to 10 show the processing executed by each device from when the user inputs position information to the user interface 11 until the current position information of the autonomous mobile body 12 is received. 8 to 10 are executed by the computers in the control units 10 to 15 executing the computer programs stored in the memory.

First, in step S201, the user uses the user interface 11 to access the WEB page provided by the system control device 10. In step S202, the system control device 10 displays the position input screen as described with reference to FIG. 2 on the display screen of the WEB page. In step S203, as described with reference to FIG. 2, the user selects an autonomous mobile object (mobility) and inputs position information (hereinafter referred to as position information) indicating a departure point, a transit point, and an arrival point.

The location information may be a word (hereinafter referred to as a location word) that specifies a specific location such as a building name, station name, or address, or a specific location on a map displayed on a web page as a point (hereinafter referred to as a point). It is also possible to use a method to

In step S204, the system control device 10 saves the type information of the selected autonomous mobile body 12 and the input information such as the input position information. At this time, if the position information is a position word, the system controller 10 stores the position word. Also, if the position information is a point, the system control device 10 searches for the latitude and longitude corresponding to the point based on the simple map information stored in the position/route information management unit 10-3, Store latitude and longitude.

Next, in step S205, the system control device 10 designates the type of route that can be moved (hereinafter referred to as route type) from the mobility type (moving body type) of the autonomous mobile body 12 designated by the user. Then, in step S206, it is transmitted to the route determination device 13 together with the positional information.

As mentioned above, the type of mobility is, for example, a legally distinguished type of mobile object, such as a car, bicycle, or drone. The route types are, for example, general roads, expressways, motorways, predetermined sidewalks, roadside strips of general roads, and bicycle lanes.

For example, if the mobility type is a car, the route type is specified as a general road, expressway, or car-exclusive road. In addition, when the mode of mobility is a bicycle, a predetermined sidewalk, a side strip of a general road, a bicycle lane, etc. are specified.

In step S207, the route determination device 13 inputs the received positional information into the owned map information as a departure point, a transit point, and an arrival point. If the positional information is a positional word, a search (preliminary search) is performed using the map information using the positional word, and the relevant latitude/longitude information is input. If the position information is latitude/longitude information, it is used by inputting it into the map information as it is. Furthermore, the route determination device 13 may search for routes in advance.

Subsequently, in step S208, the route determination device 13 searches for a route from the departure point to the arrival point via the intermediate points. At this time, the route to be searched is searched according to the route type. If a route has been searched in advance in step S208, the route searched in advance is appropriately changed based on the route type.

Then, in step S209, the route determination device 13 outputs, as a result of the search, a route from the departure point to the arrival point via the waypoints (hereinafter referred to as route information) in GPX format (GPS eXchange Format), and system control is performed. Send to device 10 .

GPX format files are mainly divided into three types: waypoints (point information without order), routes (point information with order with time information added), and tracks (collection of multiple point information: trajectories). is configured to

In addition, latitude/longitude is described as the attribute value of each point information, altitude, geoid height, GPS reception status/accuracy, etc. are described as child elements. The minimum element required for a GPX file is latitude/longitude information for a single point, and any other information is optional. A route is output as route information, and is a set of point information consisting of latitude/longitude having an order relationship. Note that the route information may be in another format as long as it satisfies the above requirements.

Here, a configuration example of formats managed by the format database 14-4 of the conversion information holding device 14 will be described in detail with reference to FIGS. 11(A), 11(B) and 12. FIG.

FIG. 11(A) is a diagram showing latitude/longitude information of the earth, and FIG. 11(B) is a perspective view showing the predetermined space 100 in FIG. 11(A). Also, in FIG. 11B, the center of the predetermined space 100 is defined as the center 101. As shown in FIG. FIG. 12 is a diagram schematically showing spatial information in the space 100. As shown in FIG.

In FIGS. 11(A) and 11(B), the format divides the three-dimensional space of the earth into spaces for each predetermined unit volume determined by the range starting from latitude/longitude/height. A unique identifier is added to the space to make it manageable. For example, here the space 100 is displayed as a predetermined three-dimensional space.

A space 100 is defined by a center 101 of 20 degrees north latitude, 140 degrees east longitude, and height (altitude, altitude) H, and the width in the latitudinal direction is defined as D, the width in the longitudinal direction as W, and the width in the height direction as T. is a partitioned space. In addition, it is one space obtained by dividing the space of the earth into spaces determined by ranges starting from latitude/longitude/height.

For the sake of convenience, only the space 100 is shown in FIG. 11(A), but in the definition of the format, spaces defined in the same manner as the space 100 are arranged side by side in the latitude/longitude/height directions as described above. shall be It is assumed that each of the arranged divided spaces has its horizontal position defined by latitude/longitude, overlaps in the height direction, and the position in the height direction is defined by height.

Although the center 101 of the divided space is set as the starting point of latitude/longitude/height in FIG. 11B, the starting point is not limited to this. It is good as Also, the shape may be a substantially rectangular parallelepiped, and when considering the case of laying on a spherical surface such as the earth, it is better to set the top surface of the rectangular parallelepiped slightly wider than the bottom surface, so that it can be arranged without gaps.

Taking the space 100 as an example in FIG. 12, in the format database 14-4, information (spatial information) on the types of objects that exist or can enter the range of the space 100 and time limits are associated with unique identifiers. ) is formatted and stored. Also, the formatted spatial information is stored in chronological order from the past to the future. Note that, in the present embodiment, associating and linking are used in the same meaning.

That is, the conversion information holding device 14 associates with the unique identifier the spatial information regarding the types of objects that can exist or can enter a three-dimensional space defined by latitude/longitude/height and the time limit, and formats the format database 14-. Saved in 4.

The spatial information is updated at predetermined update intervals based on information supplied by information supply means such as an external system (for example, the sensor node 15) communicatively connected to the conversion information holding device 14. Then, the information is shared with other external systems communicably connected to the conversion information holding device 14 .

In applications that do not require time-related information, it is also possible to use spatial information that does not contain time-related information. Also, non-unique identifiers may be used instead of unique identifiers.

In addition, information on operators/individuals who have external systems, information on how to access detection information acquired by external systems, and specification information on detection information such as metadata/communication format of detection information are also used as spatial information, as unique identifiers. can be associated and managed.

As described above, in the first embodiment, information about the type of an object that can exist or enter a three-dimensional space defined by latitude/longitude/height and the time limit (hereinafter referred to as spatial information) is associated with a unique identifier. formatted and stored in the database. Space-time can be managed by formatted spatial information.

In addition, in this embodiment, latitude/longitude/height will be used as a coordinate system that defines the position of the space (voxel). However, the coordinate system is not limited to this, and various coordinate systems can be used, such as an XYZ coordinate system having arbitrary coordinate axes, or using MGRS (Military Grid Reference System) as horizontal coordinates. .

In addition, it is also possible to use a pixel coordinate system that uses the pixel positions of an image as coordinates, or a tile coordinate system that divides a predetermined area into units called tiles and expresses them by arranging them in the X/Y directions.

In addition, the conversion information holding device 14 of the first embodiment executes a formatting step of formatting and saving information about update intervals of spatial information in association with unique identifiers. The update interval information formatted in association with the unique identifier may be the update frequency, and the update interval information includes the update frequency.

Returning to FIG. 8, the continuation of the processing executed by the autonomous mobile body control system will be described. In step S210, the system control device 10 confirms the interval between each piece of point information in the received route information. Then, the position point group data is created by matching the interval of the point information with the interval between the starting point positions of the divided spaces defined by the format.

At this time, if the interval of the point information is smaller than the interval between the starting point positions of the divided spaces, the system control device 10 thins out the point information in the route information according to the interval of the starting point positions of the divided spaces, and uses it as position point cloud data. do. Also, if the interval of point information is larger than the interval between the starting point positions of the divided spaces, the system control device 10 interpolates the point information within a range that does not deviate from the route information to obtain position point group data.

　Next, as shown in step S211 in Fig. 9, the system control device 10 transmits the latitude/longitude information of each point information of the position point cloud data to the conversion information holding device 14 in the order of the route. In step S212, the conversion information holding device 14 searches the format database 14-4 for a unique identifier corresponding to the received latitude/longitude information, and transmits it to the system control device 10 in step S213.

In step S214, the system control device 10 arranges the received unique identifiers in the same order as the original position point cloud data, and stores them as route information using the unique identifiers (hereinafter referred to as format route information). Thus, in step S214, the system control device 10 as the route generation means acquires the spatial information from the database of the conversion information holding device 14, and based on the acquired spatial information and the type information of the mobile object, Generating route information about travel routes.

Here, the process of generating position point cloud data from route information and converting it into route information using unique identifiers will be described in detail with reference to FIGS. explain. FIG. 13(A) is an image diagram of route information displayed as map information, FIG. 13(B) is an image diagram of route information using position point cloud data displayed as map information, and FIG. 13(C) is an image diagram using unique identifiers. FIG. 10 is an image diagram showing route information as map information;

In FIG. 13(A), 120 is route information, 121 is a non-movable area through which the autonomous mobile body 12 cannot pass, and 122 is a movable area where the autonomous mobile body 12 can move. The route information 120 generated by the route determining device 13 based on the positional information of the departure point, waypoints, and arrival points designated by the user passes through the departure point, waypoints, and arrival points, and is displayed on the map information. It is generated as a route passing over the movable area 122 .

In FIG. 13(B), 123 is a plurality of pieces of position information on route information. After acquiring the route information 120 , the system control device 10 generates position information 123 arranged at predetermined intervals on the route information 120 .

The position information 123 can be represented by latitude/longitude/height, respectively, and this position information 123 is called position point cloud data in the first embodiment. Then, the system control device 10 transmits the position information 123 (latitude/longitude/height of each point) one by one to the conversion information holding device 14 and converts them into unique identifiers.

In FIG. 13(C), 124 is positional space information in which the positional information 123 is converted into a unique identifier one by one, and the spatial range defined by the unique identifier is represented by a rectangular frame. The location space information 124 is obtained by converting the location information into a unique identifier. As a result, the route represented by the route information 120 is converted into continuous position space information 124 and represented.

Each piece of position space information 124 is associated with information about the types of objects that can exist or enter the space and the time limit. This continuous position space information 124 is called format route information in the first embodiment.

Returning to FIG. 9, the continuation of the processing executed by the autonomous mobile body control system will be described. After step S214, in step S215, the system control device 10 downloads the spatial information associated with each unique identifier of the format path information from the conversion information holding device 14. FIG.

Then, in step S216, the system control device 10 converts the space information into a format that can be reflected in the three-dimensional map of the cyberspace of the autonomous mobile body 12, and information indicating the positions of multiple objects (obstacles) in the predetermined space. (hereafter, cost map). The cost map may be created with respect to the space of all routes in the format route information at first, or may be created in a form divided by fixed areas and updated sequentially.

Next, in step S217, the system control device 10 associates the format route information and the cost map with the unique identification number (unique identifier) assigned to the autonomous mobile body 12 and stores them.

The autonomous mobile body 12 monitors (hereinafter, polling) its own unique identification number via the network at predetermined time intervals, and downloads the associated cost map in step S218. In step S219, the autonomous mobile body 12 reflects the latitude/longitude information of each unique identifier of the format route information as route information on the three-dimensional map of cyberspace created by itself.

Next, in step S220, the autonomous mobile body 12 reflects the cost map on the three-dimensional map of cyberspace as obstacle information on the route. If the cost map is created in a form that is divided at regular intervals, after moving the area in which the cost map was created, download the cost map of the next area and update the cost map.

In step S221, the autonomous mobile body 12 moves along the route information while avoiding the objects (obstacles) entered in the cost map. That is, movement control is performed based on the cost map.

At this time, in step S222, the autonomous mobile body 12 moves while performing object detection, and moves while updating the cost map using the object detection information if there is a difference from the cost map. Also, in step S223, the autonomous mobile body 12 transmits difference information from the cost map to the system control device 10 together with the corresponding unique identifier.

The system control device 10 that has acquired the difference information between the unique identifier and the cost map transmits the spatial information to the conversion information holding device 14 in step S224 of FIG. Update the spatial information of the unique identifier.

The content of the spatial information updated here does not directly reflect the difference information from the cost map, but is abstracted by the system control device 10 and then sent to the conversion information holding device 14 . Details of the abstraction will be described later.

In step S226, the autonomous mobile body 12 moving based on the format route information informs the system controller 10 of the space it is currently passing through each time it passes through the divided space associated with each unique identifier. Send the associated unique identifier.

Alternatively, you can associate it with your own unique identification number when polling. Based on the space unique identifier information received from the autonomous mobile body 12, the system control device 10 grasps the current position of the autonomous mobile body 12 on the format route information.

By repeating step S226, the system control device 10 can grasp where the autonomous mobile body 12 is currently located in the format route information. Incidentally, the system control device 10 may stop holding the unique identifier of the space through which the autonomous mobile body 12 has passed, thereby reducing the holding data capacity of the format route information.

In step S227, the system control device 10 creates the confirmation screen 50 and the map display screen 60 described with reference to FIGS. do. The system control device 10 updates the confirmation screen 50 and the map display screen 60 each time a unique identifier indicating the current position is transmitted from the autonomous mobile body 12 to the system control device 10 .

On the other hand, in step S228 of FIG. 8, the sensor node 15 saves the detection information of the detection range, abstracts the detection information in step S229, and transmits it as spatial information to the conversion information holding device 14 in step S230. Abstraction is information such as whether or not an object exists, or whether or not the state of existence of the object has changed, and is not detailed information about the object.

　Detailed information about the object is stored in the memory inside the sensor node. Then, in step S231, the conversion information holding device 14 stores the spatial information, which is the abstracted detection information, in association with the unique identifier of the position corresponding to the spatial information. This results in spatial information being stored in one unique identifier in the format database.

Further, when an external system different from the sensor node 15 utilizes the spatial information, the external system uses the spatial information in the conversion information holding device 14 to obtain the detected information in the sensor node 15 via the conversion information holding device 14. is obtained and utilized. At this time, the conversion information holding device 14 also has a function of connecting the communication standards of the external system and the sensor node 15 .

By storing spatial information as described above not only in the sensor node 15 but also among multiple devices, the conversion information holding device 14 has a function of connecting data of multiple devices with a relatively small amount of data. In steps S215 and S216 of FIG. 9, when the system control device 10 needs detailed object information when creating a cost map, detailed information is sent from an external system storing detailed detection information of spatial information. should be downloaded and used.

Here, assume that the sensor node 15 updates the spatial information on the route of the format route information of the autonomous mobile body 12 . At this time, the sensor node 15 acquires detection information in step S232 of FIG. 10, generates abstracted spatial information in step S233, and transmits it to the conversion information holding device 14 in step S234. The conversion information holding device 14 stores the spatial information in the format database 14-4 in step S235.

The system control device 10 checks changes in the spatial information in the managed format path information at predetermined time intervals, and downloads the spatial information in step S236 if there is a change. Then, in step S237, the cost map associated with the unique identification number assigned to the autonomous mobile body 12 is updated.

In step S238, the autonomous mobile body 12 recognizes the update of the cost map by polling and reflects it in the three-dimensional map of cyberspace created by itself.

As described above, by utilizing spatial information shared by a plurality of devices, the autonomous mobile body 12 can recognize in advance a change in the route that the self cannot recognize, and can respond to the change.
After performing the above series of systems, when the autonomous mobile body 12 arrives at the arrival point in step S239, a unique identifier is transmitted in step S240.

Upon recognizing the unique identifier, the system control device 10 displays an arrival indication on the user interface 11 in step S241, and terminates the application.

According to Embodiment 1, it is possible to provide a digital architecture format and an autonomous mobile body control system using the format as described above.

As described with reference to FIGS. 11A, 11B, and 12, the format database 14-4 stores information (spatial information) about the types of objects that can exist or enter the space 100 and the time limit from the past. It is stored in chronological order such as the future. The spatial information is updated based on information input from an external sensor or the like communicably connected to the conversion information holding device 14, and is shared with other external systems that can be connected to the conversion information holding device 14. .

One of these spatial information is the type information of objects in the space. The type information of objects in the space here is information that can be obtained from map information, such as roadways, sidewalks, and bicycle lanes on roads. In addition, information such as the traveling direction of mobility on a roadway, traffic regulations, etc. can also be defined as type information. Furthermore, as will be described later, it is also possible to define type information in the space itself.

The coordinated operation of the conversion information holding device 14 and the system control device 10 that controls the autonomous mobile body 12 has been described above with reference to FIG. However, in addition to the system control device 10, the conversion information holding device 14 can be connected to a system control device that manages information on roads and a system control device that manages information on sections other than roads.

That is, as described above, the system control device 10 can transmit position point cloud data collectively representing the position information 123 of FIG. Similarly, a system control device that manages information on roads and a system control device that manages information on sections other than roads can also transmit corresponding data to the conversion information holding device 14 .

The corresponding data is the position point cloud data information managed by the system control device that manages road information and the system control device that manages information on sections other than roads. Each point of the position point cloud data is hereinafter referred to as a position point.

After transmission, it is stored in association with the unique identifier of the format database 14-4, and by updating the information as appropriate, the current real world information is accurately reflected in the conversion information holding device 14, and the autonomous mobile body Make sure that there is no hindrance to the movement of 12.

It should be noted that in Embodiment 1, the space information update interval differs according to the type of object existing in the space. That is, when the type of object existing in the space is a moving object, the length of time is set to be shorter than when the type of object existing in the space is not a moving object. Also, when the type of the object existing in the space is a road, the type of the object existing in the space is made shorter than in the case of the partition.

Also, when there are multiple objects in the space, the update interval of the space information about each object should be different according to the type of each object (eg moving body, road, section, etc.). Spatial information about the state and time of each of a plurality of objects existing in the space is associated with the unique identifier, formatted and stored. Therefore, the load for updating spatial information can be reduced.

In the description of the present embodiment, when creating format route information (to be described later) from position point cloud data, position Thinning out/interpolating point cloud data.

However, the present invention is not limited to this, and at least the interval of the point information constituting the position point cloud data should be equal to or greater than the interval between the starting point (=reference point) positions of the divided spaces so that the divided spaces do not overlap each other. You can set the route of movement.

The closer the space between the position point cloud data, the more detailed the movement route can be specified, but on the other hand, the amount of data for the entire movement route increases. Also, if the interval between the position point cloud data is large, the moving route cannot be specified in detail, but the amount of data for the entire moving route can be suppressed.

In other words, it is possible to appropriately adjust the intervals between position point cloud data according to conditions such as the instruction granularity of the movement route to the autonomous mobile body 12 and the amount of data that can be handled. Also, it is possible to partially change the interval between the position point cloud data to set a more optimal route.

<Embodiment 2>
In the second embodiment, in addition to the configuration of the first embodiment, when an autonomous mobile body cannot detect its own position accurately by the self-position detection function, it detects its own position based on the information stored in the format database and moves. is configured as

In step S226 of FIG. 10, the operation of the system control device 10 to grasp the current position of the autonomous mobile body 12 on the format route information based on the unique identifier information received from the autonomous mobile body 12 has been described. Here, whether or not the autonomous mobile body 12 has passed through the divided space associated with each unique identifier is determined by a self-position detection function such as GPS in the detection unit 12-1.

Therefore, the accuracy may decrease when traveling in places where GPS radio waves are difficult to receive, such as underground or tunnels. Also, an autonomous mobile body that does not have a self-position detection function such as GPS cannot tell the system control device where it is currently in the format route information, and the autonomous mobile body control in the first embodiment The system could not be used.

Therefore, in the present embodiment, the conversion information holding device 14 as formatting means associates information characterizing the space (hereinafter referred to as key frame information) with the unique identifier of the position to which the information corresponds (string registered (saved or recorded). The key frame information, which will be described later in detail, is, for example, information indicating an object for which the same object does not exist around the associated space.

By using the keyframe information, the autonomous mobile body 12 can estimate its own position. As described in FIG. 4 of the first embodiment, the autonomous mobile body 12 can detect an object from a captured image using machine learning or the like. Therefore, by detecting an object included in the keyframe information with this object detection function, it is possible to estimate (specify) that the detection point is the position corresponding to the unique identifier associated with the keyframe information. is.

In this embodiment, the operation of detecting an object included in keyframe information by using a captured image or the like and specifying a position corresponding to a unique identifier associated with the keyframe information is referred to as "keyframe information to match”.

By "collating key frame information", the autonomous mobile body 12 identifies its own position even if it is a point where GPS radio waves are difficult to receive or even if the autonomous mobile body does not have a self-position detection function. can do. The self position is the position corresponding to the unique identifier to which the key frame information is linked, which is specified by the autonomous mobile body 12 . Therefore, the autonomous mobile body 12 can transmit the position information to the system control device that controls the autonomous mobile body.

In this embodiment, keyframe information is one type of spatial information. For example, as described in the first embodiment, the format database 14-4 stores information (spatial information) regarding the state and time of objects existing within the space 100 in chronological order from the past to the future.

Further, the spatial information is updated by information input by an external system or the like communicatively connected to the conversion information holding device 14, and information is shared with other external systems communicably connected to the conversion information holding device 14. . As one of these pieces of spatial information, the transformation information holding device 14 records keyframe information of objects in space.

Next, the procedure for storing key frame information in the format database 14-4 will be explained using the block diagram of FIG. 14 and the sequence diagram of FIG. FIG. 14 is a functional block diagram related to key frame information storage processing according to the second embodiment, in which the internal configurations of the map information extraction system 901, the sensor node 15, the autonomous moving body 612, and the conversion information holding device 14 are shown as functional blocks. showing.

In FIG. 14, the autonomous moving body 612 includes a detection unit 612-1, a control unit 612-2, a direction control unit 612-3, an information storage unit (memory/HDD) 612-4, a network connection unit 612-5, and a drive unit 612. -6.

612-1 to 612-6 have the same configuration as 12-1 to 12-6, which is the configuration of the autonomous mobile body 12 described in FIG. 4, so detailed description will be omitted. In addition, the autonomous mobile body 612 has an object detection function and a distance measurement function like the autonomous mobile body 12 described in FIG.

14, the map information extraction system 901 includes a map information management unit 901-1, a control unit 901-2, an information extraction unit 901-3, an information storage unit (memory/HDD) 901-4, and a network connection unit 901. -5.

The map information management unit 901-1 holds map information of the three-dimensional space on the earth. The information extraction unit 901-3 has a function of extracting information based on set conditions such as location information, place names, facility names, etc. from the map information held by the map information management unit 901-1.

The control unit 901-2 controls map information extraction in the map information extraction system 901, and sets conditions for extraction by the information extraction unit 901-3. Then, for example, a function of storing information extracted under conditions satisfying requirements as key frame information in an information storage unit (memory/HDD) 901-4 and transmitting it to the conversion information holding device 14 through a network connection unit 901-5. control. The conversion information holding device 14 and the sensor node 15 in FIG. 14 have the same configurations as those described in FIG. 4, so description thereof will be omitted.

With the configuration shown in FIG. 14, in the second embodiment, the map information extraction system 901 extracts object information that satisfies requirements as key frame information from the map information, and provides (outputs) the information to the conversion information holding device 14 . do.

FIG. 15 is a sequence diagram for explaining keyframe information storage processing according to the second embodiment. Map information extraction system 901, sensor node 15, autonomous moving body 612, and conversion information storage device related to keyframe information storage are shown in FIG. 14 indicates the processing to be executed. The computer in the

control units

14, 15, 612 and 901 executes the computer program stored in the memory to perform the operation of each step in the sequence of FIG.

First, the map information extraction system 901 as an example of an external system communicably connected to the conversion information holding device 14 extracts key frame information from the map information and stores it in the format database 14-4 with reference to FIG. explain.

In step S401, the map information extraction system 901 extracts information on objects that satisfy the requirements for key frame information from a predetermined range of owned map information. The object information that satisfies the requirements of the key frame information extracted here is information indicating an object that does not have the same object in its surroundings.

The details of the keyframe information will be described later, but for example, signs such as guide signs (road signs) with proper nouns such as "XX intersection" and "YY post office", and billboards.

Thereafter, in step S402, the map information extraction system 901 stores (stores) the extracted key frame information in the information storage unit (memory/HDD) 901-4 in association with the position information of the object. .

Further, in step S403, the map information extraction system 901 transmits the linked key frame information and position information to the conversion information holding device 14.

At step S404, the conversion information holding device 14 determines the unique identifier corresponding to the transmitted position information, and stores the key frame information corresponding to the unique identifier in the format database 14-4.

In this way, it is possible to register (store) the keyframe information corresponding to the unique identifier from the map information in the format database 14-4. Here, step S404 functions as a formatting step for formatting and registering the key frame information as spatial information in association with the unique identifier.

Next, as another means of storing keyframe information, a method of storing keyframe information based on information acquired by the sensor node 15 will be described. In FIG. 14, the sensor node 15 is an external system such as a video surveillance system such as a roadside camera unit, and the information storage unit (memory/HDD) 15-3 stores position information of the sensor node itself. It is

First, in step S411, the sensor node 15 uses the object detection function and the distance measurement function of the detection unit 15-1 to collect information such as image information, feature point information, and position information of an object existing in an area detectable by itself. to detect Then, in step S412, the sensor node 15 stores the detected information in the information storage unit (memory/HDD) 15-3.

In step S413, the control unit 15-2 of the sensor node 15 extracts object information that satisfies the requirements for keyframe information as keyframe information from the object image information and feature point information. Then, in step S414, the control unit 15-2 of the sensor node 15 associates the key frame information with the position information of the object that satisfies the requirements of the key frame information, and stores the information in the information storage unit (memory/HDD) 15-3. Record.

An object that satisfies the requirements of the keyframe information extracted here is, for example, an object that does not have the same object in its vicinity, as in the case of extracting information from map information. Other objects that meet the requirements for keyframe information are described below.

In step S415, the sensor node 15 transmits the mutually associated (linked) key frame information and position information to the conversion information holding device 14. In step S416, conversion information holding device 14 determines a unique identifier corresponding to the position information transmitted in step S415, and records key frame information corresponding to the unique identifier in format database 14-4.

In this way, the key frame information corresponding to the unique identifier can be stored in the format database 14-4 from the information detected by the sensor node 15.

Next, as another means of storing keyframe information, we will explain how to store keyframe information based on the information acquired by the autonomous mobile body. First, in step S421, the autonomous mobile body 612 detects information such as image information, feature point information, position information, etc. of an object existing in an area detectable by itself by its object detection function and distance measurement function.

Then, in step S422, the autonomous mobile body 612 stores the information detected in step S421 in the information storage unit (memory/HDD) 612-4. Furthermore, in step S423, the control unit 612-2 of the autonomous mobile body 612 extracts the information of the object that satisfies the requirements of the keyframe information as the keyframe information from the image information and feature point information of this object.

Then, in step S424, the key frame information and the position information of the object that satisfies the requirements of the key frame information are associated (linked) and stored in the control unit 612-2.

The objects that meet the requirements of the keyframe information extracted here are objects that do not have the same object in their surroundings, just like when extracting information from map information. In addition, as an object that satisfies the requirements of key frame information, there is also "image data of scenery at that point" photographed by the autonomous mobile body 612, and the like.

In step S425, the autonomous mobile body 612 transmits the mutually associated (linked) key frame information and position information to the conversion information holding device 14. In step S426, the conversion information holding device 14 determines the unique identifier corresponding to the transmitted position information, and stores the key frame information corresponding to the unique identifier in the format database 14-4. In this way, the key frame information corresponding to the unique identifier can be stored in the format database 14-4 from the information detected by the autonomous mobile body 612. FIG.

In this way, the keyframe information is stored in association with (linked to) a predetermined unique identifier in the format database 14-4. Here, in this embodiment, this key frame information is updated at appropriate timing.

For example, it is desirable that keyframe information stored from map information be updated when map information is updated. Further, for example, in the keyframe information stored by the sensor node 15 in step S424, it is determined that the information in the range detectable by the sensor node 15 changes and that change affects the keyframe information. It should be updated when That is, it is desirable that the keyframe information be updated, for example, when an object stored as keyframe information moves.

In addition, it is desirable to update the keyframe information stored by the autonomous mobile body 612 when, for example, the same point is replaced by another new keyframe information associated with it. However, a plurality of pieces of keyframe information may be linked to one divided space. In this case, information indicating an object that satisfies the requirements of the keyframe information is added to the keyframe information.

Next, the process when the autonomous mobile body 612 moves using the key frame information on a route with low reception strength of GPS radio waves will be described using FIGS. 16 and 17. FIG. It is assumed that on a route where the reception strength of GPS radio waves is low, the accuracy of the self-position detection function using GPS that the autonomous mobile body 612 originally has decreases, and an error of about ±5 m occurs, for example. It is also assumed that the direction detection function using the geomagnetic sensor does not interfere with the route where the GPS radio wave reception strength is low, and the direction of the geomagnetism of the autonomous moving body 612 is correctly detected.

FIG. 16 is an image diagram of the format route information 900 of the autonomous mobile body 612 according to the second embodiment. Formatted route information 900 in FIG. 16 is set so that the route passes through divided spaces 902-1 and 902-2 in which key frame information is stored.

The keyframe information stored in the divided space 902-1 is the information indicating the guide sign "XX intersection", and the keyframe information stored in the divided space 902-2 is the signboard of "YY post office". It is assumed that it is information that indicates It is also assumed that the autonomous mobile body 612 and the system control device 10 that controls the autonomous mobile body 612 know the position of the autonomous mobile body 612 at the departure point.

FIG. 17 is a sequence diagram explaining the operation of the autonomous mobile body 612 that moves using the format route information 900 of FIG. 16 and the system control device 10 that controls the autonomous mobile body 612. Each step of the sequence of FIG. 17 is performed by the computer in each of the

control units

10 and 12 executing the computer program stored in the memory.

First, in step S431, the system control device 10 instructs the autonomous mobile body 612 at the starting point shown in FIG. 16 to start moving rightward on the page of FIG.

In step S432, the autonomous mobile body 612 that has received the instruction starts moving rightward on the paper surface of FIG. 16 based on the information obtained from the direction detection function using the geomagnetic sensor.

After that, in step S433, when the autonomous mobile body 612 reaches a position of about ±5 m near the divided space 902-1, the imaging means of the autonomous mobile body 612 detects the guide sign "XX intersection". Based on the detected guide sign, the key frame information stored in the divided space 902-1 is collated.

Furthermore, in step S434, the autonomous mobile body 612 calculates the distance to the detected guide sign marked "XX intersection" using the distance measuring function. Subsequently, in step S435, the autonomous mobile body 612 calculates its own relative position information from the divided space 902-1 based on the distance calculation result. Then, in step S436, based on this position information, the self position on the format route information 900 is reflected in the three-dimensional map of the cyberspace created by the self.

Furthermore, the autonomous mobile body 612 also transmits this position information to the system control device 10 in step S437. In step S438, the system control device 10 grasps the current position of the autonomous mobile body 612 on the format route information 900 based on the position information received from the autonomous mobile body 612. FIG. The processing of steps S433 to S438 is continuously performed until the autonomous mobile body 612 reaches the divided space 902-1.

When the autonomous mobile body 612 reaches the divided space 902-1, in step S439, the autonomous mobile body 612 turns left based on the three-dimensional map of cyber space created in step S436 and the direction detection function (upward direction on the paper surface of FIG. 16 to).

Next, in step S440, when the autonomous mobile body 612 reaches a position of about ±5 m near the divided space 902-2, the imaging means of the autonomous mobile body 612 detects the "YY Post Office" signboard. Then, the autonomous moving body 612 collates the keyframe information stored in the divided space 902-1 based on the detected signboard.

Furthermore, in step S441, the autonomous moving body 612 calculates the distance to the detected "YY Post Office" signboard by the distance measurement function. Subsequently, in step S442, the autonomous mobile body 612 calculates its own relative position information from the divided space 902-2 based on the distance calculation result. Then, in step S443, it is reflected in the three-dimensional map of the cyberspace created by itself, and is also transmitted to the system control device 10 in step S444.

In step S445, based on the relative position information from the divided space 902-2 to the autonomous mobile body 612 received from the autonomous mobile body 612, the system control device 10 of the autonomous mobile body 612 on the format route information 900 Know your current location.
The processing of steps S440 to S445 is also continuously performed until the autonomous mobile body 612 arrives at the point of the key frame information.

When the autonomous mobile body 612 arrives at the point of the key frame information, the autonomous mobile body 612 and the system controller 10 determine that the autonomous mobile body 612 has reached the divided space 902-2 on the format route information 900, that is, has reached the destination. you can figure out what you did. Then, the system control device 10 issues an instruction to end the movement to the autonomous mobile body 612 . Then, in step S447, the autonomous mobile body 612 ends its movement.

Next, we will explain what kind of information can be used as keyframe information. So far, "information indicating an object that does not have the same object in its surroundings" has been described as an example of key frame information. An object around which the same object does not exist is, for example, an object including a proper noun. Guide signs and signboards on which proper nouns such as "XX intersection" and "YY post office" are written are considered to exist only at that point in the area.

Therefore, if a proper noun can be detected, even an autonomous mobile body that does not have a self-position detection function such as GPS can accurately identify its position at that point.

For example, objects such as statues (including statues and sculptures) such as the statue of Hachiko, the faithful dog in front of Shibuya Station, and characteristic buildings such as Tokyo Tower are also objects that satisfy the keyframe information requirements. In addition, even an object such as a "post" or a "traffic light" having similar shapes scattered in a three-dimensional space can be an object that satisfies the requirements of key frame information.

For example, in Japan, a "post" is installed at a point where there is no other post within a radius of 400m from a certain post. Therefore, even if there is an error of about ±100 m in the accuracy of the self-position detection function using GPS that the autonomous mobile body has, it will not be confused with another post.

Also, even if the autonomous mobile body does not have a self-position detection function in the first place, it is possible to avoid confusion with another post by making a composite judgment with other key frame information detected before and after. There is

For example, suppose that an autonomous mobile object with a moving speed of 4 km/h detects the key frame information "post" 3 minutes after collating the key frame information that can completely identify the point (point A). . The distance that this moving body can move in 3 minutes is 200 m, and if there is only one post within the range of 200 m from point A, that point can be completely identified.

"Existence information of objects whose point cannot be specified completely" depends on the self-location detection function of the user and how to use it. Therefore, the autonomous mobile body needs to determine whether the key frame information is information that can be used by itself. Therefore, it is desirable that "existence information of an object whose point cannot be completely specified" and "existence information of an object whose point can be specified completely" are classified even if they are the same key frame information.

For example, "existence information of an object whose point can be completely specified" is class A keyframe information, and "existence information of an object that does not have the same object within a range of ±10 km" is class B keyframe information. Attach class information.

In addition, class information such as "existence information of an object that does not have the same object within a range of ±200 m" is associated with class C key frame information. This makes it possible to reduce the burden of determining whether the information is usable.

Another example of object information that satisfies the requirements for keyframe information is "image data of the landscape at that point" captured by the autonomous mobile body 612 or the like. That is, the keyframe information includes image data captured at a point corresponding to the space.

This image data is compared with the actual scenery in front of the eyes by means of imaging means of an autonomous mobile body, etc., and if it is possible to analyze whether the matching rate of the feature points is above a certain level, the key frame information is collated. can be specified.

However, even if the image data is for the same point, if the shooting direction, shooting time, shooting equipment, and other conditions are significantly different, the matching rate of the feature points will decrease, making it difficult to match. Therefore, it is desirable to link the parameter data relating to the photographing conditions under which the image was photographed to the "image data of the landscape at that point" and store it in the format database 14-4. That is, the parameter data includes shooting conditions when image data is shot in the target space.

And the parameter data includes the position information of the point where the image was taken. If you know from which point the image data was captured, for example, by adding an action such as turning the autonomous mobile side in that direction or rotating the camera, the matching rate of the feature points can be adjusted and the difficulty of matching can be lowered. This is because

Also, information about the lighting conditions when the image data was shot can be parameter data. That is, the shooting conditions include information about the brightness at the time of shooting. This is because the lighting conditions at the time of photographing greatly affect the image quality and color of the entire image, and the edges and shadows of objects in the image.

Here, the state of lighting refers to, for example, the date and time when the photograph was taken, how strong the light was at the time of photographing and from which direction, whether there was a light source such as a street nearby, and if there was a streetlight, At least one of from which direction, color (color temperature) and illuminance is included.

By storing these conditions as parameter data, for example, an image captured by an autonomous mobile body can be processed such as image correction so as to be closer to the conditions of the parameter data, and the degree of matching can be increased. .

　In addition, it is desirable to include information on the shooting equipment used as parameter data. For example, if setting information such as the ID of the photographing equipment of the autonomous mobile body 612 that has been photographed, data regarding optical characteristics, and white balance can be prepared in advance, correction processing can be performed on the image photographed by the autonomous mobile body.

This is because the degree of matching can be increased. Also, in order to effectively use these parameter data, it is desirable to prepare in advance a data table relating to the name of shooting equipment, optical characteristics, image quality at the time of shooting, gain, and white balance setting information.

If it is difficult to collect the above parameter data in an accurate form, the parameter data may be stored as estimated values. An estimated value is, for example, an estimated value related to brightness. Among the brightness data collected as parameter data of key frame information of different points, data with similar conditions such as shooting time and topography may be stored and used as simple parameter data. .

Furthermore, another example of object information that satisfies the requirements for keyframe information is 3D data of the landscape at or from that point. If the autonomous mobile body has a function that can determine the shape of the object, such as LiDAR (Light Detection And Ranging), it will be possible to collate the 3D data stored as key frame information. That is, the keyframe information includes 3D data at points corresponding to the target space.

For example, by comparing the 3D data, which is the stored keyframe information, with the result of distance measurement by LiDAR, and analyzing whether the matching rate of the feature points is above a certain level, the keyframe information can be verified. It becomes possible to specify the location. The 3D data stored as key frame information may be only the shape of a characteristic object that is easy to analyze, instead of the data of all the points.

It should be noted that LiDAR has the feature that its ranging performance depends on the surface condition of the target. Therefore, by storing the surface information of the object having a characteristic shape in the 3D data, which is the key frame information, as the parameter data in association with the 3D data, the degree of matching can be increased.

That is, the formatting means can improve the degree of matching by enabling parameter data for matching key frame information to be registered as spatial information in association with key frame information. The parameter data also includes surface information of the 3D data object. If it is difficult to collect the parameter data here in an accurate form, the parameter data may be estimated values estimated from conditions such as the shape of the object and topography.

The difficulty of matching keyframe information largely depends on the parameter data, but if the parameter data is an estimated value, it is difficult to accurately estimate the difficulty of matching the keyframe information in advance.

It is necessary to avoid a situation in which an autonomous mobile body misunderstands that it can collate keyframe information that cannot be collated with its own detection and analysis capabilities, and loses sight of its current position after reaching that point. Therefore, it is desirable to estimate the degree of difficulty of collation, which is whether the key frame information can be collated by itself, at a high level at the route setting stage.

In general, when image data is shot under darker conditions than bright conditions, it is often difficult to recognize the image. Therefore, when image data is used as keyframe information and parameter data is stored as an estimate, if there are multiple candidates for the parameter data, it is desirable to adopt parameter data under brighter conditions as the estimated value.

Also, when accurate values can be newly collected for parameter data, it is desirable to update the estimated values to the newly collected values.

In addition, when 3D data is used as key frame information and parameter data is estimated and stored, if the estimation is difficult, the value of an object with low reflectance that is difficult to measure with LiDAR etc. , for example, 10%, which is the reflectance of black paper. As a result, it is possible to overestimate the degree of matching difficulty on the autonomous mobile side.

According to the above embodiments, it is possible to efficiently improve the safety of an autonomous mobile body control system using spatial information of a three-dimensional space defined by a predetermined coordinate system such as latitude/longitude/height. .

In addition, in the above-described embodiment, an example in which the control system is applied to an autonomous mobile body has been described. However, the mobile object of this embodiment is not limited to an autonomous mobile object such as an AGV (Automated Guided Vehicle) or an AMR (Autonomous Mobile Robot).

For example, it can be any mobile device that moves, such as automobiles, trains, ships, airplanes, robots, and drones. Also, part of the control system of the present embodiment may or may not be mounted on those moving bodies. Also, this embodiment can be applied to remote control of a moving object.

Although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention. They are not excluded from the scope of the invention. It should be noted that the present invention includes combinations of the multiple embodiments described above.

The present invention may be realized by supplying a storage medium recording software program code (control program) for realizing the functions of the above-described embodiments to a system or device. It is also achieved by the computer (or CPU or MPU) of the system or apparatus reading and executing the computer-readable program code stored in the storage medium. In that case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

(Cross reference to related applications)
This application is the previously filed Japanese Patent Application No. 2022-014166 filed on February 1, 2022, Japanese Patent Application No. 2022-110828 filed on July 8, 2022, 2023 It claims the benefit of Japanese Patent Application No. 2023-002447 filed on January 11th. Also, the contents of the above Japanese patent application are incorporated herein by reference in their entirety.

Claims

A formatting means for assigning a unique identifier to a three-dimensional space defined by a predetermined coordinate system, and for formatting and storing spatial information relating to the state and time of an object existing in the space in association with the unique identifier. ,
A control system according to claim 1, wherein said formatting means registers key frame information as said spatial information.
　The control system according to claim 1, wherein the key frame information includes existence information of an object that does not have the same object around a point corresponding to the space.
　The control system according to claim 1, wherein the key frame information includes image data taken at a point corresponding to the space.
　The control system according to claim 1, wherein the key frame information includes 3D data at a point corresponding to the space.
The control system according to claim 1, wherein the formatting means can register, as the spatial information, parameter data for matching the key frame information in association with the key frame information.
The key frame information includes image data taken at a point corresponding to the space,
6. The control system according to claim 5, wherein said parameter data includes photographing conditions under which said image data was photographed.
　The control system according to claim 6, wherein the shooting conditions include position information of the spot where the shooting was performed.
　The control system according to claim 6, wherein the shooting conditions include information about brightness at the time of shooting.
　The control system according to claim 6, wherein the shooting conditions include information about the shooting equipment used for the shooting.
The key frame information includes 3D data of points corresponding to the space,
6. The control system of claim 5, wherein the parameter data includes surface information of objects in the 3D data.
The control system according to claim 5, wherein the parameter data includes estimated values.
3. The control system according to claim 1, further comprising route generating means for generating route information relating to the movement route of said moving body based on said spatial information obtained from said formatting means and type information of said moving body.
The control system according to claim 1, characterized in that said formatting means formats the information on update intervals of said spatial information in association with said unique identifier.
14. The control system according to claim 13, wherein the information about the update interval differs depending on the type of object existing in the space.
15. A method according to claim 14, wherein the information about the update interval is shorter when the type of object existing in the space is a moving object than when the type of object existing in the space is not a moving object. control system.
A formatting step of assigning a unique identifier to a three-dimensional space defined by a predetermined coordinate system, and formatting and storing spatial information about the state and time of an object existing in the space in association with the unique identifier. ,
The control method, wherein the formatting step can register key frame information as the spatial information.
A storage medium storing a computer program for executing each step of the following control method, the control method comprising:
A formatting step of assigning a unique identifier to a three-dimensional space defined by a predetermined coordinate system, and formatting and storing spatial information about the state and time of an object existing in the space in association with the unique identifier. ,
The formatting step is characterized in that key frame information can be registered as the spatial information.