JP7461817B2 - Display information creating device, display system, and display information creating method - Google Patents

Description

This disclosure relates to a display information creation device, a display system, and a display information creation method.

As a means of recognizing the surrounding conditions of a moving object and avoiding interference with the surrounding environment, a technology is known in which three-dimensional map data of the terrain, including installed objects, is created and presented to the driver or supervisor (for example, Patent Document 1).

JP 2019-200351 A

When displaying a three-dimensional map on a display screen, the viewpoint from which the map is displayed, the number of display screens and the number of viewpoints, the display size, etc. can be set. These display settings must be set to any desired value by the operator.

The present disclosure has been made in consideration of the above, and aims to provide a display information creation device, a display system, and a display information creation method that can easily set the display format when displaying three-dimensional map data on a screen.

The display information creation device disclosed herein includes a point cloud data acquisition unit that periodically acquires point cloud data including distance information between a moving body and each point constituting the surface of a feature present in a specified area, a position information acquisition unit that acquires position information of the moving body, a map data creation unit that converts the point cloud data into a common coordinate system based on the position information to create three-dimensional map data and synthesizes it as needed, a screen setting unit that can set display settings including a combination of setting values of screen setting data related to the display format of the display screen of the display unit, and a display information creation unit that creates display information including a map identified by a specified viewpoint of the three-dimensional map data based on the display settings.

The display system of the present disclosure also includes the display information creation device, the mobile body that can move within the specified area and communicate with the display information creation device, an operation unit that can input operations to the display information creation device and the mobile body, and a display unit that can display the display information, and is equipped with a base station that can communicate with the display information creation device and the mobile body.

The display information creation method of the present disclosure includes the steps of periodically acquiring point cloud data including distance information between a moving body and each point constituting the surface of a feature present in a specified area, acquiring position information of the moving body, converting the point cloud data into a common coordinate system based on the position information to create three-dimensional map data and synthesizing the data as needed, setting display settings including a combination of setting values of screen setting data related to the display format of the display screen of the display unit, and creating display information including a map identified by a specified viewpoint of the three-dimensional map data based on the display settings.

The display information creation device, display system, and display information creation method disclosed herein make it easy to set the display format when displaying three-dimensional map data on a screen.

FIG. 1 is a schematic diagram showing an example of the configuration of a display system according to the first embodiment. FIG. 2 is a block diagram showing the internal configuration of the display information creating device according to the first embodiment together with the peripheral configuration. FIG. 3 is an explanatory diagram showing the range in which point cloud data is acquired by a moving body. FIG. 4 is a schematic diagram showing an example of the position and direction of the viewpoint when displaying three-dimensional map data. FIG. 5 is a schematic diagram showing an example of a display screen. FIG. 6 is a flowchart showing an example of processing of the display system according to the first embodiment. FIG. 7 is a block diagram showing the internal configuration of a display information creating device according to the second embodiment together with its peripheral configuration. FIG. 8 is a flowchart showing an example of processing of the display system according to the second embodiment. FIG. 9 is a block diagram showing the internal configuration of a display information creating device according to the third embodiment together with its peripheral configuration. FIG. 10 is a flowchart showing an example of processing of the display system according to the third embodiment. FIG. 11 is a block diagram showing the internal configuration of a display information creating device according to the fourth embodiment together with its peripheral configuration. FIG. 12 is a flowchart showing an example of processing of the display system according to the fourth embodiment. FIG. 13 is a block diagram showing the internal configuration of a display information creating device according to the fifth embodiment together with its peripheral configuration. FIG. 14 is a flowchart showing an example of processing of the display system according to the fifth embodiment.

The display information creation device, display system, and display information creation method according to the present disclosure will be described in detail below with reference to the drawings. Note that the present disclosure is not limited to the description of the following embodiments. In addition, the components in the following embodiments include those that are replaceable and easy for a person skilled in the art, those that are substantially the same, or those that are equivalent. Furthermore, the components described below can be omitted, replaced, or modified in various ways without departing from the gist of the present disclosure. In the following embodiments, components necessary for illustrating the embodiments of the display information creation device, display system, and display information creation method according to the present disclosure will be described, and other components will be omitted. Note that in the following description of the embodiments, the same components will be given the same reference numerals, and different components will be given different reference numerals.

In the present invention, the display system is a system that displays display information including three-dimensional map data of the terrain including various features such as the ground surface, the bottom surface, buildings, and obstacles, regarding the surrounding environment of a mobile body moving on the ground, in the air, on the water, and underwater. The mobile body includes, for example, a mobile body that requires an operator, such as a vehicle such as an automobile, an aircraft, a ship, a submarine, or a manned work robot, and an autonomous or remotely operated mobile body such as an advanced safety vehicle, an automated guided vehicle (AGV), an unmanned aerial vehicle (UAV), an autonomous underwater vehicle (AUV), or an unmanned work robot. In each embodiment described below, a display system that displays display information including three-dimensional map data to a remote operator regarding the surrounding environment of a mobile body that is a vehicle such as an automobile traveling on the ground is described.

First Embodiment
First, the configuration of a display system 1 according to the first embodiment will be described. Fig. 1 is a schematic diagram showing an example of the configuration of the display system 1 according to the first embodiment. The display system 1 includes a predetermined area 2, a base station 3, and a communication network 4.

The specified area 2 is the site for which display information including three-dimensional map data is to be created. The specified area 2 is in an environment that includes various features such as the ground surface, buildings, and obstacles (for example, features X1, X2, Y1, and Y2 in Figures 3 and 4).

A mobile body 10 capable of moving in the specified area 2 is deployed. In the first embodiment, the mobile body 10 is a vehicle. The mobile body 10 includes a housing 12, a moving mechanism 14, a communication unit 16, a control unit 18, and a measuring device 20.

In the first embodiment, the moving mechanism 14 includes wheels provided on the bottom surface of the housing 12. The moving mechanism 14 may include crawlers. The moving mechanism 14 realizes the movement of the moving body 10 on the ground including the predetermined area 2 by driving the wheels with power transmitted from a drive mechanism housed inside the housing 12.

The communication unit 16 can transmit and receive various information to and from a base station 3 located away from the specified area 2 via the communication network 4. The communication unit 16 outputs various control signals received from the base station 3 via the communication network 4 to the control unit 18. The communication unit 16 transmits various information acquired by the measurement device 20 to the base station 3 via the communication network 4.

The control unit 18 controls each component of the mobile body 10 to cause the mobile body 10 to perform various operations in the specified area 2. The control unit 18 controls the communication unit 16 and the measurement device 20. The control unit 18 includes a storage device that stores various control programs and data used in various control processes, and a processing unit that executes predetermined control programs.

The measuring device 20 measures various pieces of information necessary for creating three-dimensional map data regarding the mobile object 10 and the surrounding environment of the mobile object 10 in the specified area 2. The measuring device 20 uses, for example, SLAM (Simultaneous Localization and Mapping) technology. The measuring device 20 performs various measurements based on control signals from the control unit 18. The measuring device 20 outputs communication signals of the various pieces of measured information to the communication unit 16. In the first embodiment, the measuring device 20 includes a laser surveying sensor 22, a color data acquisition sensor 24, a self-position estimation sensor 26, and an imaging camera 28.

The laser surveying sensor 22 includes, for example, a three-dimensional laser scanner. The laser surveying sensor 22 scans features present in the surrounding environment of the mobile body 10 in the specified area 2 with laser light, and measures distances to each point constituting the surface of the feature. The laser surveying sensor 22 recognizes the surface of the features present in the specified area 2 as a point cloud, which is a collection of many points, and obtains point cloud data including distance information from the measurement point where the laser surveying sensor 22 is located to each point.

The color data acquisition sensor 24 acquires color data of features present in the surrounding environment of the mobile body 10 in the specified area 2. The color data acquisition sensor 24 acquires color data of the features as two-dimensional information by acquiring reflected light from the features. Therefore, in the first embodiment, the features present in the specified area 2 are assumed to have a brightness at least sufficient for the color data acquisition sensor 24 to acquire valid color data due to the mobile body 10, lighting devices installed in or near the specified area 2, or sunlight. Valid color data means color data that contains color information equal to or greater than a threshold set so that the operator can easily recognize it when the color data is added to the corresponding point cloud data and displayed on the display unit 34 of the base station 3 described below.

The color data acquisition sensor 24 includes a camera capable of acquiring color data. The color data acquisition sensor 24 includes, for example, a general camera such as a monocular camera, or a depth camera or stereo camera capable of simultaneously acquiring distance data to an object. In the first embodiment, since the distance data is acquired by the laser survey sensor 22, it is possible to simplify the configuration and reduce costs by adopting a simpler general camera.

The self-position estimation sensor 26 estimates the self-position of the moving body 10 in the specified area 2 and acquires the estimation result as position information. The method of acquiring the position information of the moving body 10 by the self-position estimation sensor 26 may include various known methods.

For example, when the mobile body 10 is in an environment where it can receive a GPS (Global Positioning System) signal, the self-position estimation sensor 26 may estimate the self-position of the mobile body 10 based on the positioning information from the GPS. Also, when the mobile body 10 is in an environment where it is difficult to receive a GPS signal, the self-position estimation sensor 26 may estimate the self-position of the mobile body 10 based on the rotational speed or steering angle of the wheels in the drive mechanism of the mobile body 10. More specifically, the self-position estimation sensor 26 may estimate the self-position of the mobile body 10 using odometry that geometrically determines the relative displacement from the initial position in the specified area 2, scan matching that reversely estimates the amount of movement by aligning the point cloud data acquired by the laser survey sensor 22 in a time series, or the like. In addition, when the mobile body moves underwater in the present invention, the self-position estimation sensor 26 may include, for example, an inertial navigation system (INS).

In the first embodiment, the laser surveying sensor 22, color data acquisition sensor 24, and self-position estimation sensor 26 of the measuring device 20 are arranged at intervals that are sufficiently narrow relative to the size of the specified area 2. Therefore, in the following description, the laser surveying sensor 22, color data acquisition sensor 24, and self-position estimation sensor 26 are considered to have a common measurement point.

The imaging camera 28 captures an arbitrary range of the surrounding environment of the moving body 10 in the specified area 2, thereby acquiring video data with the position of the imaging camera 28 as the viewpoint. The imaging camera 28 may be a camera constituting the color data acquisition sensor 24, or a camera of a drive recorder mounted on the moving body 10, etc.

The base station 3 is a server computer capable of transmitting and receiving various information to and from the communication unit 16 of the mobile object 10 via the communication network 4. The base station 3 transmits, for example, various control signals for operating the measurement device 20 of the mobile object 10 to the mobile object 10 via the communication network 4. The base station 3 receives, via the communication network 4, various information acquired by the measurement device 20 from the mobile object 10. The base station 3 may be capable of communicating with, for example, multiple mobile objects 10. In addition, a portion of the configuration of the base station 3 may be provided in a cloud server capable of network communication. The base station 3 includes an operation unit 32, a display unit 34, and a display information creation device 40.

The operation unit 32 can accept various operations on the moving body 10 and the display information creation device 40 by the operator. The operation unit 32 can accept, for example, operations related to the movement of the moving body 10, such as moving forward, backward, turning, accelerating, decelerating, and stopping. The operation unit 32 outputs an operation signal corresponding to the accepted operation on the moving body 10 to the display information creation device 40. The operation unit 32 can accept, for example, an operation to edit the screen setting data stored in the screen setting data storage unit 62 of the screen setting unit 60. The operation unit 32 outputs an operation signal corresponding to the accepted operation on the display information creation device 40 to the display information creation device 40. The operation unit 32 can be realized, for example, by a physical switch, a keyboard, or a touch panel provided on the display unit 34. In the example shown in FIG. 1, the operation unit 32 is realized by a keyboard connected to the display information creation device 40 by wire or wirelessly.

The display unit 34 includes a display screen 36 that displays various images. The display unit 34 displays, for example, colored point cloud data obtained by combining point cloud data and color data, three-dimensional map data obtained by combining colored point cloud data, and display information including video data on the display screen 36. The display unit 34 displays the display information created by the display information creation device 40 based on a video signal of the display information output from a display information output unit 58 of the display information creation device 40, which will be described later.

The display unit 34 is a display device including, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 34 may include a touch panel including the operation unit 32. The display unit 34 may be formed integrally with the display information creation device 40. The display format of the display unit 34 may include other formats that can be recognized by the operator using the five senses. The display unit 34 may include, for example, a speaker that notifies predetermined notification information by sound, a light-emitting device that notifies predetermined notification information by light, etc.

The display information creation device 40 displays display information including three-dimensional map data to the operator, and functions as the main server of the display system 1. The display information creation device 40 also functions as a map creation device that creates three-dimensional map data. The display information creation device 40 includes a storage device that stores various control programs and data used in various control processes, an arithmetic processing device that executes predetermined control programs, and an input/output interface device as a communication means.

The display information creation device 40, for example, causes the measuring device 20 of the mobile object 10 to measure various information of the surrounding environment in the specified area 2, and acquires various information including the measured point cloud data, color data, and position information via the communication network 4. The display information creation device 40 creates colored point cloud data, for example, based on the acquired point cloud data and color data of the surrounding environment in the specified area 2. The display information creation device 40 creates three-dimensional map data, for example, based on the colored point cloud data and the acquired position information of the mobile object 10. The display information creation device 40 creates display information including three-dimensional map data, for example, based on screen setting data of the display screen 36 set by the operator from the operation unit 32, and outputs the display information to the display unit 34.

Next, the internal configuration of the display information creation device 40 of the first embodiment will be described. FIG. 2 is a block diagram showing the internal configuration of the display information creation device 40 of the first embodiment together with the peripheral configuration. The display information creation device 40 includes a point cloud data acquisition unit 42, a color data acquisition unit 44, a position information acquisition unit 46, a video data acquisition unit 48, a colored point cloud data creation unit 50, a map data creation unit 52, a map data storage unit 54, a display information creation unit 56, a display information output unit 58, a screen setting unit 60, and a mapping control unit 70.

The point cloud data acquisition unit 42 acquires the point cloud data acquired by the laser surveying sensor 22 of the mobile body 10 at a predetermined period via the communication network 4 (see FIG. 1). The point cloud data acquired by the display information creation device 40 in the point cloud data acquisition unit 42 includes distance information between each point constituting the surface of a feature existing in the specified area 2 and the measurement point where the laser surveying sensor 22 of the mobile body 10 is located. The point cloud data acquisition unit 42 outputs the acquired point cloud data to the colored point cloud data creation unit 50 and the map data creation unit 52.

The color data acquisition unit 44 acquires the color data acquired by the color data acquisition sensor 24 of the moving body 10 at a predetermined cycle via the communication network 4 (see FIG. 1). The color data acquired by the display information creation device 40 in the color data acquisition unit 44 includes color information corresponding to each point included in the point cloud data acquired by the point cloud data acquisition unit 42. The color data acquired by the display information creation device 40 in the color data acquisition unit 44 includes color information equal to or greater than a threshold value set so that the operator can easily recognize it. Note that the cycle and timing at which the color data acquisition unit 44 acquires the color data are preferably the same as the cycle and timing at which the point cloud data acquisition unit 42 acquires the point cloud data. The color data acquisition unit 44 outputs the acquired color data to the colored point cloud data creation unit 50.

The position information acquisition unit 46 acquires the position information acquired by the self-position estimation sensor 26 of the mobile object 10 at a predetermined cycle via the communication network 4 (see FIG. 1). The position information acquired by the display information creation device 40 in the position information acquisition unit 46 is acquired, for example, as relative coordinates with respect to a predetermined reference point in an area coordinate system having a predetermined reference point set in the predetermined area 2. Note that it is preferable that the cycle and timing at which the position information acquisition unit 46 acquires the position information be the same as the cycle and timing at which the point cloud data acquisition unit 42 acquires point cloud data, and the cycle and timing at which the color data acquisition unit 44 acquires color data. The position information acquisition unit 46 outputs the acquired position information to the map data creation unit 52.

The video data acquisition unit 48 acquires the video data acquired by the imaging camera 28 of the mobile object 10 via the communication network 4 (see FIG. 1). The video data acquired by the display information creation device 40 in the video data acquisition unit 48 includes video data captured from the viewpoint of the position of the imaging camera 28 and of an arbitrary range of the surrounding environment of the mobile object 10 in the specified area 2. The video data acquisition unit 48 outputs the acquired video data to the display information creation unit 56.

The colored point cloud data creation unit 50 creates colored point cloud data by combining the point cloud data acquired by the point cloud data acquisition unit 42 and the color data acquired by the color data acquisition unit 44. The point cloud data and color data are combined by adding color information corresponding to each point contained in the point cloud data to the point cloud data. In the first embodiment, when the colored point cloud data creation unit 50 acquires an instruction to create three-dimensional map data from the mapping control unit 70, it starts creating colored point cloud data. The colored point cloud data creation unit 50 outputs the created colored point cloud data to the map data creation unit 52 and the display information creation unit 56.

The map data creation unit 52 creates three-dimensional map data based on the colored point cloud data created by the colored point cloud data creation unit 50 and the position information acquired by the position information acquisition unit 46. The point cloud data acquired by the point cloud data acquisition unit 42 is shown as distance information from the measurement point of the moving body 10 to each point. In other words, the colored point cloud data created by the colored point cloud data creation unit 50 is shown in the inertial coordinate system of the moving body 10. The three-dimensional map data is created by converting the colored point cloud data into the area coordinate system of the specified area 2. The map data creation unit 52 outputs the created three-dimensional map data to the map data storage unit 54 for storage. The map data creation unit 52 synthesizes the newly created three-dimensional map data with the three-dimensional map data already stored in the map data storage unit 54 and stores it.

The map data storage unit 54 temporarily stores the three-dimensional map data created by the map data creation unit 52. "Temporarily" refers to a time range such as until the display system 1 is terminated or until the operator performs a specified memory deletion operation. The map data storage unit 54 also stores the three-dimensional map data created by the map data creation unit 52 in addition to the three-dimensional map data already stored. In other words, the three-dimensional map data stored in the map data storage unit 54 is a collection of colored point cloud data that is acquired and created at a predetermined cycle and converted into an area coordinate system as needed. The map data storage unit 54 outputs the updated three-dimensional map data to the display information creation unit 56.

The display information creation unit 56 creates display information including three-dimensional map data based on the display settings stored in the screen setting data storage unit 62 of the screen setting unit 60. The display information may include an image of the colored point cloud data or a video of the video data. The display information includes, for example, information on the screen configuration such as the number of screens to be displayed and the display size of each screen in a physical display device (the display screen 36 of the display unit 34 in the first embodiment). The display information includes, for example, the type of data to be displayed on each screen. The type of data includes three-dimensional map data, position information and direction information of a specific viewpoint for displaying the three-dimensional map data, an image of the colored point cloud data, and a video of the video data. If the moving body 10 is equipped with multiple imaging cameras 28, the video data includes multiple video data. The three-dimensional map data is data that can be displayed as a map specified by a specific viewpoint based on the setting of the viewpoint set in the screen setting data. If the screen setting data has multiple viewpoint settings, the three-dimensional map data can be displayed as multiple maps specified by each viewpoint. The display information creation unit 56 outputs the created display information to the display information output unit 58.

The display information output unit 58 outputs the display information created by the display information creation unit 56 to the display unit 34, and displays the display information including the three-dimensional map data on the display screen 36 in a display format based on the display settings based on the screen setting data stored in the screen setting data storage unit 62 of the screen setting unit 60. At this time, the three-dimensional map data is displayed as a map specified by the position information and direction information of the specified viewpoint set in the screen setting data. That is, the three-dimensional map data is displayed on the display screen 36 as a map projected two-dimensionally from the specified viewpoint. Note that the specified viewpoint for displaying the three-dimensional map data may be appropriately changed by the operation of the operator during display on the display screen 36. When the display information includes colored point cloud data, the display information output unit 58 displays an image of the colored point cloud data on the display screen 36 in a display format based on the display settings. When the display information includes video data, the display information output unit 58 displays the video of the video data on the display screen 36 in a display format based on the display settings.

The screen setting unit 60 manages screen setting data related to the display format of the display screen 36 of the display unit 34. The screen setting unit 60 can set display settings including a combination of setting values of the screen setting data. The screen setting data includes, for example, display formats such as screen configuration settings such as the number of screens to be displayed and the display size of each screen in the physical display device (the display screen 36 of the display unit 34 in the first embodiment), settings for the type of data to be displayed on each screen, and settings for the viewpoint when displaying three-dimensional map data on the display screen 36.

In the screen setting unit 60, for example, a viewpoint can be set when the three-dimensional map data is displayed as a map on the display screen 36 of the display unit 34. The viewpoint setting items include position information that is the base point of the viewpoint, and directional information of the direction in which the viewpoint is used as the base point. The viewpoint setting items can be set, for example, the X coordinate, the Y coordinate, and the Z coordinate, and directional information about each of the three axes. In the screen setting unit 60, for example, a screen configuration such as the number of screens of the display screen 36 of the display unit 34 and the display size of each screen can be set. In the screen setting unit 60, for example, a display setting can be set to display maps specified by different viewpoints of the three-dimensional map data on the multiple display screens 36 of the display unit 34. In the screen setting unit 60, for example, a display setting can be set to display a map specified by a specific viewpoint of the three-dimensional map data, an image of the colored point cloud data, and an image of the video data on the multiple display screens 36 of the display unit 34. The screen setting unit 60 includes a screen setting data storage unit 62, a screen setting data storage control unit 64, and a screen setting data editing instruction acquisition unit 66.

The screen setting data storage unit 62 stores screen setting data. The screen setting data storage unit 62 has a display setting list and a most recent setting list.

The display setting list is data in which display settings, including combinations of setting values for various screen setting data, and the names of the display settings are listed and recorded. The names of the display settings include at least one of a list number consisting of numbers and a list name consisting of character strings. The list number and list name may be unique numbers and character strings automatically assigned by the screen setting unit 60 when the display setting list is saved, or may be numbers and character strings input by the operator from the operation unit 32 or the like. Each display setting in the display setting list is added, edited, or deleted by the operator's operation. Each display setting in the display setting list remains recorded in the screen setting data storage unit 62 until it is deleted by the operator's operation.

The most recent settings list is data in which past display settings are temporarily recorded. Each display setting in the most recent settings list is added, edited, or deleted by the operator's operation. When the most recent settings list reaches a specified capacity and a new past display setting is added, the oldest display setting is automatically deleted from the screen setting data storage unit 62. The specified capacity is, for example, the maximum capacity of the most recent settings list that is set in advance.

The screen setting data storage unit 62 outputs the display setting selected by the operator's operation from among the display settings of the display setting list and the most recent setting list to the display information creation unit 56. The operator can specify the display setting of the display setting list and the most recent setting list by the name of the display setting. This makes it possible to display the display information including the three-dimensional map data on the display unit 34 with the display setting for displaying it as a map specified by the desired viewpoint, without having to input the setting values of various screen setting data from scratch.

When the screen setting data editing instruction acquisition unit 66 acquires an operation signal from the operation unit 32 by the operator, the screen setting data storage control unit 64 edits the display setting list or the most recent setting list in the screen setting data storage unit 62 in response to the acquired operation signal.

The screen setting data editing instruction acquisition unit 66 acquires an operation signal including an instruction to edit the screen setting data stored in the screen setting data storage unit 62 from the operator's operation unit 32. The screen setting data editing instruction acquisition unit 66 outputs an instruction to edit the screen setting data according to the acquired operation signal to the screen setting data storage control unit 64.

The mapping control unit 70 outputs a control signal to the colored point cloud data creation unit 50 to instruct the start and end of creation of three-dimensional map data. The mapping control unit 70 may, for example, instruct the start and end of creation of three-dimensional map data based on an operator's operation on the operation unit 32. The mapping control unit 70 may, for example, instruct the start of creation of three-dimensional map data when the mobile body 10 enters a predetermined area 2 based on the acquired position information of the mobile body 10.

The above-mentioned components of the display information creation device 40 may share a common arithmetic processing device, storage device, or communication interface, or may be located separately in multiple locations. For example, the display information creation device 40 may be configured by dividing it into a map creation device, a screen setting device, and a display control device. The map creation device includes a point cloud data acquisition unit 42, a color data acquisition unit 44, a position information acquisition unit 46, an image data acquisition unit 48, a colored point cloud data creation unit 50, a map data creation unit 52, a map data storage unit 54, and a mapping control unit 70. The screen setting device includes a screen setting unit 60. The display control device includes a display information creation unit 56 and a display information output unit 58.

Next, the acquisition range of colored point cloud data by the mobile body 10 moving in the specified area 2 will be described. FIG. 3 is an explanatory diagram showing the acquisition range of point cloud data by the mobile body 10. In the example shown in FIG. 3, the mobile body 10 moves backwards in the specified area 2 to enter between features X1 and X2 that are located to the right (lower in FIG. 3) of the direction of travel (right in FIG. 3). Features X1 and X2 are structures that are taller than the mobile body 10 so as to block the measurement range of the measurement device 20 of the mobile body 10. The laser surveying sensor 22 and the color data acquisition sensor 24 of the mobile body 10 are provided on the front, rear, left and right sides of the mobile body 10, respectively. The range in which color data can be acquired by the color data acquisition sensor 24 includes the sensor effective range S of the laser surveying sensor 22. In the present invention, the sensor effective range S of the laser surveying sensor 22 is not limited to the angle and effective distance shown in FIG. 3.

In the diagrams shown in steps ST1 to ST5 of FIG. 3, each point of the point cloud data acquired within the sensor effective range S at the current position of the mobile body 10 is indicated by a black circle. Also, each point of the point cloud data acquired within the sensor effective range S of the mobile body 10 in the past is indicated by a white circle.

In step ST1, the moving body 10 is in a position where feature X1 is located to the right of the moving body 10's direction of travel. In step ST1, point cloud data is acquired for a portion of the side of feature X1 facing the moving body 10, and for a corner of feature X2 closer to the moving body 10.

In step ST2, the mobile body 10 moves to a position further forward than the position in step ST1. In step ST2, point cloud data is acquired for a portion of the side of feature X1 and feature X2 that is located forward of the position where the point cloud data was acquired in step ST1.

In steps ST3 and ST4, the mobile body 10 moves to a position to the right of the traveling direction where there is an opening between feature X1 and feature X2. In steps ST3 and ST4, point cloud data is acquired for the side of feature X1 facing feature X2 and the side of feature X2 facing feature X1.

In step ST5, the mobile body 10 passes to the left of feature X2 and stops after turning slightly to the left. After this, the mobile body 10 moves backward and enters between feature X1 and feature X2. At the position of step ST5, a blind spot BS from the mobile body 10 exists between feature X1 and feature X2. However, since point cloud data between feature X1 and feature X2 has been acquired in steps ST3 and ST4, the display information creation device 40 can create three-dimensional map data that also includes the blind spot BS portion. Therefore, the display information creation device 40 can output three-dimensional map data that can be displayed as a map that also includes the blind spot BS portion as display information.

Next, the setting of the viewpoint when the created three-dimensional map data is displayed on the display screen 36 of the display unit 34 as a map specified by a specific viewpoint will be described. FIG. 4 is a schematic diagram showing an example of the position and direction of the viewpoint when displaying three-dimensional map data. In the example shown in FIG. 4, it is assumed that features Y1 and Y2 are placed ahead of the moving body 10 in the direction of travel. Furthermore, the laser surveying sensor 22 and color data acquisition sensor 24 of the moving body 10 are provided ahead of the moving body 10. Furthermore, it is assumed that the range in which color data can be acquired by the color data acquisition sensor 24 includes the sensor effective range S of the laser surveying sensor 22.

As shown in FIG. 4, the laser surveying sensor 22 of the mobile body 10 can acquire point cloud data for each point constituting the surface of feature Y1 within the sensor effective range S, but cannot acquire point cloud data for each point constituting the surface of feature Y2 outside the sensor effective range S. However, if feature Y2 was previously within the sensor effective range S and point cloud data for each point constituting the surface of feature Y2 was acquired, the display information creation device 40 stores three-dimensional map data including feature Y2.

Here, the measurement position and measurement direction of the laser survey sensor 22 and color data acquisition sensor 24 of the moving body 10 are shown by the position and direction of the real sensor RP in FIG. 4. The real sensor RP moves in conjunction with the movement of the moving body 10, but the relative position and measurement direction with respect to the moving body 10 do not change.

The display information creation device 40 can create three-dimensional map data that can be displayed as a map with viewpoints of virtual sensors VP1 and VP2 other than the real sensor RP from the past to the present. Virtual sensor VP1 indicates an overhead viewpoint for features Y1 and Y2. Virtual sensor VP2 indicates an upward viewpoint for features Y1 and Y2. The viewpoint when displaying three-dimensional map data as a map specified by a specific viewpoint may be a viewpoint looking backward from the front in the direction of travel, etc.

In addition, a known method can be used to display three-dimensional map data as a map specified by a specific viewpoint, in which the three-dimensional data is projected as two-dimensional image data for an arbitrary viewpoint and displayed on a screen.

The display information creation device 40 can display three-dimensional map data created based on point cloud data and color data acquired from the sensor effective range S from the past to the present as a map specific to each viewpoint, thereby presenting the operator with information about the surrounding environment suitable for the operation of the mobile body 10.

Next, an example of the display screen 36 displayed based on the display settings will be described. FIG. 5 is a schematic diagram showing an example of the display screen 36. In the example shown in FIG. 5, the display screen 36 is divided into a first display screen 361, a second display screen 362, a third display screen 363, and a fourth display screen 364.

In the example shown in FIG. 5, the first display screen 361 is provided in the upper left area of the display screen 36. The first display screen 361 has a larger screen size than the second display screen 362, the third display screen 363, and the fourth display screen 364. Based on the display settings, the first display screen 361 displays, for example, a map identified by a specific viewpoint of three-dimensional map data.

In the example shown in FIG. 5, the second display screen 362 is provided in the upper right region of the display screen 36. The second display screen 362 has a smaller screen size than the first display screen 361. Based on the display settings, the second display screen 362 displays, for example, a map that is specified by a viewpoint different from the viewpoint of the map displayed on the first display screen 361.

In the example shown in FIG. 5, the third display screen 363 is provided in the lower right area of the display screen 36. The third display screen 363 has a smaller screen size than the first display screen 361. Based on the display settings, the third display screen 363 displays, for example, an image of colored point cloud data with the actual position of the laser survey sensor 22 as the viewpoint.

In the example shown in FIG. 5, the fourth display screen 364 is provided in the lower left area of the display screen 36. The fourth display screen 364 displays, for example, information related to other mobile objects 10.

The number of screens on the display screen 36, the screen configuration such as the display size of each screen, the type of map or image to be displayed on each screen, the viewpoint position of the map, etc. are not limited to the example shown in FIG. 5, and can be set in various ways as display settings including combinations of setting values of the screen setting data in the screen setting unit 60.

Next, a display method including a display information creation process by the display information creation device 40 of the first embodiment will be described. FIG. 6 is a flowchart showing an example of the process of the display information creation device 40 of the first embodiment. The process shown in FIG. 6 is executed by the display information creation device 40 based on a predetermined control program and data. The display information creation device 40 transitions to step ST200 shown in FIG. 6 and starts the process, for example, when the mapping control unit 70 outputs a control signal to the colored point cloud data creation unit 50 based on an operation of the operator on the operation unit 32, instructing the start of creation of three-dimensional map data.

The display information creation device 40 initializes the three-dimensional map data (step ST200). More specifically, the display information creation device 40 deletes the three-dimensional map data stored in the map data storage unit 54, and initializes the data so that new three-dimensional map data can be created. The display information creation device 40 proceeds to step ST202.

The display information creation device 40 refers to the display settings (step ST202). Specifically, the display information creation unit 56 refers to the screen setting data stored in the screen setting data storage unit 62, acquires the display settings selected from the display settings in the display setting list and the most recent setting list, and temporarily stores them. The display information creation device 40 proceeds to step ST206.

The display information creation device 40 acquires point cloud data (step ST206). Specifically, the point cloud data acquisition unit 42 acquires the point cloud data acquired by the laser survey sensor 22 of the mobile body 10 via the communication network 4. At this time, the point cloud data acquisition unit 42 acquires point cloud data in the specified area 2 that corresponds to the sensor effective range S of the mobile body 10 at the position at the time of measurement. The point cloud data acquisition unit 42 outputs the acquired point cloud data to the colored point cloud data creation unit 50 and the map data creation unit 52. The display information creation device 40 proceeds to step ST208.

The display information creation device 40 acquires color data (step ST208). Specifically, the color data acquisition unit 44 acquires the color data acquired by the color data acquisition sensor 24 of the moving body 10 via the communication network 4. At this time, the color data acquisition unit 44 acquires color data in the specified area 2 that corresponds to the sensor effective range S of the moving body 10 at the position where the point cloud data was acquired in step ST206. The color data acquisition unit 44 outputs the acquired color data to the colored point cloud data creation unit 50. The display information creation device 40 proceeds to step ST210.

The display information creation device 40 acquires position information (step ST210). Specifically, the position information acquisition unit 46 acquires the position information acquired by the self-position estimation sensor 26 of the mobile object 10 via the communication network 4. At this time, the position information acquisition unit 46 acquires the position information of the mobile object 10 at the position where the point cloud data was acquired in step ST206 in the specified area 2. The position information is acquired, for example, as relative coordinates with respect to a specified reference point in a region coordinate system having a specified reference point set in the specified area 2. The position information acquisition unit 46 outputs the acquired position information to the map data creation unit 52. The display information creation device 40 proceeds to step ST226.

The display information creation device 40 creates colored point cloud data (step ST226). Specifically, the colored point cloud data creation unit 50 creates colored point cloud data by combining the point cloud data acquired by the point cloud data acquisition unit 42 in step ST206 and the color data acquired by the color data acquisition unit 44 in step ST208. The point cloud data and color data are combined by adding color information corresponding to each point included in the point cloud data to the point cloud data. The colored point cloud data creation unit 50 outputs the created colored point cloud data to the map data creation unit 52 and the display information creation unit 56. The display information creation device 40 proceeds to step ST228.

The display information creation device 40 adds the colored point cloud data to the three-dimensional map data. Specifically, the map data creation unit 52 converts the colored point cloud data created by the colored point cloud data creation unit 50 in step ST226 into the area coordinate system of the specified area 2 based on the position information acquired by the position information acquisition unit 46 in step ST210. The map data creation unit 52 outputs the colored point cloud data converted into the area coordinate system of the specified area 2 to the map data storage unit 54 as three-dimensional map data. The map data storage unit 54 stores the three-dimensional map data. If the map data storage unit 54 has already created three-dimensional map data, the map data storage unit 54 adds the newly created three-dimensional map data to the already stored three-dimensional map data and stores it. The map data storage unit 54 outputs the updated three-dimensional map data to the display information creation unit 56. The display information creation device 40 proceeds to step ST246.

The display information creating device 40 determines whether or not there is an instruction to change the display settings (step ST246). When an operator operates to select a new display setting from among the display settings in the display setting list and the most recent setting list, the screen setting data storage unit 62 outputs the new display setting to the display information creating unit 56. In step ST246, the display information creating unit 56 determines whether or not a new display setting has been output from the screen setting data storage unit 62 since the previous acquisition of the display setting. The previously acquired display setting refers to the display setting referenced in step ST202 or the display setting updated in step ST248. If there is an instruction to change the display setting (step ST246; Yes), the display information creating device 40 proceeds to step ST248. If there is no instruction to change the display setting (step ST246; No), the display information creating device 40 proceeds to step ST250.

If there is an instruction to change the display settings (step ST246; Yes), the display information creation device 40 updates the display settings (step ST248). Specifically, the display information creation unit 56 overwrites the previously acquired display settings with the newly acquired display settings and temporarily stores them. The display information creation device 40 proceeds to step ST250.

The display information creation device 40 outputs the display information (step ST250). Specifically, the display information creation unit 56 creates display information including three-dimensional map data based on the temporarily stored display settings. The three-dimensional map data is data that can be displayed as a map specified by a specific viewpoint based on the viewpoint setting set in the display settings. When the display settings include a setting to display colored point cloud data on a specific display screen 36, the display information creation unit 56 creates display information that displays the three-dimensional map data, the position information and direction information of the specific viewpoint for displaying the three-dimensional map data, and an image of the colored point cloud data on each display screen 36. The display information creation unit 56 outputs the created display information to the display information output unit 58. The display information output unit 58 displays the acquired display information on the display screen 36 of the display unit 34 in a display format based on the display settings. The display information creation device 40 proceeds to step ST252.

The display information creating device 40 determines whether the display system 1 has been terminated (step ST252). Specifically, the display information creating device 40 determines whether an operation signal for terminating the display system 1 has been acquired, for example, by an operator operating the operation unit 32. If the display system 1 has been terminated (step ST252; Yes), the display information creating device 40 terminates the processing of the flowchart shown in FIG. 6. If the display system 1 has not been terminated (step ST252; No), the display information creating device 40 returns to step ST206 and repeatedly executes the processing from step ST206 to step ST250.

The display information creation device 40 acquires the three-dimensional map data created in the processes from step ST206 to step ST228 in succession over time, and synthesizes each piece of three-dimensional map data. At this time, each piece of synthesized three-dimensional map data is unified into a common coordinate system. The display information creation device 40 creates display information including the synthesized three-dimensional map data based on the display settings set by the screen setting unit 60. The display information creation device 40 displays the created display information on the display screen 36 of the display unit 34 in a display format based on the display settings.

Second Embodiment
Next, the internal configuration of the display information creating device 40A according to the second embodiment will be described. Fig. 7 is a block diagram showing the internal configuration of the display information creating device 40A according to the second embodiment together with the peripheral configuration. In the display information creating device 40A according to the second embodiment, the same components as those in the display information creating device 40 according to the first embodiment will be denoted by the same reference numerals and will not be described as appropriate, and only different components will be described.

The display information creation device 40A of the second embodiment differs from the display information creation device 40 of the first embodiment in that it includes a point cloud data acquisition unit 42A, a position information acquisition unit 46A, and a mapping control unit 70A instead of the point cloud data acquisition unit 42, the position information acquisition unit 46, and the mapping control unit 70.

The point cloud data acquisition unit 42A differs from the point cloud data acquisition unit 42 of the first embodiment in that it outputs the acquired point cloud data not only to the colored point cloud data creation unit 50 and the map data creation unit 52, but also to the area interior determination unit 74 of the mapping control unit 70A. The other configurations and functions of the point cloud data acquisition unit 42A are similar to those of the point cloud data acquisition unit 42 of the first embodiment.

The location information acquisition unit 46A differs from the location information acquisition unit 46 of the first embodiment in that it outputs the acquired location information not only to the map data creation unit 52 but also to the area interior determination unit 74 of the mapping control unit 70A. The other configurations and functions of the location information acquisition unit 46A are similar to those of the location information acquisition unit 46 of the first embodiment.

The mapping control unit 70A outputs a control signal to the colored point cloud data creation unit 50 to instruct the start and end of the creation of three-dimensional map data based on an operation signal including an instruction from the operator to start and end the creation of three-dimensional map data obtained from the operation unit 32. The mapping control unit 70A includes an area data storage unit 72, an area interior determination unit 74, a creation start/end instruction unit 76, an area data storage control unit 78, and an area data editing instruction acquisition unit 80.

The area data storage unit 72 stores area data. The area data includes position information of the map creation area (see, for example, the specified area 2 in FIG. 3) for which display information including three-dimensional map data is to be created. The position information stored in the area data storage unit 72 is indicated in a global coordinate system including, for example, longitude information and latitude information. In the second embodiment, the area data includes map creation area setting data. The map creation area setting data includes position information of the map creation area that is set in advance before the start of the creation of the three-dimensional map data. The map creation area setting data can be added, edited, or deleted by the operator's operation. The area data storage unit 72 outputs the area data to the area interior determination unit 74.

The area interior determination unit 74 converts the point cloud data acquired by the point cloud data acquisition unit 42A into a global coordinate system based on the position information acquired by the position information acquisition unit 46A. The area interior determination unit 74 references the map creation area setting data stored in the area data storage unit 72. The area interior determination unit 74 determines whether the position of each point of the point cloud data converted into the global coordinate system is included in the map creation area set in the map creation area setting data. The area interior determination unit 74 outputs the determination result as to whether it is within the map creation area to the creation start/end instruction unit 76.

The creation start/end instruction unit 76 obtains the determination result of whether or not it is within the map creation area from the area inside determination unit 74. The creation start/end instruction unit 76 outputs a creation start instruction and a creation end instruction for the three-dimensional map data to the colored point cloud data creation unit 50. If the area inside determination unit 74 determines that it is within the map creation area, the creation start/end instruction unit 76 outputs an instruction to start creating the three-dimensional map data to the colored point cloud data creation unit 50. Even if the area inside determination unit 74 determines that it is within the map creation area, if the three-dimensional map data is already being created, the creation start/end instruction unit 76 may omit the output of the creation start instruction for the three-dimensional map data. If the area inside determination unit 74 determines that it is not within the map creation area, the creation start/end instruction unit 76 outputs an instruction to end the creation of the three-dimensional map data to the colored point cloud data creation unit 50. Even if the area interior determination unit 74 determines that the location is not within the map creation area, the creation start/end instruction unit 76 may omit outputting an instruction to end creation of the 3D map data if the 3D map data is not being created.

When the area data editing instruction acquisition unit 80 acquires an operation signal from the operation unit 32 by the operator, the area data storage control unit 78 edits the map creation area setting data in the area data storage unit 72 in response to the acquired operation signal.

The area data editing instruction acquisition unit 80 acquires an operation signal including an instruction to edit the map creation area setting data stored in the area data storage unit 72 from the operation unit 32 operated by the operator. The area data editing instruction acquisition unit 80 outputs an instruction to edit the map creation area setting data according to the acquired operation signal to the area data storage control unit 78.

Next, a display method including a display information creation process by the display information creation device 40A of the second embodiment will be described. FIG. 8 is a flowchart showing an example of the process of the display information creation device 40A of the second embodiment. The process shown in FIG. 8 is executed by the display information creation device 40A based on a predetermined control program and data. For example, when the display system 1 is started, the display information creation device 40A proceeds to step ST300 shown in FIG. 8 and starts the process.

The processes of steps ST300, ST302, ST306, ST308, and ST310 are similar to those of steps ST200, ST202, ST206, ST208, and ST210 in the first embodiment, and therefore will not be described. However, in step ST306 in the second embodiment, the point cloud data acquisition unit 42A outputs the acquired point cloud data not only to the colored point cloud data creation unit 50 and the map data creation unit 52, but also to the area interior determination unit 74. In addition, in step ST310 in the second embodiment, the position information acquisition unit 46A outputs the acquired position information not only to the map data creation unit 52, but also to the area interior determination unit 74. When the display information creation device 40A finishes the process of step ST310, it proceeds to step ST312.

The display information creation device 40A determines whether or not it is within the map creation area (step ST312). Specifically, first, the area interior determination unit 74 converts the point cloud data acquired in step ST306 into a global coordinate system based on the position information acquired in step ST310. Next, the area interior determination unit 74 refers to the map creation area setting data stored in the area data storage unit 72. The area interior determination unit 74 determines whether or not the position of each point of the point cloud data converted into the global coordinate system is included in the map creation area set in the map creation area setting data. The area interior determination unit 74 outputs the determination result to the creation start/end instruction unit 76.

If it is within the map creation area (step ST312; Yes), the creation start/end instruction unit 76 outputs an instruction to start creating three-dimensional map data to the colored point cloud data creation unit 50. The display information creation device 40A proceeds to step ST326. If it is not within the map creation area (step ST312; No), the creation start/end instruction unit 76 outputs an instruction to end creating three-dimensional map data to the colored point cloud data creation unit 50. The display information creation device 40A proceeds to step ST330.

The processing of steps ST326 and ST328 is similar to the processing of steps ST226 and ST228 in the first embodiment, and therefore the description is omitted. After completing the processing of step ST328, the display information creation device 40A proceeds to step ST330.

The display information creation device 40A determines whether or not there is three-dimensional map data (step ST330). Specifically, the display information creation device 40A determines whether or not there is already created three-dimensional map data in the map data storage unit 54. If there is three-dimensional map data (step ST330; Yes), the display information creation device 40A proceeds to step ST346. If there is no three-dimensional map data (step ST330; No), the display information creation device 40A proceeds to step ST352.

The processing of steps ST346, ST348, ST350, and ST352 is similar to the processing of steps ST246, ST248, ST250, and ST252 in the first embodiment, and therefore will not be described.

The display information creation device 40A continuously acquires the three-dimensional map data created in the processes from step ST306 to step ST328 while the measurement range of the moving body 10 is included within the map creation area, and synthesizes each piece of three-dimensional map data. Each piece of three-dimensional map data synthesized at this time is unified into a common coordinate system. The display information creation device 40A creates display information including the synthesized three-dimensional map data based on the display settings set by the screen setting unit 60. The display information creation device 40A displays the created display information on the display screen 36 of the display unit 34 in a display format based on the display settings.

Third Embodiment
Next, the internal configuration of the display information creating device 40B according to the third embodiment will be described. Fig. 9 is a block diagram showing the internal configuration of the display information creating device 40B according to the third embodiment together with the peripheral configuration. In the display information creating device 40B according to the third embodiment, the same components as those in the display information creating device 40A according to the second embodiment will be denoted by the same reference numerals and will not be described as appropriate, and only different components will be described.

The display information creation device 40B of the third embodiment differs from the display information creation device 40A of the second embodiment in that it includes a mapping control unit 70B instead of the mapping control unit 70A. The mapping control unit 70B differs from the mapping control unit 70A of the second embodiment in that it includes an area data storage unit 72B, a creation start/end instruction unit 76B, and an area data storage control unit 78B instead of the area data storage unit 72, the creation start/end instruction unit 76, and the area data storage control unit 78, and further includes a creation start/end instruction acquisition unit 82.

The area data storage unit 72B differs from the area data storage unit 72 of the second embodiment in that the area data includes map creation history data and a counter in addition to map creation area setting data. The map creation history data includes position information of a map creation area for which an operator has previously instructed the creation of three-dimensional map data. The map creation history data can be deleted by the operator. The counter counts the number of times that an instruction to create three-dimensional map data has been issued for the same area. If the number of times that an instruction to create three-dimensional map data for the same area has been issued is equal to or greater than a predetermined number, that is, if there is a predetermined amount of overlap in the map creation areas of the accumulated map creation history data, the area data storage unit 72 changes the map creation history data to map creation area setting data. Other configurations and functions of the area data storage unit 72B are the same as those of the area data storage unit 72 of the second embodiment.

Compared to the creation start/end instruction unit 76 of the second embodiment, the creation start/end instruction unit 76B further acquires a creation start instruction and a creation end instruction of three-dimensional map data by an operator's operation from the creation start/end instruction acquisition unit 82. When the creation start/end instruction unit 76B acquires a creation start instruction of three-dimensional map data by an operator's operation from the creation start/end instruction acquisition unit 82, it outputs a creation start instruction of three-dimensional map data to the colored point cloud data creation unit 50 even if it is not within the map creation area. When the creation start/end instruction unit 76B acquires a creation end instruction of three-dimensional map data by an operator's operation from the creation start/end instruction acquisition unit 82, it outputs a creation end instruction of three-dimensional map data to the colored point cloud data creation unit 50 even if it is within the map creation area. The other configurations and functions of the creation start/end instruction unit 76B are the same as those of the creation start/end instruction unit 76 of the second embodiment.

The area data storage control unit 78B differs from the area data storage control unit 78 of the second embodiment in that it further acquires an instruction to update the map creation history data from the creation start/end instruction acquisition unit 82. The area data storage control unit 78B updates the map creation history data in the area data storage unit 72B in response to the update instruction for the map creation history data acquired from the creation start/end instruction acquisition unit 82. The other configurations and functions of the area data storage control unit 78B are similar to those of the area data storage control unit 78 of the second embodiment.

The creation start/end instruction acquisition unit 82 acquires an operation signal including an instruction to start and end the creation of three-dimensional map data from the operation unit 32 by the operator. The creation start/end instruction acquisition unit 82 outputs an instruction to start and end the creation of three-dimensional map data according to the acquired operation signal to the creation start/end instruction unit 76B. The creation start/end instruction acquisition unit 82 outputs an instruction to update the map creation history data to the area data storage control unit 78B based on the acquired instruction to start and end the creation of three-dimensional map data.

Next, a display method including a display information creation process by the display information creation device 40B of the third embodiment will be described. FIG. 10 is a flowchart showing an example of the process of the display information creation device 40B of the third embodiment. The process shown in FIG. 10 is executed by the display information creation device 40B based on a predetermined control program and data. For example, when the display system 1 is started, the display information creation device 40B proceeds to step ST400 shown in FIG. 10 and starts the process.

The processing of steps ST400, ST402, ST406, ST408, ST410, and ST412 is similar to the processing of steps ST300, ST302, ST306, ST308, ST310, and ST312 in the second embodiment, and therefore description thereof will be omitted. If it is determined in step ST412 that the display information creation device 40B is within the map creation area (step ST412; Yes), the display information creation device 40B proceeds to step ST426. If it is determined that the display information creation device 40B is not within the map creation area (step ST412; No), the display information creation device 40B proceeds to step ST414.

If the location is not within the map creation area (step ST412; No), the display information creation device 40B determines whether or not there is an instruction to start map creation (step ST414). Specifically, the display information creation device 40B determines whether or not the creation start/end instruction unit 76B has acquired an instruction to start creating 3D map data by the operator from the creation start/end instruction acquisition unit 82. If there is an instruction to start map creation (step ST414; Yes), the display information creation device 40B proceeds to step ST416. If there is no instruction to start map creation (step ST414; No), the display information creation device 40B proceeds to step ST430.

If there is an instruction to start map creation (step ST414; Yes), the display information creation device 40B updates the map creation history data (step ST416). Specifically, the area data storage control unit 78B acquires an instruction to update the map creation history data from the creation start/end instruction acquisition unit 82. The area data storage control unit 78B updates the map creation history data in the area data storage unit 72B in response to the instruction to update the map creation history data acquired from the creation start/end instruction acquisition unit 82. The display information creation device 40B proceeds to step ST418.

The display information creation device 40B determines whether the number of creation instructions for the same area is equal to or greater than a predetermined number (step ST418). Specifically, when map creation history data is updated in the area data storage unit 72B, if it overlaps with map creation history data already stored, the counter increments the counter value of the number of creation instructions for 3D map data for the same area by one and stores it. If the number of creation instructions for the same area is equal to or greater than the predetermined number (step ST418; Yes), the display information creation device 40B proceeds to step ST420. If the number of creation instructions for the same area is less than the predetermined number (step ST418; No), the display information creation device 40B proceeds to step ST426.

If the number of creation instructions for the same area is equal to or greater than the predetermined number (step ST418; Yes), the display information creation device 40B updates the map creation area setting data (step ST420). Specifically, if there is a predetermined amount of overlap in the map creation areas of the accumulated map creation history data, the area data storage unit 72B changes the map creation history data to map creation area setting data. The display information creation device 40B proceeds to step ST426.

The processes of steps ST426, ST428, ST430, ST446, ST448, ST450, and ST452 are similar to the processes of steps ST326, ST328, ST330, ST346, ST348, ST350, and ST352 in the second embodiment, and therefore will not be described. If it is determined in step ST452 that the display system 1 has been terminated (step ST452; Yes), the display information creation device 40B terminates the process of the flowchart shown in FIG. 10. If it is determined that the display system 1 has not been terminated (step ST452; No), the display information creation device 40B proceeds to step ST454.

The display information creation device 40B determines whether or not there is an instruction to end map creation (step ST454). Specifically, the display information creation device 40B determines whether or not the creation start/end instruction unit 76B has acquired an instruction to end the creation of the three-dimensional map data by the operator's operation from the creation start/end instruction acquisition unit 82. If there is an instruction to end map creation (step ST454; Yes), the display information creation device 40B ends the process of the flowchart shown in FIG. 10. If there is no instruction to end map creation (step ST454; No), the display information creation device 40B returns to step ST406 and repeatedly executes the processes from step ST406 to step ST452.

The display information creation device 40B continuously acquires the three-dimensional map data created in time during the processing from step ST406 to step ST428 while the measurement range of the moving body 10 is included within the map creation area, or during the period from the instruction to start creating the three-dimensional map data by an operation to the instruction to end creation, and synthesizes each piece of three-dimensional map data. At this time, each piece of synthesized three-dimensional map data is unified into a common coordinate system. The display information creation device 40B creates display information including the synthesized three-dimensional map data based on the display settings set by the screen setting unit 60. The display information creation device 40B displays the created display information on the display screen 36 of the display unit 34 in a display format based on the display settings.

Fourth Embodiment
Next, the internal configuration of the display information creating device 40C of the fourth embodiment will be described. Fig. 11 is a block diagram showing the internal configuration of the display information creating device 40C of the fourth embodiment together with the peripheral configuration. In the display information creating device 40C of the fourth embodiment, the same components as those of the display information creating device 40B of the third embodiment will be denoted by the same reference numerals and will not be described as appropriate, and only different components will be described.

The display information creation device 40C of the fourth embodiment differs from the display information creation device 40B of the third embodiment in that it includes a mapping control unit 70C instead of the mapping control unit 70B. The mapping control unit 70C differs from the mapping control unit 70B of the third embodiment in that it includes an area interior determination unit 74C, a creation start/end instruction unit 76C, and a creation start/end instruction acquisition unit 82C instead of the area interior determination unit 74, the creation start/end instruction unit 76B, and the creation start/end instruction acquisition unit 82, and further includes a notification information output unit 84.

The area inside determination unit 74C differs from the area inside determination unit 74 of the second and third embodiments in that the area inside determination unit 74 outputs the result of the determination as to whether or not it is within the map creation area to the notification information output unit 84, rather than to the creation start/end instruction unit 76C. The other configurations and functions of the area inside determination unit 74C are similar to those of the area inside determination unit 74 of the second and third embodiments.

The creation start/end instruction unit 76C differs from the creation start/end instruction unit 76B of the third embodiment in that it does not acquire a determination result of whether or not it is within the map creation area from the area interior determination unit 74C. The creation start/end instruction unit 76C outputs a creation start instruction and a creation end instruction for the three-dimensional map data to the colored point cloud data creation unit 50. When the creation start/end instruction unit 76C acquires an instruction to start creating three-dimensional map data by an operator's operation from the creation start/end instruction acquisition unit 82C, it outputs an instruction to start creating three-dimensional map data to the colored point cloud data creation unit 50. When the creation start/end instruction unit 76C acquires an instruction to end creating three-dimensional map data by an operator's operation from the creation start/end instruction acquisition unit 82C, it outputs an instruction to end creating three-dimensional map data to the colored point cloud data creation unit 50.

The notification information output unit 84 obtains the determination result of whether or not the location is within the map creation area from the area interior determination unit 74C. The notification information output unit 84 outputs notification information to the display unit 34 to notify the operator to select whether or not to start creating three-dimensional map data. The operator checks the notification information displayed on the display unit 34 and performs an operation via the operation unit 32 to select whether or not to start creating three-dimensional map data.

The creation start/end instruction acquisition unit 82C differs from the creation start/end instruction acquisition unit 82 of the third embodiment in that it further acquires the operator's selection result in response to the notification information output by the notification information output unit 84. When the operator performs an operation to select to start creating three-dimensional map data in response to the notification information, the creation start/end instruction acquisition unit 82C outputs an instruction to start creating three-dimensional map data to the creation start/end instruction unit 76C. The other configurations and functions of the creation start/end instruction acquisition unit 82C are similar to those of the creation start/end instruction acquisition unit 82 of the third embodiment.

Next, a display method including a display information creation process by the display information creation device 40C of the fourth embodiment will be described. FIG. 12 is a flowchart showing an example of the process of the display information creation device 40C of the fourth embodiment. The process shown in FIG. 12 is executed by the display information creation device 40C based on a predetermined control program and data. For example, when the display system 1 is started, the display information creation device 40C proceeds to step ST500 shown in FIG. 12 and starts the process.

The processing of steps ST500, ST502, ST506, ST508, ST510, and ST512 is similar to the processing of steps ST400, ST402, ST406, ST408, ST410, and ST412 in the third embodiment, and therefore description thereof will be omitted. If it is determined in step ST512 that the display information creation device 40C is within the map creation area (step ST512; Yes), the display information creation device 40C proceeds to step ST522. If it is determined that the display information creation device 40C is not within the map creation area (step ST512; No), the display information creation device 40C proceeds to step ST514.

If it is determined that the location is within the map creation area (step ST512; Yes), the display information creation device 40C notifies the operator to select whether or not to start map creation (step ST522). Specifically, the notification information output unit 84 outputs notification information to the display unit 34 to notify the operator to select whether or not to start creating three-dimensional map data. The operator checks the notification information displayed on the display unit 34 and performs an operation via the operation unit 32 to select whether or not to start creating three-dimensional map data. The display information creation device 40C proceeds to step ST524.

The display information creation device 40C determines whether or not the start of map creation has been selected (step ST524). Specifically, the creation start/end instruction acquisition unit 82C acquires the operator's selection result for the notification information output by the notification information output unit 84 via the operation unit 32. When the creation start/end instruction acquisition unit 82C acquires an operation signal that accepts the start of creation of three-dimensional map data, the display information creation device 40C determines that the start of map creation has been selected. When the map creation start is selected (step ST524; Yes), the display information creation device 40C proceeds to step ST526. When the creation start/end instruction acquisition unit 82C acquires an operation signal that rejects the start of creation of three-dimensional map data, or when a predetermined time has elapsed since the output of the notification information, the display information creation device 40C determines that the start of map creation has not been selected. When the map creation start is not selected (step ST524; No), the display information creation device 40C proceeds to step ST530.

The processing of steps ST514, ST516, ST518, ST520, ST526, ST528, ST530, ST546, ST548, ST550, ST552, and ST554 is similar to the processing of steps ST414, ST416, ST418, ST420, ST326, ST328, ST330, ST346, ST348, ST350, ST352, and ST354 in the third embodiment, and therefore will not be described.

The display information creation device 40C continuously acquires the three-dimensional map data created in time during the processing from step ST506 to step ST528 while the measurement range of the moving body 10 is included within the map creation area, or during the period from the instruction to start creating the three-dimensional map data by an operation to the instruction to end creation, and synthesizes each piece of three-dimensional map data. At this time, each piece of synthesized three-dimensional map data is unified into a common coordinate system. The display information creation device 40C creates display information including the synthesized three-dimensional map data based on the display settings set by the screen setting unit 60. The display information creation device 40C displays the created display information on the display screen 36 of the display unit 34 in a display format based on the display settings.

Fifth Embodiment
Next, the internal configuration of the display information creating device 40D of the fifth embodiment will be described. Fig. 13 is a block diagram showing the internal configuration of the display information creating device 40D of the fifth embodiment together with the peripheral configuration. In the display information creating device 40D of the fifth embodiment, the same components as those of the display information creating device 40 of the first embodiment are denoted by the same reference numerals and the description thereof will be omitted as appropriate, and only different components will be described.

The display information creation device 40D of the fifth embodiment differs from the display information creation device 40 of the first embodiment in that, instead of the position information acquisition unit 46, the map data storage unit 54, and the display information creation unit 56, it includes a position information acquisition unit 46D, a map data storage unit 54D, and a display information creation unit 56D, and further includes an interference warning unit 90.

The location information acquisition unit 46D differs from the location information acquisition unit 46 of the first embodiment in that it outputs the acquired location information not only to the map data creation unit 52 but also to the movement physical quantity calculation unit 94 of the interference warning unit 90. The other configurations and functions of the location information acquisition unit 46D are similar to those of the location information acquisition unit 46 of the first embodiment.

The map data storage unit 54D differs from the map data storage unit 54 of the first embodiment in that it outputs updated three-dimensional map data not only to the display information creation unit 56D but also to the interference determination unit 102 of the interference warning unit 90. The other configurations and functions of the map data storage unit 54D are similar to those of the map data storage unit 54 of the first embodiment.

The display information creation unit 56D differs from the display information creation unit 56 of the first embodiment in that it further acquires warning information from the warning information output unit 104 of the interference warning unit 90. The warning information acquired by the display information creation unit 56D includes information on the interference location determined by the interference warning unit 90. The interference location indicates an area where the mobile body 10 may interfere with the surrounding environment. In the fifth embodiment, interference indicates that the colored point cloud data constituting the three-dimensional map data stored in the map data storage unit 54D and the mobile body three-dimensional model of the mobile body 10 overlap or are close to each other by a predetermined distance or more. In the fifth embodiment, the display information creation unit 56D updates the colored point cloud data indicating the interference location to the three-dimensional map data of the display information. Other configurations and functions of the display information creation unit 56D are similar to those of the display information creation unit 56 of the first embodiment.

The interference warning unit 90 outputs predetermined warning information to warn of the possibility of interference when the moving body 10 is likely to interfere with the surrounding environment. The interference warning unit 90 includes a moving body data storage unit 92, a movement physical quantity calculation unit 94, an operation information acquisition unit 96, an operation physical quantity calculation unit 98, a possible existence area calculation unit 100, an interference determination unit 102, and a warning information output unit 104.

The mobile body data storage unit 92 stores mobile body data related to the mobile body 10 in advance. The mobile body data includes mobile body three-dimensional model data, which is data of a three-dimensional model of the mobile body 10. The mobile body data storage unit 92 outputs the mobile body three-dimensional model data to the possible existence area calculation unit 100.

The movement physical quantity calculation unit 94 calculates physical quantities related to the movement of the moving body 10. The physical quantities related to the movement of the moving body 10 include the relative coordinates and attitude angle of the reference point of the moving body 10 in the area coordinate system, and the time change amount of the relative coordinates and attitude angle of the reference point of the moving body 10 in the area coordinate system. The movement physical quantity calculation unit 94 first acquires the position information of the moving body 10 from the position information acquisition unit 46D. The position information of the moving body 10 acquired by the movement physical quantity calculation unit 94 includes the relative coordinates and attitude angle of the reference point of the moving body 10 in the area coordinate system. The movement physical quantity calculation unit 94 temporarily stores the acquired position information of the moving body 10. The movement physical quantity calculation unit 94 then calculates the relative coordinates and time change amount of the attitude angle of the reference point of the moving body 10 based on the position information of the moving body 10 that is continuous in time. The movement physical quantity calculation unit 94 outputs the physical quantities related to the movement of the moving body 10 to the existence possible area calculation unit 100.

When the operation unit 32 receives an operation by the operator related to the movement of the moving body 10, the operation information acquisition unit 96 acquires operation information for the moving body 10 from the operation unit 32. The operation information includes operation information related to movement such as forward movement, backward movement, turning, acceleration, deceleration, and stopping. The operation information acquisition unit 96 outputs the operation information for the moving body 10 to the operation physical quantity calculation unit 98.

The operation physical quantity calculation unit 98 calculates a physical quantity corresponding to an operation on the moving body 10. The operation physical quantity calculation unit 98 calculates a physical quantity corresponding to an operation on the moving body 10 based on operation information on the moving body 10 acquired from the operation information acquisition unit 96. The physical quantity corresponding to the operation on the moving body 10 includes the moving direction, moving speed, and amount of change in the moving direction and moving speed of the moving body 10. The operation physical quantity calculation unit 98 outputs the calculated physical quantity corresponding to the operation on the moving body 10 to the possible existence area calculation unit 100.

The existence possible area calculation unit 100 acquires moving body data including moving body three-dimensional model data, which is data of a three-dimensional model of the moving body 10, from the moving body data storage unit 92. The existence possible area calculation unit 100 acquires physical quantities related to the movement of the moving body 10 from the movement physical quantity calculation unit 94. The existence possible area calculation unit 100 acquires physical quantities corresponding to the calculated operation on the moving body 10 from the operation physical quantity calculation unit 98. The existence possible area calculation unit 100 calculates the existence possible area of the moving body 10 within a predetermined time from the present based on the acquired moving body three-dimensional model data, the physical quantities related to the movement of the moving body 10, and the physical quantities corresponding to the operation on the moving body 10. The existence possible area refers to an area in which the moving body 10 may exist within a predetermined time from the present. The existence possible area calculation unit 100 outputs relative coordinate information of the existence possible area in the area coordinate system to the interference determination unit 102.

The interference determination unit 102 determines whether or not there is a possibility that the moving object 10 will interfere with the surrounding environment. More specifically, the interference determination unit 102 first calculates the shortest distance between the possible presence area and each point of the colored point cloud data included in the three-dimensional map data based on the three-dimensional map data acquired from the map data storage unit 54D and the possible presence area calculation unit 100. The interference determination unit 102 then determines whether or not the calculated shortest distance is equal to or greater than a predetermined threshold. If the shortest distance is less than the predetermined threshold, the interference determination unit 102 outputs a determination result indicating that there is a possibility that the moving object 10 will interfere with the surrounding environment to the warning information output unit 104.

When the interference determination unit 102 determines that the moving body 10 may interfere with the surrounding environment, the warning information output unit 104 outputs predetermined warning information to warn of the possibility of interference to the display information creation unit 56D and the display unit 34. The warning information output by the warning information output unit 104 to the display information creation unit 56D includes information on the interference location. The interference location indicates each point of the colored point cloud data whose shortest distance to the possible existence area is less than a predetermined threshold. The warning information output by the warning information output unit 104 to the display unit 34 may include, for example, text information to be displayed on the display screen 36 of the display unit 34, sound information to be transmitted by a speaker, light information to be emitted by a light-emitting device, etc.

Next, a display method including a display information creation process by the display information creation device 40D of the fifth embodiment will be described. FIG. 14 is a flowchart showing an example of the process of the display information creation device 40D of the fifth embodiment. The process shown in FIG. 14 is executed by the display information creation device 40D based on a predetermined control program and data. For example, the mapping control unit 70 outputs a control signal to the colored point cloud data creation unit 50 based on an operator's operation on the operation unit 32, instructing the start of creation of three-dimensional map data, and the display information creation device 40D proceeds to step ST600 shown in FIG. 14 and starts the process.

The processing of steps ST600 and ST602 is similar to the processing of steps ST200 and ST202 in the first embodiment, and therefore description thereof is omitted. After completing the processing of step ST602, the display information creation device 40D proceeds to step ST604.

The display information creation device 40D refers to the three-dimensional moving body model (step ST604). Specifically, the possible existence area calculation unit 100 acquires moving body data including three-dimensional moving body model data, which is data of a three-dimensional model of the moving body 10, from the moving body data storage unit 92. The display information creation device 40D proceeds to step ST606.

The processing of steps ST606, ST608, ST610, ST626, and ST628 is similar to the processing of steps ST206, ST208, ST210, ST226, and ST228 in the first embodiment, and therefore description thereof will be omitted. However, in step ST610 in the fifth embodiment, the position information acquisition unit 46D outputs the acquired position information not only to the map data creation unit 52 but also to the movement physical quantity calculation unit 94. Also, in step ST628 in the fifth embodiment, the map data storage unit 54 outputs the updated three-dimensional map data not only to the display information creation unit 56 but also to the interference determination unit 102. When the display information creation device 40D ends the processing of step ST628, it proceeds to step ST632.

The display information creation device 40D calculates physical quantities related to the movement of the moving body 10 (step ST632). Specifically, first, the movement physical quantity calculation unit 94 acquires position information of the moving body 10 from the position information acquisition unit 46D. The position information of the moving body 10 acquired by the movement physical quantity calculation unit 94 includes information on the relative coordinates of the reference point of the moving body 10 in the area coordinate system and the attitude angle. The movement physical quantity calculation unit 94 then calculates the amount of change over time in the relative coordinates of the reference point of the moving body 10 in the area coordinate system and the attitude angle based on the position information of the moving body 10 that is continuous in time. The display information creation device 40D proceeds to step ST634.

The display information creation device 40D acquires operation information for the moving body 10 (step ST634). Specifically, when the operation unit 32 accepts an operation by the operator related to the movement of the moving body 10, the operation information acquisition unit 96 acquires operation information for the moving body 10 from the operation unit 32. The operation information acquisition unit 96 outputs the operation information for the moving body 10 to the operation physical quantity calculation unit 98. The display information creation device 40D proceeds to step ST636.

The display information creation device 40D calculates physical quantities corresponding to the operation on the moving body 10 (step ST636). Specifically, the operation physical quantity calculation unit 98 calculates physical quantities corresponding to the operation on the moving body 10, including the moving direction, moving speed, and amount of change in the moving direction and moving speed of the moving body 10, based on the operation information on the moving body 10 acquired from the operation information acquisition unit 96. The operation physical quantity calculation unit 98 outputs the calculated physical quantities corresponding to the operation on the moving body 10 to the possible existence area calculation unit 100. The display information creation device 40D proceeds to step ST638.

The display information creation device 40D calculates the possible existence area of the moving body 10 within a predetermined time from the present (step ST638). Specifically, the possible existence area calculation unit 100 calculates the possible existence area of the moving body 10 based on the moving body three-dimensional model data acquired in step ST604, the physical quantities related to the movement of the moving body 10 acquired and calculated in step ST632, and the physical quantities corresponding to the operation on the moving body 10 calculated in step ST634. The display information creation device 40D proceeds to step ST640.

The display information creation device 40D determines whether or not there is a possibility that the moving body 10 will interfere with the surrounding environment (step ST640). Specifically, the interference determination unit 102 first calculates the shortest distance between the possible existence area and each point of the colored point cloud data included in the three-dimensional map data based on the three-dimensional map data acquired from the map data storage unit 54D and the possible existence area calculation unit 100. The interference determination unit 102 then determines whether or not the calculated shortest distance is equal to or greater than a predetermined threshold value.

If the shortest distance is less than a predetermined threshold, the interference determination unit 102 determines that the moving object 10 may interfere with the surrounding environment. If the moving object 10 may interfere with the surrounding environment (step ST640; Yes), the determination result indicating that the moving object 10 may interfere with the surrounding environment is output to the warning information output unit 104. The display information creation device 40D proceeds to step ST642. If the shortest distance is equal to or greater than a predetermined threshold, the interference determination unit 102 determines that the moving object 10 is not likely to interfere with the surrounding environment. If the moving object 10 is not likely to interfere with the surrounding environment (step ST640; No), the display information creation device 40D proceeds to step ST646.

If there is a possibility that the moving body 10 will interfere with the surrounding environment (step ST640; Yes), the display information creation device 40D updates the colored point cloud data indicating the interference points to three-dimensional map data (step ST642). Specifically, first, the warning information output unit 104 outputs to the display information creation unit 56D information on the interference points indicating each point of the colored point cloud data whose shortest distance to the possible existence area is less than a predetermined threshold. Next, the display information creation unit 56D updates the colored point cloud data indicating the interference points to the three-dimensional map data of the display information. The display information creation device 40D proceeds to step ST644.

The display information creation device 40D outputs warning information indicating the possibility of interference to the display unit 34 (step ST644). Specifically, the warning information output unit 104 outputs warning information including, for example, character information to be displayed on the display screen 36 of the display unit 34, sound information to be emitted by the speaker, and light information to be emitted by the light-emitting device to the display unit 34. Note that step ST644 may be executed before step ST642. The display information creation device 40D proceeds to step ST646.

The processing of steps ST646, ST648, ST650, and ST652 is similar to the processing of steps ST246, ST248, ST250, and ST252 in the first embodiment, and therefore will not be described.

In step ST638, when calculating the possible presence area of the moving body 10 within a predetermined time from the present, the values of each physical quantity may be used as is, or weighted values may be used. An example of weighting will be described below. In the moving range of the moving body 10, one direction on the horizontal plane is the X direction, and the direction perpendicular to the X direction on the horizontal plane is the Y direction. If the speed of the moving body 10 in the X direction is Vx, the speed in the Y direction is Vy, and the weighting coefficients are α and β, the possible presence area calculation unit 100 calculates the possible presence area by setting the speed of the moving body 10 in the X direction to α×Vx and the speed in the Y direction to β×Vy. Note that Vx<Vy and α<β. As a result, the possible presence area is expanded in the direction in which the movement amount is expected to be large, so that the interference determination unit 102 can make a safer determination.

The display information creation device 40D continuously acquires the three-dimensional map data created in the processes from step ST606 to step ST628, and synthesizes each of the three-dimensional map data. Each of the three-dimensional map data synthesized at this time is unified into a common coordinate system. The display information creation device 40D creates display information including the synthesized three-dimensional map data based on the display settings set by the screen setting unit 60. The display information creation device 40D displays the created display information on the display screen 36 of the display unit 34 in a display format based on the display settings. Furthermore, if there is a possibility that the moving body 10 will interfere with the surrounding environment, the display information creation device 40D outputs warning information and updates the three-dimensional map data by adding colored point cloud data that clearly indicates the location of interference.

(Effects of each embodiment)
The display information creating device 40, 40A, 40B, 40C, and 40D, the display system 1, and the display information creating method described in each embodiment can be understood, for example, as follows.

The display information creation device 40, 40A, 40B, 40C, 40D according to the first embodiment includes a point cloud data acquisition unit 42, 42A that acquires point cloud data including distance information between the moving body 10 and each point constituting the surface of a feature present in a predetermined area 2 at a predetermined period, a position information acquisition unit 46, 46A, 46D that acquires position information of the moving body 10, a map data creation unit 52 that converts the point cloud data (in the embodiment, colored point cloud data in which color data is assigned to each point of the point cloud data) into a common coordinate system based on the position information to create three-dimensional map data and synthesize it as needed, a screen setting unit 60 that can set display settings including a combination of setting values of screen setting data related to the display format of the display screen 36 of the display unit 34, and a display information creation unit 56, 56D that creates display information including a map specified by a predetermined viewpoint of the three-dimensional map data based on the display settings.

The display information creation device 40, 40A, 40B, 40C, 40D according to the first aspect can set display settings in the screen setting unit 60, including a combination of setting values of screen setting data related to the display format of the display screen 36 of the display unit 34. By setting the display settings as a combination of setting values in advance, it is possible to selectively display the three-dimensional map data on the screen in any of a plurality of display settings, and therefore the display format can be easily set. For example, when high operating accuracy is required for the mobile object 10, when there are multiple patterns of operating the mobile object 10, or when it is desirable to have a screen display suitable for the operation of the mobile object 10, any viewpoint and screen configuration can be appropriately set.

In the display information creation device 40, 40A, 40B, 40C, 40D according to the second embodiment, the screen setting unit 60 has a display setting list that can be added, edited, or deleted by an operator's operation, and indicates data in which a plurality of display settings and names corresponding to the display settings are listed and recorded, and outputs a selected display setting to the display information creation unit 56, 56D. This allows the operator to specify a name corresponding to the display setting including a combination of each setting value, thereby allowing the display information including the map of the three-dimensional map data to be displayed on the display unit 34 in the desired display format, without having to input each setting value from scratch.

In the display information creation device 40, 40A, 40B, 40C, 40D according to the third aspect, the screen setting unit 60 has a most recent setting list in which past display settings are temporarily recorded and which indicates data that is automatically deleted, starting with the oldest display setting, when a predetermined capacity is reached. This allows the display unit 34 to quickly display display information including a map of the three-dimensional map data in a display format that was used in the recent past.

In the display information creation device 40, 40A, 40B, 40C, 40D according to the fourth aspect, the screen setting unit 60 can set a display setting for displaying maps from different viewpoints of the three-dimensional map data on the multiple display screens 36 of the display unit 34. The map displayed on the display screen 36 of the display unit 34 is a map of a predetermined viewpoint of the three-dimensional map data. This allows the viewpoint to be set at a position where the measuring device 20 that originally measures the point cloud data and color data is not attached. Since various viewpoints can be set, information on the surrounding environment from an appropriate viewpoint according to the movement and work of the mobile body 10 can be presented to the operator. Furthermore, presenting a map specified by multiple viewpoints contributes to improving the efficiency and accuracy of the movement and work of the mobile body 10. In addition, new three-dimensional map data is added and updated as needed to the three-dimensional map data based on the point cloud data and color data acquired in the past to create the three-dimensional map data, so that maps from multiple viewpoints can be provided without disposing a large number of measuring devices 20.

In the display information creation device 40, 40A, 40B, 40C, 40D according to the fifth aspect, the screen setting unit 60 can set display settings to display a map specified by a specific viewpoint of the three-dimensional map data and an image of the point cloud data (in the embodiment, colored point cloud data in which color data is assigned to each point of the point cloud data) on the multiple display screens 36 of the display unit 34. The colored point cloud data is created based on the point cloud data including distance information and color data which is two-dimensional color information, and the three-dimensional map data is created by converting and synthesizing the coordinate system of the colored point cloud data, so that not only the map of the three-dimensional map data but also the image of the colored point cloud data can be displayed.

The display information creation device 40A, 40B, 40C according to the sixth aspect has area data including position information of a map creation area for which display information including three-dimensional map data is to be created, and is equipped with a mapping control unit 70A, 70B, 70C that outputs a control signal to start creating three-dimensional map data when the position of each point of the point cloud data is included in the map creation area. This allows three-dimensional map data to be created during movement before a specific operation is performed in the mobile body 10. In addition, when three-dimensional map data is created from before the position where a specific operation is performed, the created map data accumulates point cloud data and color data measured for more points that make up the surface of the feature, so that more information on the surrounding environment can be presented to the operator. In addition, since three-dimensional map data is created only for the required area, it can contribute to reducing the load on the power consumption and memory capacity associated with processing.

In the display information creation devices 40A, 40B, and 40C according to the seventh aspect, the area data includes map creation area setting data including position information of the map creation area that is set in advance so that it can be added, edited, or deleted via the operation unit 32 that can be operated by the operator before starting the creation of the three-dimensional map data. This allows areas where three-dimensional map data is predicted to be required to be set in advance as map creation areas, making it possible to automatically create three-dimensional map data for the required areas without excess or deficiency. This eliminates the need for the operator to give an instruction to start creating three-dimensional map data each time, reducing the effort required and suppressing delays in starting creation.

In the display information creation device 40B, 40C according to the eighth aspect, when the mapping control unit 70B, 70C receives an operation signal instructing the start of creation of three-dimensional map data via the operation unit 32 that can be operated by the operator, it outputs a control signal instructing the start of creation of three-dimensional map data. This makes it possible to create three-dimensional map data even for areas where it was not expected that three-dimensional map data would be needed.

In the display information creation devices 40B and 40C according to the ninth aspect, the area data includes map creation history data including position information of a map creation area for which three-dimensional map data was instructed to be created in the past. As a result, when an instruction is given to create three-dimensional map data for the same area in which the mobile body 10 has moved or worked multiple times, three-dimensional map data can be automatically created based on the history even if the area is not a preset map creation area.

In the display information creation device 40C according to the tenth aspect, when the position of each point of the point cloud data is included in the map creation area, the mapping control unit 70C outputs notification information to notify the operator to select whether or not to start creating three-dimensional map data, and when an operation signal to select the start of three-dimensional map data creation is acquired, outputs a control signal to instruct the start of three-dimensional map data creation. This allows the operator to confirm whether or not a three-dimensional map needs to be created, thereby preventing the creation of unnecessary three-dimensional map data and contributing to a reduction in the load on the processing, such as power consumption and memory capacity.

The display information creation device 40D according to the eleventh aspect includes an interference warning unit 90 that outputs warning information when the shortest distance between a possible presence area indicating an area in which the moving body 10 may be present within a predetermined time from the present and each point of the point cloud data (in the embodiment, colored point cloud data in which each point of the point cloud data is assigned color data) included in the three-dimensional map data is less than a predetermined threshold. This makes it possible to warn the operator of the possibility of interference between the moving body 10 and the surrounding environment, thereby suppressing interference between the moving body 10 and the surrounding environment.

In the display information creation device 40D according to the twelfth aspect, the interference warning unit 90 calculates the possible existence area based on the moving body 3D model data indicating the data of the 3D model of the moving body 10, physical quantities related to the movement of the moving body 10, and operation information for the moving body 10. As a result, by calculating the possible existence area based not only on the current position, moving direction, and moving speed of the moving body 10, but also on operation information for changing the moving direction and moving speed of the moving body 10, it is possible to more appropriately determine the possibility of interference between the moving body 10 and the surrounding environment.

The display system 1 according to the thirteenth aspect includes a display information creation device 40, 40A, 40B, 40C, 40D according to any one of the first to twelfth aspects, a mobile body 10 capable of moving within a predetermined area 2 and communicating with the display information creation devices 40, 40A, 40B, 40C, 40D, an operation unit 32 capable of inputting operations to the display information creation devices 40, 40A, 40B, 40C, 40D and the mobile body 10, and a display unit 34 capable of displaying display information, and a base station 3 capable of communicating with the display information creation devices 40, 40A, 40B, 40C, 40D and the mobile body 10.

In the display system 1 according to the thirteenth aspect, the display setting unit 60 of the display information creation device 40, 40A, 40B, 40C, 40D can set display settings including a combination of setting values of screen setting data related to the display format of the display screen 36 of the display unit 34. By setting the display settings as a combination of setting values in advance, it is possible to selectively display the three-dimensional map data on the display screen 36 in any of a plurality of display settings, so that the display format can be easily set.

The display information creation method according to the fourteenth aspect includes steps ST206, ST306, ST406, ST506, and ST606 of periodically acquiring point cloud data including distance information between the moving body 10 and each point constituting the surface of a feature present in a predetermined region 2, steps ST210, ST310, ST410, ST510, and ST610 of acquiring position information of the moving body 10, steps ST228, ST328, ST428, ST528, and ST628 of converting the point cloud data (in the embodiment, colored point cloud data in which color data is assigned to each point of the point cloud data) into a common coordinate system based on the position information to create three-dimensional map data and synthesizing it as needed, a step of setting display settings including a combination of setting values of screen setting data related to the display format of the display screen 36 of the display unit 34, and steps ST250, ST350, ST450, ST550, and ST650 of creating display information including a map identified by a predetermined viewpoint of the three-dimensional map data based on the display settings.

The display information creation method according to the fourteenth aspect can set display settings including a combination of setting values of screen setting data related to the display format of the display screen 36 of the display unit 34. By setting the display settings as a combination of setting values in advance, it is possible to selectively display the three-dimensional map data on the screen in any of a plurality of display settings, and therefore the display format can be easily set. For example, when high operating accuracy is required of the mobile object 10, when there are multiple patterns of operating the mobile object 10, or when it is desirable to have a screen display suitable for the operation of the mobile object 10, any viewpoint and screen configuration can be appropriately set.

Although the embodiments of the present disclosure have been described above, the embodiments are not limited to the contents of these embodiments. For example, instead of the laser survey sensor 22 that acquires point cloud data in the moving body 10 and the color data acquisition sensor 24 that acquires color data, a distance image sensor that acquires colored point cloud data may be provided. In this case, the display information creation devices 40, 40A, 40B, 40C, and 40D each have a colored point cloud data acquisition unit that acquires colored point cloud data instead of the point cloud data acquisition unit 42 and the color data acquisition unit 44.

The display information creation devices 40, 40A, 40B, 40C, and 40D may also extract a predetermined effective area from the colored point cloud data by filtering, and create three-dimensional map data from only the colored point cloud data that corresponds to the extracted effective area. The predetermined effective area may include, for example, an area that corresponds to the illumination range of a lighting device mounted on the mobile body 10.

In addition, in the embodiment, colored three-dimensional map data is created from colored point cloud data in which each point of the point cloud data is assigned color data, but color data is not essential in the present disclosure. That is, three-dimensional map data with luminance may be created from luminance-assigned point cloud data in which luminance data is assigned to each point of the point cloud data instead of color data, or three-dimensional map data that does not include color information or luminance information may be created from point cloud data. In this case, for example, the mapping control unit 70, 70A, 70B, 70C may instruct the map data creation unit 52 to start creating three-dimensional map data instead of the colored point cloud data creation unit 50. Also, an image of the colored point cloud data may be displayed on the display screen 36 of the display unit 34.

In addition, in the embodiment, the mobile body 10 and the base station 3 are connected via the communication network 4, but the present disclosure is not limited to a configuration via communication. In other words, the present disclosure is not limited to a configuration in which an operator is present remotely from the specified area 2. For example, when the present disclosure is applied to a mobile body such as an automobile, construction machine, or manned machine such as a forklift, the mobile body may be provided with the configuration of the base station 3, i.e., the operation unit 32, the display unit 34, and the display information creation device 40, and the control unit 18 and the configuration of the base station 3 may be electrically connected.

REFERENCE SIGNS LIST 1 Display system 2 Predetermined area 3 Base station 4 Communication network 10 Mobile body 32 Operation unit 34 Display unit 36 Display screen 40, 40A, 40B, 40C, 40D Display information creation device 42, 42A Point cloud data acquisition unit 44 Color data acquisition unit 46, 46A, 46D Position information acquisition unit 48 Video data acquisition unit 50 Colored point cloud data creation unit 52 Map data creation unit 54, 54D Map data storage unit 56, 56D Display information creation unit 58 Display information output unit 60 Screen setting unit 70, 70A, 70B, 70C Mapping control unit 90 Interference warning unit

Claims (13)

a point cloud data acquisition unit that acquires point cloud data including distance information between the moving body and each point constituting a surface of a feature present in a predetermined area at a predetermined period;
a color data acquisition unit that acquires color data including color information corresponding to each point included in the point cloud data;
a location information acquisition unit for acquiring location information of the moving object;
a video data acquisition unit that acquires video data having a viewpoint of the moving object;
a colored point cloud data creation unit that creates colored point cloud data by combining the point cloud data and the color data;
a map data creation unit that converts the colored point cloud data into a common coordinate system based on the position information to create three-dimensional map data and synthesizes the data as needed;
a screen setting unit capable of setting display settings including a combination of setting values of screen setting data relating to a display format of a display screen of the display unit;
a display information creation unit that creates display information including a map specified by a predetermined viewpoint of the three-dimensional map data , an image of the colored point cloud data, and an image of the image data based on the display setting;
Equipped with
the screen setting unit has a display setting list indicating data in which a plurality of the display settings and names corresponding to the display settings are listed and recorded, and outputs the selected display setting to the display information creation unit;
Display information creation device. the screen setting unit has a most recent setting list in which the past display settings are temporarily recorded and which indicates data to be automatically deleted in order from the oldest display setting when a predetermined capacity is reached;
2. The display information generating device according to claim 1 . the screen setting unit is capable of setting the display settings to display maps of the three-dimensional map data from different viewpoints on a plurality of display screens of the display unit, respectively.
3. The display information creating device according to claim 1 or 2 . the screen setting unit is capable of setting the display settings to display a map specified by a predetermined viewpoint of the three-dimensional map data and an image of the colored point cloud data on each of a plurality of display screens of the display unit,
The display information creating device according to claim 1 . a mapping control unit that has area data including position information of a map creation area for which the display information including the three-dimensional map data is to be created, and that outputs a control signal to instruct a start of creation of the three-dimensional map data when the positions of the points of the point cloud data are included in the map creation area;
The display information creating device according to claim 1 . the area data includes map creation area setting data including position information of the map creation area that is set in advance via an operation unit that can be operated by an operator before starting the creation of the three-dimensional map data, so as to be addable, editable, or deletable;
6. The display information generating device according to claim 5 . the mapping control unit outputs a control signal instructing to start creating the three-dimensional map data when an operation signal instructing to start creating the three-dimensional map data is acquired via an operation unit that can be operated by an operator.
7. The display information creating device according to claim 5 or 6 . the area data includes map creation history data including position information of the map creation area for which an instruction to create the three-dimensional map data was given in the past;
8. The display information generating device according to claim 7 . the mapping control unit outputs notification information for notifying an operator to select whether or not to start creating the three-dimensional map data when a position of each point of the point cloud data is included in the map creation area, and outputs a control signal instructing the start of creation of the three-dimensional map data when an operation signal for selecting the start of creation of the three-dimensional map data is acquired.
The display information creating device according to any one of claims 5 to 8 . an interference warning unit that outputs warning information when a shortest distance between a possible presence area indicating an area where a moving object may exist within a predetermined time from the present and each point of the colored point cloud data included in the three-dimensional map data is less than a predetermined threshold;
The display information creating device according to any one of claims 1 to 9 . the interference warning unit calculates the possible presence area based on moving object three-dimensional model data indicating data of a three-dimensional model of the moving object, physical quantities related to the movement of the moving object, and operation information for the moving object.
The display information creating device according to claim 10 . A display information creating device according to any one of claims 1 to 11 ,
a moving object capable of moving within the predetermined area and communicating with the display information creating device;
a base station including an operation unit capable of inputting operations to the display information creation device and the mobile object, and a display unit capable of displaying the display information, the base station being capable of communicating with the display information creation device and the mobile object;
A display system comprising: acquiring point cloud data including distance information between the moving body and each point constituting a surface of a feature present in a predetermined area at a predetermined period;
acquiring color data including color information corresponding to each point included in the point cloud data;
acquiring location information of the moving object;
acquiring video data with the moving object as a viewpoint;
creating colored point cloud data by combining the point cloud data and the color data;
a step of converting the colored point cloud data into a common coordinate system based on the position information to generate three-dimensional map data and synthesizing the three-dimensional map data as needed;
setting a display setting including a combination of setting values of screen setting data related to a display format of a display screen of a display unit;
creating display information including a map specified by a predetermined viewpoint of the three-dimensional map data , an image of the colored point cloud data, and an image of the video data based on the display setting;
Including ,
The method further includes a step of allowing an operator to add, edit, or delete a display setting list indicating data in which a plurality of the display settings and names corresponding to the display settings are listed and recorded, and outputting the selected display setting.
How to create display information.

Images