JP2021182177A - Vehicle steering system and vehicle steering method - Google Patents

Abstract

To provide a vehicle steering system and a vehicle steering method capable of obtaining highly accurate ambient environment information including dead angle information in a wide range even in a case of a long interval between a self vehicle and the other vehicle.SOLUTION: A vehicle steering system includes: a self vehicle 10A; and the other vehicle 10B which is bi-directionally communicable with the self vehicle. Each one of the self vehicle and the other vehicle includes: an environment measurement device 12 for acquiring ambient three-dimensional information 1A and 1B; a self position orientation device 13 for acquiring position posture information 2A and 2B; a calculation device 14; and an inter-vehicle communication device 18 for bi-directionally communicating three-dimensional information and position posture information. Besides, the calculation device 14 of the self vehicle 10A has an information fusion function of generating fusion three-dimensional information 4 by fusing three-dimensional information of the self vehicle with three-dimensional information of the other vehicle with the use of the three-dimensional information and the position posture information received from the other vehicle.SELECTED DRAWING: Figure 2

Description

The present invention relates to a vehicle maneuvering system and a vehicle maneuvering method for unmanned or manned vehicles.

Unmanned vehicles may be used as vehicles for transporting goods, civil engineering work, reconnaissance work, etc. in disaster areas such as volcanic eruptions and accidents at nuclear power plants, or in dangerous places.
In addition, an unmanned vehicle equipped with a monitoring sensor such as a camera may be used as a vehicle for monitoring a vast place such as an airport or an air base for security.

The unmanned vehicle runs and performs various tasks according to the command from the remote control operator in a safe place. Further, in an unmanned vehicle, it is common practice to transmit ambient environment information such as surrounding images and three-dimensional information to the remote control so that the remote control can easily control the vehicle.

The remote control operator refers to this information to perform remote control. For remote control, it is necessary to operate the accelerator, brake, steering, shift, etc. (hereinafter, "control equipment") and the driving amount of various work devices. The remote control means is a means for directly transmitting these operation amounts, a means for indicating only the coordinates where the vehicle should travel, and a means for autonomously determining the operation amount of the control device so that the vehicle side does not collide with an obstacle or the like. It may be any of the intermediate means.

During this remote control, in order for an unmanned vehicle to avoid obstacles when driving or performing various tasks, it acquires environmental recognition information around the vehicle (for example, an environmental recognition map) and recognizes where and what kind of obstacles exist. Is being done. "Obstacle avoidance" means recognizing an obstacle to be avoided and safely stopping or circumventing it in front of it.

For example, Patent Document 1 is disclosed as a means for acquiring the above-mentioned surrounding environment information and environment recognition information.
Patent Document 1 provides a moving body with a laser radar and a camera, and generates a three-dimensional image using camera data obtained while moving, laser data, INS data consisting of position and orientation, and GPS time.

Further, Patent Document 2 discloses a work vehicle in which a plurality of adjacent vehicles can be driven by remote control without interfering with each other.

In Patent Document 2, each work vehicle can move independently, and an obstacle recognition sensor that detects an obstacle in the direction of another vehicle and outputs the obstacle information, and mutual communication capable of communicating with the other vehicle. The device is provided with an interference prevention device for preventing interference between the own vehicle and another vehicle. When there is a possibility that the other vehicle interferes with the own vehicle, a suppression command to stop or decelerate the drive of the drive arm is output to the own vehicle, and an alarm or suppression command to the other vehicle is output to the mutual communication device. do.

Japanese Patent No. 4486737 Japanese Unexamined Patent Publication No. 2016-45674

It is required to realize smooth remote control by acquiring 3D information of the surrounding environment of the unmanned vehicle and providing it to the remote controller in the work and running by remote control of unmanned vehicles such as construction machinery. ..

In order to satisfy this demand, it is possible to generate ambient environment information by synthesizing the results measured at a plurality of positions by using a laser radar or a camera in a single vehicle by the technique of Patent Document 1.
However, in the means of Patent Document 1, since the ambient environment information is generated using the data obtained while moving, it takes time to generate the ambient environment information. Further, since it is based only on the data obtained by the remote-controlled unmanned vehicle (own vehicle), it is not possible to grasp the information of the blind spot portion of the own vehicle.

On the other hand, in the means of Patent Document 2, since the obstacle in the direction of the other vehicle is detected and the obstacle information is communicated with the other vehicle, it is possible to prevent the own vehicle and the other vehicle from interfering with each other.
Further, in the means of Patent Document 2, since the complex obstacle information obtained by superimposing the obstacle information of the own vehicle and the other vehicle is transmitted to the control device, the obstacle information of the blind spot portion of the own vehicle can be remotely controlled by the own vehicle. Can be grasped by a person.

However, in the means of Patent Document 2, since it is premised that the own vehicle and the other vehicle are close to each other, highly accurate blind spot information cannot be obtained when the distance between the own vehicle and the other vehicle is large.

The present invention has been devised to solve the above-mentioned problems. That is, an object of the present invention is to provide a vehicle maneuvering system and a vehicle maneuvering method capable of obtaining a wide range and highly accurate surrounding environment information including blind spot information even when the distance between the own vehicle and another vehicle is large.

According to the present invention, the own vehicle and another vehicle capable of bidirectionally communicating with the own vehicle are provided.
The own vehicle and the other vehicle have an environment measurement device that acquires three-dimensional information of the surroundings, a self-positioning device that acquires position / orientation information, an arithmetic device that generates an environment recognition map, and the three-dimensional information and the above-mentioned three-dimensional information. It has an inter-vehicle communication device that bidirectionally communicates the position / attitude information.
The arithmetic unit of the own vehicle further fuses the three-dimensional information of the own vehicle and the three-dimensional information of the other vehicle by using the three-dimensional information and the position / attitude information received from the other vehicle. A vehicle maneuvering system having an information fusion function for generating fusion three-dimensional information is provided.

Further, according to the present invention, the own vehicle and another vehicle capable of bidirectional communication with the own vehicle are prepared.
The own vehicle and the other vehicle acquire surrounding three-dimensional information and position / attitude information, respectively, and communicate the three-dimensional information and the position / attitude information in both directions.
Further, using the three-dimensional information and the position / attitude information received from the other vehicle, the three-dimensional information of the own vehicle and the three-dimensional information of the other vehicle are fused to generate fusion three-dimensional information. A vehicle maneuvering method is provided.

According to the present invention, the arithmetic unit uses the three-dimensional information and the position / attitude information received from the other vehicle, the three-dimensional information of the own vehicle, and the position / attitude information, and the three-dimensional information of the own vehicle and the three-dimensional information of the other vehicle. To generate fusion 3D information.
This fused three-dimensional information includes point cloud information in which common point cloud information of a plurality of three-dimensional information matches and does not common.
Therefore, even when the distance between the own vehicle and the other vehicle is large, it is possible to correct the error of the relative position / posture of the other vehicle as seen from the own vehicle and obtain a wide range and highly accurate surrounding environment information including blind spot information.

In addition, by being able to combine information from multiple vehicles with high accuracy and in real time, the vehicle operator can grasp a wide range of highly accurate surrounding environment information including blind spot information, and it is possible to operate the vehicle efficiently. Become.

It is an overall block diagram of the vehicle maneuvering system by this invention. It is a functional block diagram of a vehicle control system. It is a conceptual diagram of reduction of blind spot by information fusion. It is a figure which shows typically the three-dimensional information of the surroundings by own vehicle and other vehicle. It is a figure which shows typically the information provision by the vehicle-to-vehicle communication of own vehicle and other vehicle. It is a figure which shows the correction by scan matching schematically. It is a schematic diagram which shows the display content by the display device of a remote control device. It is a conceptual diagram similar to FIG. 3 when there are a plurality of other vehicles. It is an overall flow chart of information fusion processing in own vehicle. It is an overall flow chart of the surrounding environment 3D image generation processing in own vehicle. It is an overall flow chart of information provision processing in another vehicle. It is a schematic diagram which shows the adjustment method of a flat / uneven area. It is explanatory drawing of common measurement area calculation. It is explanatory drawing of filtering.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, the same reference numerals are given to the common parts in each figure, and duplicate description is omitted.

FIG. 1 is an overall configuration diagram of a vehicle control system 100 according to the present invention.
In this figure, the vehicle maneuvering system 100 includes the own vehicle 10A and another vehicle 10B capable of bidirectional communication with the own vehicle 10A.
The own vehicle 10A may be a manned vehicle and may be operated by the driver by displaying the surrounding environment 3D image 6 (described later) based on the present invention on the display device in the vehicle.
In this example, the own vehicle 10A is an unmanned vehicle that can be remotely controlled, and the vehicle control system 100 further includes a remote control device 20 that remotely controls the unmanned vehicle.
Hereinafter, the own vehicle 10A is also referred to as an information fusion side. Further, the other vehicle 10B is also referred to as an information providing side.

In FIG. 1, the own vehicle 10A includes a vehicle body 11, an environment measuring device 12, a self-positioning device 13, an arithmetic device 14, a drive-by-wire device 15, a remote communication device 16, and an inter-vehicle communication device 18.

The vehicle body 11 has a basic traveling function as a vehicle. There are wheeled type or tracked (tracked) type, or a hybrid type thereof.

FIG. 2 is a functional block diagram of the vehicle control system 100.

In FIGS. 1 and 2, the environment measuring device 12 of the own vehicle 10A is a camera, a laser range finder (LRF), a radar, and the like attached to the vehicle body 11.
The environment measuring device 12 of the own vehicle 10A acquires the three-dimensional information 1A around the own vehicle 10A. The environment measuring device 12 of the other vehicle 10B also acquires the three-dimensional information 1B around the other vehicle 10B in the same manner.
Hereinafter, the case where the three-dimensional information 1A is the point cloud information will be described.

Further, the self-positioning device 13 of the own vehicle 10A is a sensor such as a global positioning system (GNSS) or an inertial measurement unit (IMU) for acquiring absolute coordinates, velocity, attitude, and acceleration information of the own vehicle 10A. To determine the position and attitude information of the own vehicle (hereinafter, "position / attitude information 2A") including.

Further, the self-positioning device 13 of the other vehicle 10B also determines the position and posture information of the other vehicle (hereinafter, “position / posture information 2B”) in the same manner.

The arithmetic unit 14 is composed of a computer or the like.
The arithmetic unit 14 of the own vehicle 10A can pass the obstacle information around the own vehicle 10A, the place where the own vehicle 10A can pass, and the own vehicle 10A from the three-dimensional information 1A measured by the environmental measurement device 12. Find the speed and generate these as an environment recognition map.

Hereinafter, the environment recognition map and the environment recognition information mean the map and information necessary for the autonomous traveling function of the own vehicle 10A. The environment recognition map is generated by the own vehicle itself.
Further, the surrounding environment information and the surrounding environment 3D image 6 described later mean information, a map, and a 3D image (three-dimensional image) presented to the remote operator. The remote control operator remotely controls the own vehicle 10A using these information (control support information).

Further, the arithmetic unit 14 has an obstacle in the environment recognition map generated by itself based on the accelerator, the brake command (or speed command), the steering command (or direction command), the coordinate command to be traveled, etc. from the remote control device 20. Plan a route that does not collide with objects.

Further, the arithmetic unit 14 calculates the control amount of each actuator based on the planned path and outputs it as a control command to the drive-by-wire device 15.

Further, the arithmetic unit 14 of the own vehicle 10A has an information fusion function of fusing the three-dimensional information 1B of the other vehicle 10B obtained from the other vehicle 10B with its own three-dimensional information 1A.

In addition, hereinafter, "fusion" means one three-dimensional information as a whole including common point group information of a plurality of three-dimensional information 1A and 1B and not common to each point group information ("fusion". It means to create (called 3D information 4). Therefore, a scan matching method such as ICP, which will be described later, is used.
It is preferable that the information fusion function by the arithmetic unit 14 is performed in substantially real time.
The arithmetic unit 14 that generates the fusion three-dimensional information 4 and the arithmetic unit 14 that generates the environment recognition map may be different devices.

The drive-by-wire device 15 of the own vehicle 10A is an actuator or the like attached to an accelerator, a brake, a steering, a shift, or the like (“control device”) in order to allow the vehicle body 11 to travel unmanned. The drive-by-wire device 15 has a function of operating a control device by a predetermined amount or setting a predetermined position according to a control command received from the arithmetic unit 14.
Depending on the vehicle, the control device itself may input a control command to perform a predetermined amount of operation. In this case, no actuator is required.

The remote communication device 16 of the own vehicle 10A is composed of a communication device and an antenna for communicating with the remote control device 20.
Further, the remote communication device 16 is not only for direct communication with the remote control device 20, but also for another own vehicle 10A, another vehicle 10B, equipped with a relay communication device for communication with the remote control device 20. It is also used to communicate with manned vehicles or just repeaters.

The communication distance between the own vehicle 10A and the remote control device 20 is a long distance (for example, 1 km or more), and generally has a low communication capacity. Therefore, the communication information from the own vehicle 10A to the remote control device 20 is not the three-dimensional information from a plurality of vehicles, but the surrounding environment 3D which is a two-dimensional image of the fused three-dimensional information 4 fused by the single own vehicle 10A. Image 6 shows that communication is possible even with a low communication capacity by reducing the communication load.

The inter-vehicle communication device 18 is composed of a communication device and an antenna. The vehicle-to-vehicle communication device 18 uses the own vehicle 10A (information fusion side) to combine the three-dimensional information 1B and the position / attitude information 2B acquired by the other vehicle 10B (information providing side) in order to fuse the three-dimensional information by a plurality of vehicles. Used to provide to. The other vehicle 10B is not limited to one, and may be two or more.

The communication distance between the own vehicle 10A and the other vehicle 10B is a short distance (for example, 50 to 100 m), and a decrease in communication capacity does not substantially occur. Therefore, the own vehicle 10A can receive the three-dimensional information 1B and the position / attitude information 2B of the other vehicle 10B in substantially real time, and can improve the accuracy of the fused three-dimensional information 4.

If the communication capacity and communication speed of the remote communication device 16 are sufficient, the remote communication device 16 may also serve as the inter-vehicle communication device 18. Further, the fusion 3D information 4 may be transmitted to the remote control device 20 instead of the surrounding environment 3D image 6 in which the fusion 3D information 4 is imaged.

In FIG. 2, the own vehicle 10A and the other vehicle 10B each have an environment measuring device 12 and an inter-vehicle communication device 18.

The own vehicle 10A further has the above-mentioned arithmetic unit 14. The arithmetic unit 14 uses the three-dimensional information 1B and the position / attitude information 2B received from the other vehicle 10B to fuse the three-dimensional information 1A of the own vehicle and the three-dimensional information 1B of the other vehicle to generate the fused three-dimensional information 4. It has an information fusion function.
The fusion three-dimensional information 4 includes a wide range of highly accurate ambient environment information including blind spot information.

The information fusion function has a relative position / posture correction function A and a fusion information generation function B.
The relative position / posture correction function A corrects the relative position and relative posture of the other vehicle 10B by scan matching using the three-dimensional information 1A, 1B and the position / posture information 2A, 2B of the own vehicle 10A and the other vehicle 10B.
The fusion information generation function B generates the fusion 3D information 4 by fusing the 3D information 1B of the other vehicle 10B with the 3D information 1A of the own vehicle based on the corrected relative position and relative posture of the other vehicle 10B. ..

In FIG. 2, the arithmetic unit 14 of the own vehicle 10A further has an ambient environment 3D image generation function C that creates an ambient environment 3D image 6 at an arbitrary viewpoint / gazing point from the fusion 3D information 4.
The remote communication device 16 transmits the surrounding environment 3D image 6 to the remote control device 20.

The other vehicle 10B may have the same configuration as the own vehicle 10A. In this case, the remote control of the other vehicle 10B is performed by another remote control device. Further, the other vehicle 10B may be the information providing side and at the same time the information fusion side for its own remote control.
The other vehicle 10B is not limited to an unmanned vehicle and may be a manned vehicle as long as it has the above-mentioned environment measuring device 12 and the inter-vehicle communication device 18.
Further, the other vehicle 10B is not limited to one, and may be two or more.

In FIG. 2, the remote control device 20 has a remote communication device 22 that communicates bidirectionally with the own vehicle 10A.

The remote control device 20 wirelessly communicates with the own vehicle 10A, and the remote control operator who operates the remote control device 20 gives a control command such as a running instruction or a work instruction to the own vehicle 10A, and also from the own vehicle 10A. It is a device that presents or records the obtained information to the remote control operator.

The remote control device 20 may be a stationary type for maneuvering the own vehicle 10A remotely from a fixed place, or while the remote controller moves with the own vehicle 10A or moves at a place away from the own vehicle 10A. There are forms such as wearing and portable type for maneuvering.

The remote control device 20 includes a remote communication device 22, a control device 24, a command input device 26, and a display device 28.
The remote communication device 22 is composed of an antenna and a communication device, and communicates bidirectionally with the own vehicle 10A. The control device 24 is composed of a computer or the like that performs information processing. The command input device 26 inputs a control command such as an instruction, a speed, and other setting commands to the control device for the own vehicle 10A. The display device 28 is a display or the like that displays information (surrounding environment 3D image 6, etc.) sent from the own vehicle 10A.

It is preferable that the bidirectional communication between the remote control device 20 and the own vehicle 10A is performed in substantially real time. However, when the own vehicle 10A is autonomously traveling, communication may be performed only when necessary.

In the stationary remote control device 20, the command input device 26 may have a control device similar to that of the vehicle body 11. There is also a device that doubles as a command input device and a display device, such as a touch panel display.

FIG. 3 is a conceptual diagram of reducing blind spots by information fusion of the present invention. In this figure, (A) shows a conventional example, and (B) shows the present invention.
In the vehicle maneuvering method of the present invention, the own vehicle 10A, the remote control device 20, and the other vehicle 10B described above are prepared, and the three-dimensional information 1B and the position / attitude information 2B received from the other vehicle 10B and the three-dimensional information 1A of the own vehicle are prepared. And the position / orientation information 2A is used to generate the fusion three-dimensional information 4.

As shown in FIG. 3A, when based only on the data obtained by the remotely controlled own vehicle 10A (own vehicle), the information of the blind spot portion seen from the own vehicle 10A generated behind an obstacle or the like is grasped. Can't.
In the present invention, in order to obtain the blind spot information seen from the own vehicle 10A, as shown in FIG. 3B, the other vehicle 10B acquires the three-dimensional information 1B and the position / attitude information 2B of the surrounding environment, and these acquisitions. Using the information, the three-dimensional information 1A of the own vehicle and the three-dimensional information 1B of the other vehicle 10B are fused. The fused information is provided to the remote controller in real time. Thereby, it is possible to provide good ambient environment information (for example, ambient environment 3D image 6) with few blind spots.

In order to perform information fusion by a plurality of vehicles, it is necessary to acquire the relative position and the relative posture between the vehicles, but the method using only GPS / INS as the method may have a large error. Therefore, in the present invention, a more accurate relative position / orientation can be obtained by performing correction by scan matching such as an ICP (Iterative Closest Point) algorithm using the three-dimensional point cloud information itself of the surrounding environment.

4 to 6 are explanatory views of scan matching.

FIG. 4 is a diagram schematically showing the surrounding three-dimensional information 1A and 1B by the own vehicle 10A and the other vehicle 10B.
In FIG. 4A, the three-dimensional information 1A acquired by the own vehicle 10A is indicated by a black circle (●). The three-dimensional information 1A that can be acquired by the own vehicle 10A by the structure 5 such as a building and rubble is limited to the front surface and the side surface of the structure 5, and the back side is a blind spot portion of the own vehicle 10A.
In FIG. 4B, the three-dimensional information 1B acquired by the other vehicle 10B is indicated by a white circle (◯). The three-dimensional information 1B that can be acquired by the other vehicle 10B by the structure 5 such as a building and rubble is limited to the back surface and the side surface of the structure 5, and the front side is the blind spot portion of the other vehicle 10B.

FIG. 5 is a diagram schematically showing information provision by vehicle-to-vehicle communication between the own vehicle 10A and another vehicle 10B.
In FIG. 5A, the three-dimensional information 1B and the position / attitude information 2B around the other vehicle 10B (information providing side) are provided to the own vehicle 10A (information fusion side) by vehicle-to-vehicle communication.
In FIG. 5B, since there is an error in the position / attitude information obtained from the GPS / INS of the own vehicle 10A and the other vehicle 10B, it is relative to the relative position of the other vehicle 10B with respect to the own vehicle 10A obtained from the difference. There is also an error in the posture.
Therefore, a "deviation" occurs between the three-dimensional information 1A of the own vehicle 10A and the three-dimensional information 1B of the other vehicle 10B, which are fused based on the relative position and the relative posture with an error. This means that the black circles and the white circles on the side surface of the structure 5 are inconsistent with each other in FIG. 5 (B).

FIG. 6 is a diagram schematically showing correction by scan matching.
In this figure, the three-dimensional information 1A (black circle ●) of the own vehicle 10A and the three-dimensional information 1B (white circle ○) of the other vehicle 10B have common point cloud information. The relative position and the relative posture of the other vehicle 10B with respect to the own vehicle are corrected by scan matching based on the corresponding point cloud information so that the common point cloud information matches.
Specifically, when the ICP algorithm is used, the calculation is repeated so that the sum of squares of the distances between both point clouds is minimized, and the relative position and orientation at the time of the most match is obtained.

In other words, the black circles that are the information (1A, 2A) by the own vehicle 10A and the white circles that are the information (1B, 2B) by the other vehicle 10B are shown in FIG. It's different. Therefore, as shown in FIG. 6, by moving the "common data" so as to match (scan matching), the entire black circle and white circle data are translated and rotated. The amount of translation and the amount of rotation correspond to the amount of correction of the relative position and the relative posture of the other vehicle 10B as seen from the own vehicle 10A.

By the above-mentioned scan matching, the relative position and the relative posture of the other vehicle 10B with respect to the own vehicle can be corrected. By fusing the three-dimensional information 1A and 1B of the two vehicles based on the corrected relative position and the relative posture, the fused three-dimensional information 4 having no blind spot for the own vehicle 10A is created.

FIG. 7 is a schematic diagram showing the display contents of the remote control device 20 by the display device 28.
The display device 28 displays the surrounding environment 3D image 6 sent from the own vehicle 10A. The surrounding environment 3D image 6 represents a state of viewing from the designated viewpoint and the designated distance with respect to the designated gazing point in the fusion 3D information 4 as a two-dimensional image.

FIG. 7A is an example of a vehicle front image of an example of the surrounding environment 3D image 6. In this figure, the shaded area is a place with severe undulations (for example, grassland or bush), and the white part in the center is a place with few undulations and suitable for traveling (for example, a lane). The vehicle front image is suitable when the operator virtually gets on the own vehicle 10A and directly operates the control device of the own vehicle 10A via the command input device 26.
When this display is performed, the remote controller may indicate the gaze point to be in front of the own vehicle 10A and the viewpoint and distance to be inside the own vehicle 10A.
In the case of FIG. 7A, it is preferable that the point cloud information obtained by the laser range finder and the image taken by the camera are color-matched and the point cloud information is displayed in color like a camera image.

FIG. 7B is another example of the ambient environment 3D image 6. In this case, a part or all of the vehicle body 11 of the own vehicle 10A and the other vehicle 10B can be displayed in the surrounding environment 3D image 6.
When this display is performed, the remote controller may indicate the viewpoint to be in front of the own vehicle 10A and the viewpoint and distance to be above the gazing point.
Further, this display example is suitable for indicating a position or direction in which the vehicle should travel with respect to the self-driving vehicle 10A, or for a bird's-eye view of the traveling area or the entire work area.

The surrounding environment 3D image 6 is not limited to this example, and can be freely changed to a two-dimensional image or the like viewed from an angle according to the purpose of remote control.

FIG. 8 is a conceptual diagram similar to FIG. 3 when there are a plurality of other vehicles 10B.
In this case, it is preferable that the other vehicle 10B has the same configuration as the own vehicle 10A.

The own vehicle 10A searches for another vehicle 10B capable of bidirectional communication by the inter-vehicle communication device 18 of the own vehicle. The communicable range is indicated by a broken line circle. When there are a plurality of other vehicles 10B searched for, the own vehicle 10A communicates with a part or all of them to generate the above-mentioned fusion three-dimensional information 4.
The selection of "part or all" may be set in advance in the own vehicle 10A, or may be commanded from the remote control device 20.
Further, the own vehicle 10A has an information sharing function of transmitting the generated fusion three-dimensional information 4 to the other vehicle 10B.

With this configuration, the range of the fusion three-dimensional information 4 shared via the other vehicle 10B having the same configuration as the own vehicle 10A can be expanded to a range in which the own vehicle cannot directly communicate.
Further, since each of the plurality of other vehicles 10B has an information fusion function, the calculation load of each own vehicle 10A can be reduced even when a wide range of fusion three-dimensional information 4 is created.

According to the above-described embodiment of the present invention, the arithmetic unit 14 uses the three-dimensional information 1B and the position / attitude information 2B received from the other vehicle 10B to obtain the three-dimensional information 1A of the own vehicle and the three-dimensional information 1B of the other vehicle. Is fused to generate fusion three-dimensional information 4.
The fused three-dimensional information 4 includes point cloud information in which common point cloud information of the plurality of three-dimensional information 1A and 1B is matched and is not common.
Therefore, even when the distance between the own vehicle and the other vehicle 10B is large, the error in the relative position and orientation of the other vehicle 10B as seen from the own vehicle 10A is corrected, and a wide range and highly accurate surrounding environment information (surrounding environment 3D image) including blind spot information is corrected. 6) can be obtained.

In addition, by fusing information from multiple vehicles with high accuracy and in real time, the driver of the vehicle can grasp a wide range of highly accurate surrounding environment information (surrounding environment 3D image 6) including blind spot information, and the efficient vehicle. It becomes possible to perform maneuvering.

FIG. 9 is an overall flow chart of information fusion processing in the own vehicle 10A.
In this figure, the information fusion process comprises each step (step) of S1 to S5.
It should be noted that each step of S1 to S5 is sequentially carried out from n = 1 to n = N with n = n + 1 with respect to the total number N of other vehicles for information fusion.

In step S1, the information (1B, 2B) of the other vehicle (n) is acquired by the inter-vehicle communication device 18, and the three-dimensional information 1A of the own vehicle is acquired by the environment measurement device 12.
In step S2, the position / attitude information 2A of the own vehicle is calculated by the self-positioning device 13.
In step S3, the three-dimensional information used for scan matching is adjusted.
In step S4, the relative position / posture by scan matching is calculated.
In step S5, the conversion of the three-dimensional information of the other vehicle (n) to the own vehicle coordinate system based on the relative position / orientation (that is, the fusion three-dimensional information is generated).

FIG. 10 is an overall flow chart of the surrounding environment 3D image generation processing in the own vehicle 10A.
In this figure, each step (step) of T1 to T3 is repeatedly carried out in the ambient environment 3D image generation process.

In step T1, the operator's display instruction (viewpoint, gazing point, distance) is acquired by the remote control device 20 via the remote communication device 22. For example, when the command input device 26 is a touch panel type display and also serves as a display device 28, the display is shifted with respect to the display of the surrounding environment 3D image 6 by a touch operation such as a swipe operation or a pinch operation such as operating a smartphone. By rotating or enlarging / reducing, the viewpoint, gaze point, and distance are specified.
In step T2, the surrounding environment 3D image 6 is generated from the fusion 3D information 4 based on the display instruction of the operator.
In step T3, the surrounding environment 3D image 6 is transmitted to the remote control device 20 via the remote communication device 22.

FIG. 11 is an overall flow chart of information provision processing in the other vehicle 10B.
In this figure, the information provision process is repeatedly carried out in each step (step) of U1 to U3.

In step U1, the environment measuring device 12 acquires the three-dimensional information 1B of another vehicle.
In step U2, the position / attitude information 2B of another vehicle is calculated by the self-positioning device 13.
In step U3, the information (1B, 2B) of the other vehicle (n) is transmitted to the own vehicle 10A by the inter-vehicle communication device 18.

In FIG. 9, in S3, each step (step) of S31, S32, and S33 is carried out.
In step S31, the flat / uneven area is adjusted.
In step S32, the common measurement area of the three-dimensional information 1A and 1B of the own vehicle and the other vehicle is calculated.
In step S33, fine irregularities are removed by filtering.
Hereinafter, steps S31, S32, and S33 will be specifically described.

(A) Adjustment of flat / uneven area (step S31)
FIG. 12 is a schematic diagram showing a method of adjusting a flat / uneven area.

In scan matching between point clouds, the matching accuracy is affected by the shape of the matching target.
For example, when the terrain in which information is fused is close to flat and has few irregularities, the deviation in the height direction has a greater effect on the overall error than the deviation in the horizontal direction in the process of matching processing. Therefore, the matching accuracy in the height direction is relatively high, and conversely, the matching accuracy in the horizontal direction is relatively low.
On the other hand, when the terrain in which information is fused has many irregularities and few flat portions, the matching accuracy in the horizontal direction is relatively high, and conversely, the matching accuracy in the height direction is relatively low.

Therefore, in order to achieve the same level of matching accuracy in both the height direction and the horizontal direction, it is conceivable to remove the information on the flat portion or the uneven portion before subjecting the three-dimensional information to scan matching.

In order to do this, first, it is classified whether each measurement point included in the three-dimensional information 1A of the own vehicle belongs to the flat area or the uneven area.

FIG. 12A shows a step of extracting flat candidates.
Since each measurement point has coordinates (x, y, z) in the three-dimensional space, the height difference between the measurement point and the adjacent measurement point at a certain measurement point is equal to or less than the threshold value Hf [cm] in the flat candidate extraction step. If the measurement point is a flat candidate. The adjacent measurement points are, for example, points measured immediately before or immediately after the target measurement point in the measurement by LRF. The threshold value Hf is, for example, 5 to 20 cm (preferably 10 cm).

After making this determination at all the measurement points, as shown in FIG. 12 (C), a portion where flat candidates continuously exist is obtained by a method such as clustering, and is used as a flat area. In addition, the other areas are uneven areas.

Next, as shown in FIG. 12 (D), one of the measurement points is matched to 3 so that the final ratio of the total area of the flat area to the total area of the uneven area is F: R. Remove from dimensional information.

It should be noted that F: R is a predetermined numerical value because the fusion accuracy in the height direction and the horizontal direction is about the same as a result of scan matching. F: R is, for example, 3: 1 to 8: 1 (preferably 5: 1).
However, in order to prevent deterioration of scan matching accuracy due to excessive removal of 3D information, the upper limit value St [m ² ] of the area to be removed or the upper limit ratio Rt [%] to the original area of the area to be removed may be provided. good. The upper limit value St is, for example, 250 m ² , and the upper limit ratio Rt is, for example, 10%.

It is preferable that this process is performed only on the three-dimensional information 1A of the own vehicle. The other vehicle 10B is particularly implemented because the common measurement area with the own vehicle is used for scan matching in the process of step S32 (calculating the common measurement area of the three-dimensional information of the own vehicle and the three-dimensional information of the other vehicle). You don't have to.

(B) Calculation of common measurement area for 3D information of own vehicle and other vehicles (step S32)
FIG. 13 is an explanatory diagram of common measurement area calculation.

When determining the relative position and posture between vehicles, if scan matching is performed using the data of the location (blind spot in FIG. 13 (A)) that is measured by only one vehicle and not measured by the other vehicle, the matching accuracy is correct. Will be adversely affected.
Therefore, it is desirable to use the common measurement points that both vehicles are measuring in common for scan matching.

To do this, if there are no measurement points of other vehicles (or no more than a certain number) around a certain measurement point (within the radius r [m]), that measurement point is set as an isolated point. Remove from scan matching targets. By doing this at all the measurement points of both vehicles, it is considered possible to extract only the points that are measured in common.

As another method, a method using the surrounding environment information owned or constructed by the own vehicle can be considered.
In FIG. 13B, the measurement point A is a common measurement point because there is no shielding object between the measurement point A and the other vehicle and the measurement range of the other vehicle. Further, the measurement point B is a non-common measurement point because the interference determination line is blocked by the shielding object. Further, the measurement point C is a non-common measurement point because there is no shielding object between the measurement point C and the other vehicle but it is out of the measurement range of the other vehicle.

That is, for a certain measurement point A, B, C, the measurement point is connected to the sensor of the environment measurement device 12 of the other vehicle 10B by a line on the surrounding environment information (this is called an "interference determination line"). If there is no shielding object between the measurement point and the other vehicle 10B and the interference determination line is included in the measurement range of the environmental measurement device 12 of the other vehicle (that is, the other vehicle 10B), the measurement point is the measurement point of both vehicles 10A. , 10B is considered as a common measurement point that can be measured in common.
By performing this at all measurement points, it is possible to extract only the points that are generally measured in common.

Regarding the interference determination line, when the surrounding environment information is composed of a point cloud, the interference determination line passes between the points on the surrounding environment information even though a shielded object actually exists, and the other vehicle (that is, the other vehicle (that is,). It is conceivable that it will be possible to measure from another vehicle 10B). This can be avoided by determining the interference determination line as a cylindrical object having a radius c [cm] and performing an interference determination with a shielding object.

By the above method, the non-common measurement area of both vehicles is removed from the three-dimensional information, and only the part that is measured in common is used for scan matching, so that deterioration of matching accuracy can be prevented.

(C) Removal of fine irregularities by filtering (step S33)
FIG. 14 is an explanatory diagram of filtering.

Known applications for scan matching include mapping by SLAM (Simultaneus Localization And Mapping). This is a measurement in which a moving object such as a vehicle moves around by LRF (laser range finder), etc., and scan matching of the information obtained every moment is performed to calculate the self-position / posture and generate a map at the same time. Is.

In this way, when scan matching is performed moment by moment with a single moving object, the change in the distance and orientation of the moving object during each scan matching is generally small. Therefore, the appearance of the two terrains before and after the movement, which is the scan matching target, from the moving body does not change significantly, so that matching can be performed with high accuracy.

However, in the case of scan matching for information fusion by multiple vehicles in an rough terrain environment, the position and orientation for measuring the terrain are larger than those of a single moving object, and the terrain measured by both vehicles due to the unevenness of the rough terrain environment. It looks different. Therefore, as shown in FIG. 14A, there is a problem that the fine shapes do not match and the matching accuracy is deteriorated.

Therefore, in order to make the appearance of the terrain measured from each vehicle as similar as possible, it is conceivable to remove fine irregularities by filtering.

As a filtering method, as shown in FIG. 14B, a three-dimensional space is set as a set of three-dimensional objects (this is called a grid), and measured three-dimensional information is arranged. The three-dimensional object is preferably a cube having a side having a predetermined length g [cm], but may be a rectangular parallelepiped or a sphere having a radius having a predetermined length.
At this time, there may be a case where a plurality of measurement points are contained in one grid, only one measurement point is contained in another grid, and there is no measurement point in another grid.
When one or more measurement points are included in the grid, the three-dimensional coordinates (x, y, z) obtained by averaging the three-dimensional coordinates (x, y, z) of each measurement point in the one grid are used. Find it and use this as the 3D coordinates of the grid. As shown in FIG. 14C, this is performed for the three-dimensional information of both vehicles and subjected to scan matching.

If the predetermined length g of the grid is too small, the filtering effect is small and fine irregularities cannot be removed. Also, if it is too large, even the characteristics of the terrain when matching will be lost, so it is necessary to select an appropriate size. The length g is, for example, 10 to 50 cm.

Further, as another effect of filtering, the density of the data in the portion where the data of the three-dimensional information exists is eliminated and the data is made uniform. When performing scan matching such as ICP using coarse and dense data, the data in the dense portion has a greater influence than the data in the sparse portion, which may increase the matching error. Therefore, by eliminating the density of data by filtering, there is an effect of suppressing the error of scan matching.

By performing scan matching using grids, that is, filtered three-dimensional information in this way, deterioration of matching information can be suppressed even in information fusion of a plurality of vehicles in an rough terrain environment.

In FIG. 9, in S5, the step (step) of S51 is carried out.
In step S51, the range of other vehicle fusion information is selected.

(D) Range selection of other vehicle fusion information (step S51)
Information fusion is performed by converting the three-dimensional information of another vehicle into the coordinate system of the own vehicle based on the relative position / orientation obtained by scan matching.
At this time, the three-dimensional information on the other vehicle side may be fused by using all the information.

However, since the obtained relative position and orientation include some error, the 3D information fused by this error is originally displayed twice as one object. Therefore, it may be desired to fuse the information not with all the information on the other vehicle side, but only with the blind spot portion for the own vehicle or the portion where the own vehicle does not have the information.
It should be noted that the three-dimensional information of the own vehicle does not need to be removed in steps S31, S32, and S33, or the information after grid formation is used, and all the information measured so as to be the most abundant is used. good.

In order to select the range of the information of the other vehicle to be fused, the result of the common measurement area calculation of the three-dimensional information of the own vehicle and the other vehicle in step S32 is used.
That is, in the process of step S32, the common measurement point of the three-dimensional information of the own vehicle and the other vehicle was obtained, but the portion removed from the three-dimensional information of the other vehicle as the non-common measurement area at this time is for the own vehicle. It corresponds to the blind spot and the part where the own vehicle does not have information.
Therefore, the portion removed as the non-common measurement area in the process of step S32 may be recorded, and the coordinate may be converted into the coordinate system of the own vehicle based on the relative position / orientation obtained by scan matching.

It should be noted that the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made without departing from the gist of the present invention.

A Relative position / posture correction function, B Fusion information generation function,
C Surrounding environment 3D image generation function,
1A, 1B 3D information, 2A, 2B position / orientation information, 4 fusion 3D information,
5 Structure, 6 Surrounding environment 3D image, 10A own vehicle, 10B other vehicle,
11 car body, 12 environmental measurement device, 13 self-positioning device, 14 arithmetic unit,
15 drive-by-wire equipment, 16 remote communication equipment, 18 vehicle-to-vehicle communication equipment,
20 remote control device, 22 remote communication device, 24 control device,
26 Command input device, 28 Display device, 100 Vehicle maneuvering system

According to the present invention, the own vehicle and another vehicle capable of bidirectionally communicating with the own vehicle are provided.
The own vehicle and the other vehicle have an environment measurement device that acquires three-dimensional point cloud information, which is three-dimensional information of surrounding discrete point clouds, a self-positioning device that acquires position / orientation information, and a calculation device. , A vehicle-to-vehicle communication device that bidirectionally communicates the three-dimensional point cloud information and the position / attitude information.
The vehicle of the computing device further said received from other vehicles using a three-dimensional point group information and the position and orientation information, the three-dimensional point of the said another vehicle and the three-dimensional point group data of the vehicle have a data fusion function of generating fusion three-dimensional information by fusing the group information,
The information fusion function uses the three-dimensional point cloud information and the position / attitude information of the own vehicle and the other vehicle, and scan-matches based on the three-dimensional point cloud information of the own vehicle and the other vehicle. A relative position / attitude correction function that corrects the relative position and posture of the other vehicle as seen from the own vehicle, and
A fusion information generation function that generates the fusion 3D information by fusing the 3D point cloud information of the other vehicle with the 3D point cloud information of the own vehicle based on the corrected relative position and the relative posture. And, a vehicle maneuvering system is provided.

Further, according to the present invention, the own vehicle and another vehicle capable of bidirectional communication with the own vehicle are prepared.
The own vehicle and the other vehicle acquire three-dimensional point cloud information and position / orientation information, which are three-dimensional information of surrounding discrete point clouds, respectively, and bidirectionally exchange the three-dimensional point cloud information and the position / attitude information. Communicate with
Furthermore, said received from other vehicles using a three-dimensional point group information and the position and orientation information, the fused the 3D point group information of the 3D point group information and the other vehicles of the vehicle fusion 3 Generate dimensional information ,
The own vehicle uses the three-dimensional point cloud information and the position / attitude information of the own vehicle and the other vehicle, and scan-matches the own vehicle based on the three-dimensional point cloud information of the own vehicle and the other vehicle. Correct the relative position and relative posture of the other vehicle as seen from the vehicle,
A vehicle maneuvering method for generating the fused three-dimensional information by fusing the three-dimensional point cloud information of the other vehicle with the three-dimensional point cloud information of the own vehicle based on the corrected relative position and the relative posture. Is provided.

According to the present invention, computing device, using a three-dimensional point group information and the position and orientation information of the three-dimensional point group information and the position and orientation information and the vehicle received from the other vehicle, and the three-dimensional point group data of the vehicle Fusion 3D information is generated by fusing 3D point cloud information of other vehicles.
This fused three-dimensional information includes point cloud information in which common point cloud information of a plurality of three-dimensional point cloud information matches and does not common.
Therefore, even when the distance between the own vehicle and the other vehicle is large, it is possible to correct the error of the relative position / posture of the other vehicle as seen from the own vehicle and obtain a wide range and highly accurate surrounding environment information including blind spot information.

In FIGS. 1 and 2, the environment measuring device 12 of the own vehicle 10A is a camera, a laser range finder (LRF), a radar, and the like attached to the vehicle body 11.
The environment measuring device 12 of the own vehicle 10A acquires three-dimensional point cloud information (hereinafter, three-dimensional information 1A ) which is three-dimensional information of the discrete point cloud around the own vehicle 10A. Similarly, the environment measuring device 12 of the other vehicle 10B also acquires the three-dimensional point cloud information (hereinafter, three-dimensional information 1B ) which is the three-dimensional information of the discrete point cloud around the other vehicle 10B.
Hereinafter, the case where the three-dimensional information 1A is the point cloud information will be described.

Claims (15)

It is equipped with its own vehicle and another vehicle capable of bidirectional communication with the own vehicle.
The own vehicle and the other vehicle both have an environment measuring device for acquiring three-dimensional information of the surroundings, a self-positioning device for acquiring position / orientation information, a calculation device, and the three-dimensional information and the position / attitude information. It has an inter-vehicle communication device that communicates in the direction of the vehicle.
The arithmetic unit of the own vehicle further fuses the three-dimensional information of the own vehicle and the three-dimensional information of the other vehicle by using the three-dimensional information and the position / attitude information received from the other vehicle. Fusion A vehicle maneuvering system with an information fusion function that generates three-dimensional information. The information fusion function uses the three-dimensional information of the own vehicle and the other vehicle and the position / attitude information, and scans based on the point cloud information included in the three-dimensional information of the own vehicle and the other vehicle. A relative position / posture correction function that corrects the relative position and posture of the other vehicle as seen from the own vehicle by matching,
It has a fusion information generation function that generates the fusion three-dimensional information by fusing the three-dimensional information of the other vehicle with the three-dimensional information of the own vehicle based on the corrected relative position and the relative posture. , The vehicle maneuvering system according to claim 1. The vehicle maneuvering system according to claim 1, wherein the arithmetic unit of the own vehicle further has an ambient environment 3D image generation function for creating an ambient environment 3D image at an arbitrary viewpoint, gazing point, and distance from the fused three-dimensional information. .. The vehicle maneuvering system according to claim 1, wherein the own vehicle has an information sharing function of transmitting the fusion three-dimensional information to the other vehicle. The own vehicle is an unmanned vehicle that can be remotely controlled.
The vehicle control system according to claim 1, further comprising a remote control device for remotely controlling the unmanned vehicle. The vehicle control system according to claim 5, wherein the own vehicle further includes a remote communication device that transmits a 3D image of the surrounding environment at an arbitrary viewpoint, gaze point, and distance from the fusion three-dimensional information to the remote control device. The vehicle according to claim 6, wherein the remote control device includes a remote communication device that bidirectionally communicates with the own vehicle, a control device that performs information processing, and a display device that displays a 3D image of the surrounding environment. Maneuvering system. Prepare your own vehicle and another vehicle that can communicate bidirectionally with the own vehicle.
The own vehicle and the other vehicle acquire surrounding three-dimensional information and position / attitude information, respectively, and communicate the three-dimensional information and the position / attitude information in both directions.
Further, using the three-dimensional information and the position / attitude information received from the other vehicle, the three-dimensional information of the own vehicle and the three-dimensional information of the other vehicle are fused to generate fusion three-dimensional information. Vehicle maneuvering method. The own vehicle uses the three-dimensional information of the own vehicle and the other vehicle and the position / attitude information, and scan matching is performed based on the point group information included in the three-dimensional information of the own vehicle and the other vehicle. Corrects the relative position and relative posture of the other vehicle as seen from the own vehicle.
The vehicle according to claim 8, wherein the fused three-dimensional information is generated by fusing the three-dimensional information of the other vehicle with the three-dimensional information of the own vehicle based on the corrected relative position and the relative posture. Maneuvering method. The claim that the own vehicle searches for the other vehicle capable of bidirectional communication, and when there are a plurality of the searched other vehicles, communicates with a part or all of the searched other vehicles to generate the fusion three-dimensional information. 8. The vehicle maneuvering method according to 8. The vehicle maneuvering method according to claim 8, wherein the total area ratio of the flat area and the uneven area is adjusted with respect to the three-dimensional information of the own vehicle before the fusion three-dimensional information is generated. Regarding the three-dimensional information of the own vehicle, the sensor of the environment measuring device that acquires the measurement point and the three-dimensional information of the other vehicle on the surrounding environment information with respect to the measurement point of the own vehicle is used as an interference determination line. In the case where there is no shielding object between them and the interference determination line is included in the measurement range of the environment measurement device of the other vehicle, the measurement point is set as a common measurement point common to the own vehicle and the other vehicle. The vehicle maneuvering method according to claim 8, wherein common measurement points are scan-matched with each other. The vehicle maneuvering method according to claim 12, wherein the interference determination line is regarded as a cylindrical object and interference determination with a shielding object is performed. Regarding the three-dimensional information of the own vehicle, the measurement point obtained by removing the common measurement point from the measurement point of the own vehicle is set as a non-common measurement point, and the self is based on the relative position and orientation obtained by the scan matching. The vehicle maneuvering method according to claim 12, wherein the coordinates are converted into the coordinate system of the vehicle. Before generating the fusion 3D information, the 3D space is set as a set of grids of three-dimensional objects, and the measured 3D information is arranged.
The vehicle maneuvering method according to claim 8, wherein when one or a plurality of measurement points are included in the grid, the three-dimensional coordinates of the measurement points are averaged and filtering is performed using the three-dimensional coordinates of the grid as the three-dimensional coordinates.

Images