EP4613605A1

EP4613605A1 - Obstacle detecting system, and train equipped with same

Info

Publication number: EP4613605A1
Application number: EP23885498.8A
Authority: EP
Inventors: Seiji Imagawa
Original assignee: Hitachi Ltd
Current assignee: Hitachi Ltd
Priority date: 2022-11-04
Filing date: 2023-10-13
Publication date: 2025-09-10
Also published as: WO2024095736A1; JP2024067251A

Abstract

An object of the present invention is to provide a technique capable of continuously operating a vehicle even when malfunction occurs in some of a plurality of sensors.

A representative obstacle detection system of the present invention is an obstacle detection system including M sensors and signal processing units corresponding to the M sensors, the obstacle detection system including: an m-th sensor that detects an m-th detection range that is a range extending in a traveling direction that is a direction in which a vehicle travels toward the course; and a controller that controls the m-th sensor and the signal processing unit, in which the controller performs at least one change processing of first change processing of causing a first end to approach the vehicle and second change processing of causing a second end to move away from the vehicle, and changes at least a p-th detection range to change the obstacle detection range.

Description

Technical Field

The present invention relates to an obstacle detection system and a train on which the obstacle detection system is mounted.

Background Art

Conventionally, a technique for detecting an abnormality of an obstacle sensor used in an obstacle detection system for a vehicle is known.
PTL 1 discloses a driving assistance device including: a monitoring unit that monitors operation/non-operation of a sensor mountable on a vehicle; and an output unit that outputs information on the operation/non-operation monitored by the monitoring unit, in which the monitoring unit detects malfunction of the sensor on the basis of detection accuracy of the sensor input when the sensor is operating, and in which the output unit also outputs information on the malfunction together with the information on the operation/non-operation when the monitoring unit detects the malfunction of the sensor.

Citation List

Patent Literature

PTL 1: JP 2017-178267 A

Summary of Invention

Technical Problem

Here, in a case where the obstacle detection system is applied to a moving object such as a train that needs to have a long braking distance, it is conceivable to divide a detection section in a traveling direction of the moving object, allocate a sensor for each detection section, and secure a detection range up to a position away from the moving object. It is conceivable to substitute the detection range when malfunction of the sensor occurs, but PTL 1 does not sufficiently study such a viewpoint.
Therefore, an object of the present invention is to provide a technique capable of continuously operating a vehicle even when malfunction occurs in some of a plurality of sensors.

Solution to Problem

In order to solve the above problem, a representative obstacle detection system of the present invention is an obstacle detection system including M (M is a positive integer of 2 or more) sensors and signal processing units corresponding to the M sensors for detecting an obstacle in a course of a vehicle, the obstacle detection system including: an m-th (m is an integer of 1 or more and M or less) sensor that detects an m-th detection range that is a range extending in a traveling direction that is a direction in which the vehicle travels toward the course; and a controller that controls the m-th sensor and the signal processing unit, in which the signal processing unit performs signal processing of detecting an obstacle on a basis of an output signal of an m-th sensor, in which an obstacle detection range that is a range of the entire obstacle detection system is configured to include an m-th detection range, in which a distance from the (m + 1) th detection range to the vehicle is larger than a distance from an m-th detection range to the vehicle, and in which, when it is determined that malfunction has occurred in a n-th (n is an integer of 1 or more and M or less) sensor, on a basis of at least one of an n-th detection range of the n-th sensor, presence or absence of a human substitution means for obstacle detection, a traveling speed of the vehicle, and a traveling speed scheduled after a certain period of time or after a certain distance of travel of the vehicle, when an end of a p-th detection range of a p-th sensor (p is an integer of 1 or more and M or less other than n) closest to the vehicle in the p-th detection range is defined as a first end, and an end of the p-th detection range farthest from the vehicle is defined as a second end, the controller performs at least one change processing of first change processing of causing the first end to approach the vehicle and second change processing of causing the second end to move away from the vehicle, and changes at least the p-th detection range to change the obstacle detection range.

Advantageous Effects of Invention

According to the present invention, even when malfunction occurs in some of the plurality of sensors, it is possible to continuously operate the vehicle.
Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

Brief Description of Drawings

[FIG. 1] FIG. 1 is a diagram illustrating a schematic configuration of an obstacle detection system and a vehicle on which the obstacle detection system is mounted.
[FIG. 2] FIG. 2 is a diagram schematically illustrating an arrangement of a normal detection range of each sensor.
[FIG. 3] FIG. 3 is a diagram illustrating a detection range when malfunction occurs in a millimeter wave radar sensor.
[FIG. 4] FIG. 4 is a diagram illustrating a case where a detection range of a stereo camera sensor and a detection range of a monocular camera sensor are changed.
[FIG. 5] FIG. 5 is a diagram illustrating a case where a detection range of a LiDAR sensor becomes an undetectable range and the detection range of the stereo camera sensor is enlarged.
[FIG. 6] FIG. 6 is a diagram illustrating a case where the detection range of the monocular camera sensor becomes an undetectable range and the detection range of the stereo camera sensor is enlarged.
[FIG. 7] FIG. 7 is a diagram illustrating a case where the detection range of the stereo camera sensor becomes an undetectable range and the detection range of the monocular camera sensor is enlarged.
[FIG. 8] FIG. 8 is a diagram illustrating a configuration of a signal processing means of the stereo camera sensor.
[FIG. 9] FIG. 9 is a flowchart illustrating an operation performed by a controller of the obstacle detection system.
[FIG. 10] FIG. 10 is a diagram schematically illustrating an image on which cutting and compression processing has been performed by cutting and compression means.
[FIG. 11] FIG. 11 is a diagram schematically illustrating an image on which cutting and compression processing has been performed by the cutting and compression means in a case where the detection range is changed.
[FIG. 12] FIG. 12 is a diagram illustrating an angle of view of the stereo camera.
[FIG. 13] FIG. 13 is a diagram illustrating a case where a direction of one optical axis of the camera constituting the stereo camera is changed.

Description of Embodiments

Embodiments will be described below with reference to the drawings. Note that the present invention is not limited by these embodiments. In the drawings, the same portions are denoted by the same reference numerals.
In the present disclosure, the "detection range" indicates a range in which information such as a position and a speed of an obstacle can be acquired using a detection device such as a sensor. When a stereo camera is taken as an example of the detection device, the detection range is determined through functions of a camera constituting the stereo camera, a sensor that outputs a captured image as a signal, and a signal processing means that processes an output signal of the sensor and performs signal processing to identify an obstacle. Such a detection range is specified by the detection device, and is referred to as a "detection range of the stereo camera" or a "detection range of the sensor of the stereo camera".
In addition, "the detection range includes a range from the distance x1 to the distance x3" means that the detection range also includes x1 and x3 which are boundaries of the detection range.
In the present disclosure, the signal processing means may be a signal processing unit. The other means in the present disclosure are similar to the case of the signal processing means.

[First Embodiment]

In a first embodiment, an example of an obstacle detection system and a vehicle on which the obstacle detection system is mounted will be described. FIG. 1 is a diagram illustrating a schematic configuration of the obstacle detection system and the vehicle on which the obstacle detection system is mounted. First, each functional block will be described.

(Configuration)

FIG. 1 illustrates a case where an obstacle detection system is mounted on a vehicle. An obstacle detection system 100 detects an obstacle in the course of a vehicle 400. The vehicle 400 is, for example, a train. The obstacle detection system 100 mounted on the vehicle 400 includes a plurality of sensors 111 to 114. The left camera sensor 111 constitutes a left camera of a stereo camera sensor. The right camera sensor 112 constitutes a right camera of the stereo camera. The left camera sensor 111 and the right camera sensor 112 constitute the stereo camera sensor. The LiDAR sensor 113 constitutes light detection and ranging (LiDAR). The millimeter wave radar sensor 114 constitutes a millimeter wave radar. Signal processing means 121 to 124 perform signal processing of detecting an obstacle on the basis of an output signal of each of the sensors 111 to 114. That is, the signal processing means 121 performs signal processing of detecting an obstacle on the basis of the left and right camera images acquired by the left camera sensor 111 and the right camera sensor 112. A switch 130 selectively sends the image signal to a signal processing means 122. The signal processing means 122 performs signal processing of detecting an obstacle on the basis of a monocular camera image acquired by either the left camera sensor 111 or the right camera sensor 112. The signal processing means 123 performs signal processing of detecting an obstacle on the basis of an output signal of the LiDAR sensor 113. The signal processing means 124 performs signal processing of detecting an obstacle on the basis of an output signal of the millimeter wave radar sensor 114. The signal processing means 121 to 124 perform each signal processing using calculation resources of a memory 131 and a computing unit 132. A user interface 133 is controlled by a controller 140 and makes a notification to the crew member. The controller 140 controls the sensors 111 to 114 and the signal processing means 121 to 124. Other functions of the controller 140 will be described later.
Note that the left camera sensor 111 and the right camera sensor 112 constitute a stereo camera, and are also configured to function as a monocular camera by switching the switch 130. Such a configuration is an example, and the embodiment is not limited thereto. The sensor for the stereo camera and the sensor for the monocular camera may be separately provided.
For example, in a case where the obstacle detection system 100 is installed in the leading vehicle of the vehicle 400, the obstacle detection system 200 is installed in the last vehicle, and is used to detect an obstacle when the vehicle 400 performs a turnaround operation and travels in the opposite direction. The obstacle detection system 200 has components equivalent to those of the obstacle detection system 100. That is, the obstacle detection system 200 includes a plurality of sensors 211 to 214. The left camera sensor 211 constitutes a left camera of the stereo camera. The right camera sensor 212 constitutes a right camera of the stereo camera. The left camera sensor 111 and the right camera sensor 112 constitute the stereo camera sensor. A LiDAR sensor 213 constitutes light detection and ranging (LiDAR). The millimeter wave radar sensor 214 constitutes a millimeter wave radar. Signal processing means 121 to 124 perform signal processing of detecting an obstacle on the basis of an output signal of each of the sensors 111 to 114. That is, the signal processing means 221 performs signal processing of detecting an obstacle on the basis of the left and right camera images acquired by the left camera sensor 211 and the right camera sensor 212. A switch 230 selectively sends the image signal to the signal processing means 222. The signal processing means 222 performs signal processing of detecting an obstacle on the basis of a monocular camera image acquired by either the left camera sensor 111 or the right camera sensor 112. The signal processing means 223 performs signal processing of detecting an obstacle on the basis of an output signal of the LiDAR sensor 113. The signal processing means 224 performs signal processing of detecting an obstacle on the basis of the output of the millimeter wave radar sensor 214. The signal processing means 221 to 224 perform each signal processing using the calculation resources of the memory 231 and the computing unit 232. A user interface 233 is controlled by a controller 240 and makes a notification to the crew member. The controller 240 controls the sensors 211 to 214 and the signal processing means 221 to 224. Other functions of the controller 240 will be described later.
In addition, the vehicle controller 300 controls the obstacle detection system 100 and the obstacle detection system 200, or the vehicle 400 on which these systems are mounted. The vehicle 400 includes a communication means through which the controller 140 of the obstacle detection system 100 and the controller 240 of the obstacle detection system 200 exchange information.

(Relationship between Functions)

Next, a relationship between the blocks will be described. For example, in a case where the vehicle 400 is at one end of the vehicles included in the train, the sensors 111 to 114 included in the obstacle detection system 100 detect an obstacle present in the traveling direction and around the traveling direction when the train travels with the vehicle at the head. The traveling direction is, for example, a direction toward a track on which the train travels. The sensors 211 to 214 included in the obstacle detection system 200 are installed on the front surface of the vehicle at the other end opposite to the vehicle on which the sensors 111 to 114 included in the obstacle detection system 100 are installed, and detect obstacles ahead in the traveling direction and around when the train travels with the vehicle at the head. The vehicle 400 may not be included in the train. When the train includes only one vehicle, the sensors 111 to 114 of the obstacle detection system 100 may be installed on the front side of the vehicle, and the sensors 211 to 214 of the obstacle detection system 200 may be installed on the rear side of the vehicle.

(Details of each sensor)

The left camera sensor 111 and the right camera sensor 112 generate image signals indicating images captured in synchronization from different viewpoints, and send the generated image signals to the signal processing means 121. In addition, the image signal is sent to the switch 130, and the switch 130 selectively sends the image signal to the signal processing means 122. The LiDAR sensor 113 transmits a signal indicating scattered light detected by the LiDAR to the signal processing means 123. The millimeter wave radar sensor 114 transmits a signal indicating the electromagnetic wave detected by the millimeter wave radar to the signal processing means 124. The signal processing means 121 to 1124 perform signal processing such as noise removal, grouping, labeling, and tracking using the memory 131 and the computing unit 132 on the basis of each input signal, and sends object information including coordinates and speed of the detected object to the controller 140. The controller 140 identifies an obstacle on the basis of the object information transmitted from each signal processing means 121 to 124, detects the distance to the obstacle, and transmits the detected information to the user interface 133 and the vehicle controller 300. During a period in which the obstacle detection system 100 is operating, the operation may be stopped because the obstacle detection system 200 detects an obstacle in a place that the vehicle has already passed. However, when malfunction of a sensor to be described later occurs, it is necessary to grasp the state of both systems, and thus, a state in which communication is possible is maintained between the controller 140, the controller 240, and the vehicle controller 300.
In the obstacle detection system 100, the signal processing means 121 to 124 and the controller 140 may include a storage unit that accumulates an input signal sent by the sensors 111 to 114 and target object information detected from the input signal. By analyzing the change in the input signal, it is also possible to detect the orientation and the moving speed of the object. The obstacle detection system 200 may also include a storage unit.

(Detection range of sensor in obstacle detection system)

As described above, since the obstacle detection systems 100 and 200 include a plurality of sensors, it is possible to ensure robustness against environment and use conditions. On the other hand, since the sensors have different distance ranges and angles of view for obstacle detection which are their strengths depending on their characteristics, the detection ranges of the sensors are set so that an optimal detection range can be configured.

(Example of sensor allocation)

A normal detection range of each sensor will be described with reference to FIG. 2. FIG. 2 is a diagram schematically illustrating an arrangement of a normal detection range of each sensor. The sensors 111 to 114 of the obstacle detection system 100 are installed in the vehicle 400. Detection ranges of the respective sensors are indicated by arrows, and are a detection range 403 of the LiDAR sensor 113, a detection range 404 of the millimeter wave radar sensor 114, a detection range 401 of the stereo camera sensor (that is, the sensor including the left camera sensor 111 and the right camera sensor 112), and a detection range 402 of the monocular camera sensor (that is, one of the left camera sensor 111 and the right camera sensor 112). An obstacle detection range that is a detection range of the entire obstacle detection system 100 includes the detection ranges 401 to 404.
Here, in FIG. 2, a direction in which the vehicle 400 travels toward a course is indicated by an arrow as a traveling direction Td. The m-th (m is an integer of 1 or more and M or less, and M is a positive integer of 2 or more) sensor detects an m-th detection range that is a range extending in the traveling direction Td. The m-th signal processing means performs signal processing of detecting an obstacle on the basis of an output signal of an m-th sensor. Distances sequentially away from the vehicle 400 in the traveling direction Td are set as x1 (first distance), x2 (second distance), x3 (third distance),..., m-th distance (m is an integer of 1 or more and M or less, and M is a positive integer of 2 or more),.... The LiDAR sensor 113 (first sensor) detects a detection range 403 (first detection range) including at least a range from x1 to x3. In other words, in the detection range 403 of the LiDAR sensor 113, x1 is an end (first end) of the detection range 403 closest to the vehicle 400, and x3 is an end (second end) of the detection range 403 farthest from the vehicle 400. Similarly, the millimeter wave radar sensor 114 (second sensor) detects the detection range 404 (second detection range) including at least a range from x2 to x5. The stereo camera sensor (third sensor) detects the detection range 401 (third detection range) including at least a range from x4 to x7. The monocular camera sensor (fourth sensor) detects the detection range 402 (fourth detection range) including at least a range from x6 to x8. Although the obstacle detection system 100 is configured to include M sensors and M signal processing means (here, M=4), the configurations of the sensors and the signal processing means are not limited thereto. A configuration in which four or more sensors and signal processing means are provided or a configuration in which signal processing corresponding to M sensors is performed by one signal processing means is also possible.
As described above, the detection range can be arranged according to the characteristics of each sensor, and the detection range of the vehicle 400 can be configured as a whole. Here, the detection range 403 and the detection range 404 have an overlapping range from x2 to x3. It can be said that the detection range 403 and the detection range 404 are arranged in a so-called continuous manner. The detection range 404 and the detection range 401 have an overlapping range from x4 to x5. The detection range 401 and the detection range 402 have an overlapping range from x6 to x7. For example, when an obstacle is detected by the LiDAR sensor 113 but is not detected by the millimeter wave radar sensor 114 in an overlapping range from x2 to x3, it can be determined that malfunction has occurred in the millimeter wave radar sensor 114. By arranging the detection ranges continuously in this manner, it is possible to determine the malfunction of the sensor at an early stage. Since the obstacle detection systems 100 and 200 include a plurality of sensors, it is possible to ensure robustness against environment and use conditions.
Note that what is illustrated here is an example of the arrangement of the detection ranges, and the order of the detection ranges and the length of the detection distance are not limited thereto. Here, the detection ranges 401 to 404 include a range between the distances measured from the vehicle 400 in the traveling direction, but the detection ranges 401 to 404 also extend in a direction other than the traveling direction. For example, in a case where the traveling direction Td is the X-axis direction, the detection ranges 401 to 404 extend to a certain degree also in the ±Y direction and the ±Z direction.

(Countermeasures against malfunction of some sensors)

When determining that malfunction has occurred in the sensor, the controller 140 considers at least one of a detection range of the sensor in which the malfunction has occurred, presence or absence of a human substitution means for obstacle detection, a traveling speed of the vehicle 400, and a traveling speed scheduled after a certain period of time or after a certain distance of traveling of the vehicle 400. For example, the traveling speed of the vehicle is limited by the detection range of the obstacle. That is, until the vehicle stops, a stop distance obtained by adding an idle running distance where the brake is not applied and a braking distance where the brake is applied is required. Here, since the deceleration of the vehicle has a predetermined upper limit value, there is a limit to shortening the braking distance. In addition, there are limitations on the hardware and software of the sensor in order to increase the detection range. Therefore, in order to stop the vehicle before reaching the obstacle, it is necessary to limit the traveling speed of the vehicle on the basis of the detection range of the obstacle. In FIG. 2, the upper limit value of the traveling speed of the vehicle 400 is set on the basis of the farthest x8 in the detection range. Note that human substitution means will be described later.
As the detection range becomes smaller, it is necessary to limit the traveling speed in order to suppress the stop distance. Therefore, in the first embodiment, by changing the detection range of the normally functioning sensor, the detection range is prevented from being reduced. Accordingly, it is possible to suppress a decrease in traveling speed.

(When intermediate-distance sensor malfunctions)

With reference to FIGS. 3 to 6, changes in the detection ranges of other normal sensors when any one of the sensors malfunctions will be described. FIG. 3 is a diagram illustrating a detection range when malfunction occurs in the millimeter wave radar sensor 114. When malfunction occurs in the millimeter wave radar sensor 114, an undetectable range 409 in which an obstacle cannot be detected occurs between the detection range 403 of the LiDAR sensor 113 and the detection range 401 of the stereo camera sensor. Since the vehicle 400 is moving, the undetectable range 409 is once a range detected by the monocular camera sensor or the stereo camera sensor, but there is a possibility that an obstacle enters the area of the vehicle after the absence of the obstacle is confirmed. Therefore, in this case, the range in which the obstacle detection system 100 can guarantee the presence or absence of an obstacle is equivalent to the detection range 403 by the LiDAR sensor 113, and the upper limit value of the traveling speed of the vehicle is set on the basis of x3 that is the boundary of the detection range 403. That is, when determining that malfunction has occurred in the millimeter wave radar sensor 114, the controller 140 sets the upper limit value of the traveling speed of the vehicle 400 on the basis of the distance x3 from the vehicle 400 to the boundary of the detection range 403.
Therefore, measures are taken to eliminate the undetectable range. FIG. 4 is a diagram illustrating a case where the detection range of the stereo camera sensor and the detection range of the monocular camera sensor are changed. Here, when determining that malfunction has occurred in the n-th sensor (n is an integer of 1 or more and M or less), the controller 140 performs change processing of changing the p-th detection range of the p-th sensor (p is an integer of 1 or more and M or less excluding n) to include at least a part of the n-th detection range. That is, change processing (first change processing) is performed to bring the end x4 of the detection range 401 closest to the train 400 closer to the train 400, and the detection range 401 is changed to a detection range 401a. As a result, the detection range 401 of the stereo camera sensor is enlarged as a whole. Furthermore, in a case where the distance xi1 between x5 and x6 is set, the detection range 401a after the change of the stereo camera sensor includes a range from x2 to xi1. As a result, the detection range 401a includes the detection range 404 (range from x2 to x5) of the millimeter wave radar sensor 114.
Note that, since the detection range of the stereo camera sensor has changed in a direction approaching the vehicle 400, the change processing is performed by the monocular camera sensor so as to compensate for the detection range of the stereo camera sensor. Here, in a case where the distance between x7 and x8 is xi2, the detection range 402a after the change of the monocular camera sensor includes a range from x5 to xi2. With respect to the obstacle detection range that is the detection range of the entire obstacle detection system 100, the obstacle detection range after the change processing includes the detection range 403, the detection range 401a, and the detection range 402a that are continuously arranged.
By changing the detection range 401a of the stereo camera sensor and the detection range 402a of the monocular camera sensor in this manner, after the undetectable range 409 is eliminated by the detection range 401a, the detection range of the obstacle detection system 100 as a whole can be arranged continuously over the distance xi2 from the vehicle 400 to the boundary of the detection range 402a. In addition, the controller 140 sets the upper limit value of the traveling speed of the vehicle 400 on the basis of the detection range after the change processing is performed. Since the boundary of the detection range of the obstacle detection system 100 as a whole can be narrowed only from x8 to xi2, the limit of the upper limit value of the traveling speed of the vehicle 400 can be minimized. Note that, as a method of shortening the obstacle detection range by the stereo camera, for example, the image may be compressed according to the degree of shortening. In the method of extending the detection range, after the magnification of the image is increased, the detection based on the super-resolved image is performed in the case of corresponding in the image processing, and the geometric correction of the stereo camera is performed as necessary in the case of corresponding in the optical system such as the lens. A method of changing the detection range of the stereo camera sensor will be described later in detail.
Note that the detection regions of the stereo camera sensor and the monocular camera sensor in FIG. 4 are indicated by two arrows to clearly indicate that the detection range is larger than that in FIG. 2. This indicates that since the operation of the signal processing means 124 of the millimeter wave radar sensor 114 becomes unnecessary, the calculation resources of the memory 131 and the computing unit 132 used by the signal processing means 124 are allocated to and used by the signal processing means 121 and 122 of the stereo camera and the monocular camera, and it becomes possible to detect an obstacle in a wider range. Such allocation of the calculation resources can be performed by the controller 140, for example. For example, the signal processing means 121 to 124 determine that the detection by the sensor is disabled, send a flag signal indicating that the detection is disabled to the controller 140, and stop and release the use of the calculation resources. When detecting the flag signal from the signal processing means 121 to 124, the controller 140 determines that the malfunction has occurred in the sensor corresponding to the flag signal, and preferentially allocates the calculation resource allocated to the sensor for which the malfunction is determined to have occurred to the sensor to be subjected to the change processing and the signal processing means of the sensor.

(When short-distance sensor malfunctions)

FIG. 5 is a diagram illustrating a case where the detection range 403 of the LiDAR sensor 113 becomes an undetectable range and the detection range of the stereo camera sensor is enlarged. FIG. 5 illustrates a case where malfunction has occurred in the LiDAR sensor 113, and a detection range 401b of the stereo camera sensor compensates for the detection range 401 including the undetectable range of the LiDAR sensor 113, that is, the range from x1 to x3. The operation of the stereo camera in this case is illustrated in FIG. 8.
Note that the detection range 401b from x1 to x3 and the detection range 401 from x4 to x7 are allocated to the stereo camera sensor, while the range from x3 to x4 is excluded from the detection range of the stereo camera sensor. Although the calculation resource allocated to the malfunctioning LiDAR sensor 113 is also allocated to the stereo camera sensor, the example illustrated in FIG. 5 indicates that sufficient calculation resources for setting the entire detection range from x1 to x7 are not allocated to the stereo camera sensor. Since the processing load increases when the stereo camera sensor detects the range from x3 to x4, the detection range of the stereo camera sensor is limited to the detection range 401b and the detection range 401.
FIG. 8 is a diagram illustrating a configuration of the signal processing means 121 of the stereo camera sensor. In a normal case, the operation of the stereo camera geometrically corrects the image signal Sl based on the left camera and the image signal Sr based on the right camera using the geometric correction means 1211 and 1212, and cuts out and appropriately compresses a necessary area of the corrected image subjected to the geometric correction using cutting and compression means 1213 and 1214. Thereafter, a parallax image is calculated using the left-right cut-out image using a stereo matching means 1215, and the distance to the obstacle is detected by an obstacle detection means 1216 using the parallax image.
On the other hand, in a case where the detection range 401b in the vicinity of the vehicle is detected by the stereo camera sensor as a substitution to the detection range 401 of the LiDAR sensor 113, a necessary area of the corrected image is cut out using cutting and compression means 1217 and 1218, and the image is compressed at a compression rate higher than the compression rate in the normal case. This is processing of detecting a shorter distance than the normal detection range 401. Thereafter, the stereo matching means 1219 and the obstacle detection means 1220 detect an obstacle in a short distance, and transmit obstacle detection information to the controller 140 in the subsequent stage.
Note that, in a case where the resource for compensating for the undetectable range of the LiDAR sensor 113 is insufficient in addition to the detection range of the conventional stereo camera sensor, priority is given to detecting a substitution range of the undetectable range of the LiDAR sensor 113. This is to prevent the undetectable range from remaining in the distance x2 from the vehicle 400 to the boundary of the detection range 404 of the millimeter wave radar sensor. Note that, in a case where the signal processing resources of the normal stereo camera are insufficient and the detection range 401 of the stereo camera is shortened and an undetectable range occurs between x4 and x7, the undetectable range can be compensated by changing the boundary x6 of the detection range 402 of the monocular camera sensor to a point on the vehicle 400 side.

(When long-distance sensor malfunctions)

FIG. 6 is a diagram illustrating a case where the detection range 402 of the monocular camera sensor becomes the undetectable range and the detection range 401 of the stereo camera sensor is enlarged. FIG. 6 illustrates a case where malfunction has occurred in the monocular camera sensor. A detection range 401c after the change of the stereo camera sensor is extended farther than the normal detection range 401. That is, change processing (second change processing) is performed such that the end x7 farthest from the train 400 in the detection range 401 is moved away from the train 400, and the detection range 401 is changed to the detection range 401c. As a result, the detection range 401 of the stereo camera sensor is enlarged as a whole. That is, in a case where the distance between x7 and x8 is set to xi3, the detection range 401c after the change of the stereo camera sensor includes a range from x4 to xi3. This is a result of enlarging the detection range of the stereo camera sensor by the signal processing means 121 of the stereo camera sensor using the resource used by the signal processing means 122 of the monocular camera. In this manner, the detection range 401c of the stereo camera can compensate for a part of the detection range 402 of the monocular camera in the undetectable range. Although the entire region of the detection range 402 of the monocular camera is not compensated, the limit of the traveling speed can be minimized because the boundary of the detection range of the obstacle detection system 100 as a whole can be kept from x8 to xi3.

(When stereo camera malfunctions)

FIG. 7 is a diagram illustrating a case where the detection range 401 of the stereo camera sensor becomes the undetectable range and the detection range of the monocular camera sensor is enlarged. FIG. 7 illustrates a case where the stereo camera including the left camera sensor 111 and the right camera sensor 112 malfunctions. The detection range 402a of the monocular camera sensor is enlarged to compensate for the range that is the detection range 401 of the stereo camera sensor. That is, in a case where the distance between x7 and x8 is xi4, the changed detection range 402a of the monocular camera sensor includes a range from x4 to xi4. The signal processing means 122 of the monocular camera sensor uses the calculation resource used by the signal processing means 121 of the stereo camera, thereby enlarging the detection range of the monocular camera and compensating for a part of the undetectable range. In the first embodiment, since the input image of the monocular camera is used in combination with one of the left and right images of the stereo camera, when the malfunction of the stereo camera is caused by the camera unit, the input of the signal processing means 122 is switched by the switch 130. This enables the switch 130 to switch to the signal processing means 122 when an abnormality is detected in at least one of the left camera sensor 111 and the right camera sensor 112. Alternatively, the signal processing means 121 may send a flag for notifying that the stereo camera cannot detect an obstacle to the controller 140, and the controller 140 may issue an instruction to switch the switch. Note that the present invention is not limited thereto when a camera and a sensor dedicated to a monocular camera are provided.

(Overview of change of detection range)

As described above, when malfunction occurs in some sensors, the detection range is changed so as to compensate for the range in which another normal sensor becomes undetectable. That is, the detection range of each sensor is changed so that the detection range of the obstacle detection system 100 as a whole is the detection range continuously from the vehicle 400 to as far as possible. This is not merely a change in the detection range of the system due to the stop of the function of some sensors, but is a change in which a range different from the detection range that is usually arranged is set as a new detection range in order to compensate for the undetectable range, and is also a change in which at least a part of the undetectable range is compensated.
By performing such change processing, it is possible to continuously operate the vehicle even when malfunction occurs in some of the plurality of sensors.

(Handover to crew member)

The controller 140 recognizes whether or not an obstacle can be detected by a human resource (such as a crew member of the train 400). In a case where it is determined that at least one of the M sensors is malfunctioning when the obstacle can be detected by the human resources, the controller 140 determines whether or not a malfunction range can be compensated by a sound sensor other than the obstacle detection by the human resource, and whether or not the malfunction range falls within an acceptable range as actual harm. As a result, when the obstacle detection by the human resource is necessary, the controller 140 outputs information indicating that the handover for passing the operation and range of the obstacle detection to the human resource is performed. Here, information indicating that a handover for passing the operation of the obstacle detection and the range thereof to the human resource is performed is output, and the handover is performed after information indicating that the handover is possible is output by the human resource, or switching of the handover is performed by the human resource to recognize the execution of the handover, and the human detection range is transmitted to the human resource via a user interface such as a monitor or a head mount display (HMD).
Here, when the controller 140 recognizes that the obstacle detection in the undetectable range is possible by the obstacle detection by the crew member of the train, it is not necessary to change the detection ranges of the other sensors. When determining that the malfunction has occurred in any of the sensors, the controller 140 determines whether the detection range of the sensor in which the malfunction has occurred is a range that can be substituted (handover) by the crew member of the vehicle. As a criterion for the determination, for example, whether or not the detection range is at a position where a visual distance is possible is used as a criterion. Taking the case of FIG. 3 as an example, if the crew member can immediately substitute the detection range of the millimeter wave radar sensor 114, the obstacle detection is handed over to the crew member, and the normal sensor operates the system without changing the detection range. The controller 140 of the obstacle detection system 100 may share information with the controller 240 and the vehicle controller 300 to grasp in advance whether or not there is a crew member capable of substituting obstacle detection. The controller 140 indicates the human detection range to the crew member of the vehicle 400 via the user interface 133 when substitution is possible. The controller 140 performs the handover related to the obstacle detection after receiving the information indicating that the substitution is possible from the crew member via the user interface 133. In order to reliably take over the monitoring, the handover trigger is desirably issued by a crew member that performs substitution. The human detection range is not limited to the detection range of the sensor in which the malfunction occurs. The obstacle detection range after the correction processing also includes a range in which the crew member is in charge of obstacle detection. In addition, since the visible distance is limited, it is also assumed that another sensor A covers the failed range and the crew member covers the range of the sensor A. For example, the obstacle detection range after the change processing includes a human detection range by visual observation of human resources.
At this time, it is important to more specifically indicate the monitoring range to the crew member assigned to detect the obstacle. For example, the user interface 133 includes a monitor or an HMD, and notifies the crew member of the detection range by a so-called AR technology in which the detection range assigned to the crew member is clearly superimposed on a real-time front image. In addition, it is also possible to provide a head-up display function on the windshield of the vehicle and project the detection range in a windshield shape.

(Operation from occurrence of malfunction of sensor to change of detection range)

(Detection and recovery from malfunction)

Determination of the malfunction and an operation procedure after the determination will be described with reference to FIG. 9. FIG. 9 is a flowchart illustrating an operation performed by the controller 140 of the obstacle detection system 100. The controller 140 determines whether malfunction has occurred (step 901). The malfunction can be determined by the sensors 111 to 114 and the signal processing means 121 to 124, or can be determined by hardware or software which is not described. When the malfunction is determined by any one of the methods (yes in Step 901), the controller 140 attempts to recover the place where the malfunction has occurred (Step 902). When the malfunction does not occur (no in step 901), the controller 140 continues the obstacle detection.
Due to the cause of the malfunction, there are a situation in which immediate recovery from the malfunction is possible and a situation in which the immediate recovery from the malfunction is impossible. For example, the former is a situation in which an object that blocks the field of view adheres to a windowpane in front of the camera, a situation in which a headlight or a wiper is left on, or the like. Since there is a case where the cause of the temporary malfunction can be removed by the crew member, the controller 140 issues a warning by voice or screen display via the user interface 133, and when it is detected that an operation for urging the cause removal has been performed, it is assumed that the recovery is successful.

(Determination based on vehicle situation)

When it is determined that the recovery is impossible, for example, when a certain period of time elapses without detecting that an operation for recovery is performed (step 903), the controller 140 formulates a rearrangement plan of the remaining detection ranges of the normal sensors. In this case, the controller 140 considers, for example, a detection range of a malfunctioning sensor, presence or absence of a crew member covering the detection range, and the like. In addition, the controller 140 calculates an upper limit value of the traveling speed of the vehicle on the basis of the rearrangement plan (step 904). This is because the operation schedule of the other vehicle is also adjusted on the basis of the traveling speed of the vehicle 400, but there is an influence on the operation schedule when the vehicle 400 continues traveling in a state in which the upper limit value of the traveling speed is exceeded.
Note that information such as the presence or absence of a crew member, the maximum speed of the vehicle on the track, the current position, map information, and the presence or absence of a crew member may be acquired by the vehicle controller 300 in advance or from the outside, and transmitted to the controllers 140 and 240.

(Determination of traveling direction)

Subsequently, the controller 140 determines the traveling direction (step 905). Usually, when malfunction is detected in the obstacle detection system, it is conceivable that a crew member of the vehicle performs substitution and continues normal operation. However, in a case where the detection range is rearranged, it is sufficient that the detection range can be compensated by visual observation by the crew member of the vehicle, but there is a possibility that an undetectable range that cannot be visually detected remains. In addition, in a case where the traveling speed is limited and there is a high possibility that the operation of other vehicles is hindered, it is necessary to evacuate the vehicle and to promptly evacuate the passengers.
In addition, in a case where the obstacle detection system is applied to an automatic driving system that realizes driverless or license-less (license-free) in which a crew member does not operate, when malfunction is detected in the obstacle detection system, continuation of operation becomes difficult. Therefore, it is necessary to cause passengers to safely get off and then evacuate the vehicle from the main line so as not to hinder other services. In a case where the traveling speed of the vehicle is limited and the vehicle cannot travel, not only convenience of passengers is impaired but also disturbance of the subsequent operation schedule is widened.
In consideration of these, the controller 140 determines the traveling direction in consideration of the arrival time to the front and rear nearest stations, the presence or absence of a crew member at the front and rear nearest stations, and the influence on the operation schedule on the basis of map information such as the upper limit value of the traveling speed, the current location, and the distance to the front and rear stations, and information such as the presence or absence of a passenger, the checking result of the soundness of the obstacle detection system 200 at the rear, and the position where assistance by human resources can be obtained. When the traveling direction is not changed (no in step 905), the controller 140 compares the calculated upper limit value of the traveling speed with the current actual speed, and decelerates the speed when the actual speed is higher (yes in step 906). When the deceleration ends after the actual speed becomes lower than the upper limit value of the traveling speed (yes in step 907), the controller 140 performs change processing on the detection range of each sensor on the basis of the proposed rearrangement plan (step 908).
On the other hand, when the traveling direction is changed (yes in step 905), the controller 140 safely stops the vehicle 400, and then starts traveling using the obstacle detection system 200 (step 909). When arriving at a target station, passengers are caused to get off or an assistant crew member is caused to get on. The controller 140 again formulates and executes an arrangement plan of the detection range. When the arrangement plan is formulated, the vehicle may be moved to the final repair shop, or may be temporarily moved to the evacuation place in order to suppress the influence on the operation schedule as much as possible, and may be moved to the final repair shop or the garage after business.
When the obstacle detection system is applied to a driverless or license-free vehicle in this manner, it is possible to cause passengers to promptly get off and evacuate the vehicle so as not to hinder other operations.
In addition, since the traveling speed is limited by the change processing on the detection range, it is desirable to avoid changing the detection range more than necessary. The operation illustrated in FIG. 9 is periodically performed while the vehicle 400 is in operation. By performing such an operation, it is possible to reliably determine the occurrence of the malfunction and the situation in which the change processing on the detection range is necessary, and to avoid performing the change processing on the detection range more than necessary.

(Details of stereo camera sensor change processing)

(When detection range is changed)

Here, details of the change processing on the detection range in the stereo camera will be described with reference to FIGS. 10 to 13. FIG. 10 is a diagram schematically illustrating an image on which cutting and compression processing has been performed by the cutting and compression means 1213 and 1214. FIG. 11 is a diagram schematically illustrating an image on which cutting and compression processing has been performed by the cutting and compression means 1217 and 1218 in a case where the detection range is changed.
FIG. 10 is an image after the input images of the cutting and compression means 1213 and the cutting and compression means 1214 in FIG. 8 are appropriately cut. Here, since the parallax obtained from the left and right images increases as the object is closer, the search range of stereo matching may be widened in order to make the detection range a short distance. However, simply extending the search range increases the processing load and the memory to be used. On the other hand, a closer object has a smaller influence on the distance error with respect to the parallax error. Therefore, in a case where the detection range is changed and expanded, a method of compressing an image as illustrated in FIG. 11 is appropriate for the detection range in the vicinity. This is performed by the cutting and compression means 1217 and the cutting and compression means 1218. An arithmetic expression for obtaining the object distance from the parallax is expressed as the following (Equation 1).
[Math. 1] $L = d \cdot pp / fc \cdot B$
Here, L is the object distance, d is the parallax of the object, pp is the pixel pitch, fc is the focal length of the camera lens, and B is the base length of the stereo camera. Here, since the compression reduces the image size, the number of pixels is reduced and the pixel pitch is increased. For example, since the parallax obtained from an image compressed to 1/2 is d/2 and the pixel pitch considering compression is 2 · pp, the obtained object distance L is equal in the case of the following (Equation 2) .
[Math. 2] $L = (d / 2) \cdot (2 \cdot pp) / fc \cdot B$
Therefore, if the obstacle detection means 1220 grasps the compression rate of the parallax image sent from the stereo matching means 1219, the object distance can be appropriately obtained. For example, the object distance can be obtained by determining the compression rate when the controller 140 rearranges the detection range and performing signal processing in the signal processing means 121 on the basis of the determined compression rate.

(Nicking control of stereo camera)

The correction of inclining the optical axis of the stereo camera will be described with reference to FIGS. 12 and 13. FIG. 12 is a diagram illustrating an angle of view of the stereo camera. FIG. 13 is a diagram illustrating a case where the direction of one optical axis of the camera constituting the stereo camera is changed. The controller 140 changes the detection range by performing at least one of changing the magnification of the acquired image of the left and right cameras, changing the stereo matching search range, and rotating the optical axis of at least one of the left and right cameras inward in the yaw direction.
In order to perform the detection by the stereo camera, the three-dimensional object needs to appear in the left and right images. When the extreme vicinity is detected, the three-dimensional object does not enter the angle of view of any image. FIG. 12 illustrates a left camera corresponding to the left camera sensor 111 and a right camera corresponding to the right camera sensor 112. The left camera and the right camera have an angle of view fov. The optical axis of the left camera is represented by oa1, and the optical axis of the right camera is represented by oa2. The optical axes of the left and right cameras are generally installed in parallel as illustrated in FIG. 12, but the optical axis of one of the left and right cameras may be directed inward as illustrated in FIG. 13. In this case, there is a possibility that detection of a distant place becomes difficult, but this is effective in a case where detection of a nearby object is prioritized. In a case where the optical axis is directed inward, the geometric correction means 1211 or the geometric correction means 1212 in FIG. 8 needs to correct the image from the rotation angle, the rotation axis, and the position of the imaging surface in addition to the normal geometric correction. At this time, the correction calculation can be simplified by matching the rotation axis with the entrance pupil position of the camera lens.

(Modification of change of detection range)

After the detection range is changed, the detection performance and the ranging performance of the sensor may be simply verified. The controller 140 checks the detection performance in the detection range after the change using the sensor including the first end changed after the first change processing in the detection range or the sensor including the second end changed after the second change processing in the detection range. For example, by overlapping the detection range after the change with the detection range of another normal sensor as illustrated in FIGS. 4 and 5, the object detection performance and the detection distance performance can be evaluated in the overlapping range. For example, in FIG. 4, the controller 140 evaluates the detection performance and the detected distance performance of the detection range 401a after the change processing by the LiDAR sensor 113 corresponding to the detection range 403 and the monocular camera sensor corresponding to the detection range 402a.

(Method of determining malfunction)

An example of a method for determining occurrence of malfunction in a stereo camera or a monocular camera will be described. In a case where the frame rate of the camera is constant, when the output signal sent from the camera is not updated for a certain period of time, there is a possibility that malfunction has occurred in the camera body, the camera cable, or the like.
In addition, the controller 140 estimates the cause of the malfunction of the n-th sensor on the basis of statistical information such as a luminance average value, a luminance variance value, and a luminance histogram for the entire image or each area or a difference in statistical information for each area. For example, when the luminance is significantly lower than the luminance information of the acquired image as a cause of the malfunction of the stereo camera, there is a possibility that there is a shielding object in front of the camera lens. When the luminance variance value of the entire screen is small even if the luminance is an appropriate value, there is a possibility of snowstorm or backlight. In a case where the difference from the adjacent pixel is small or the signal level of the spatial frequency of the high frequency is small, there is a possibility that the wiper is not operating despite heavy rain. As described above, the controller 140 or the stereo camera sensor may analyze the image included in the output signal of the stereo camera sensor, estimate the failure factor of the malfunction, and instruct the crew member of the vehicle 400 to take a measure to solve the cause.

(Case where malfunction has occurred in stereo camera sensor)

Malfunction may occur in the stereo camera sensor. For example, it is assumed that the left camera sensor 111 and the right camera sensor 112 constitute a stereo camera sensor, and the signal processing means 121 performs obstacle detection by a monocular image using the left camera sensor 111 as an input. Here, in a case where malfunction has occurred in the left camera sensor 111, the controller 140 causes the right camera sensor 112 to perform obstacle detection by a monocular image as an input. Specifically, the controller 140 specifies a sensor in which malfunction occurs among the left camera sensor 111 and the right camera sensor 112 constituting the stereo camera. In addition, it is assumed that the signal processing means 122 performs obstacle detection using a monocular image such as AI. In a case where it is found that one sensor malfunctions and the other sensor does not malfunction, the controller 140 switches the switch 130 so that the sensor in which the malfunction does not occur is used as an input of the signal processing means 122.

(Operation and effect)

As described above, in the present disclosure, it is possible to continuously drive the vehicle while suppressing the restriction of the traveling speed by changing the detection range of the sensor that normally functions. According to the present disclosure, it is possible to continuously drive a vehicle even when malfunction occurs in some of a plurality of sensors.
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.
For example, in the first embodiment, the obstacle detection systems 100 and 200 are provided in the vehicle, but the present invention is not limited to in-vehicle use. For example, only the sensors 111 to 114 and the sensors 211 to 214 may be installed in a vehicle, and the other signal processing means, controller, memory, and CPU may be provided outside the train.

[Other Embodiments]

The present disclosure also includes the following aspects.

(Aspect 1)

An obstacle detection system including M (M is a positive integer of 2 or more) sensors and signal processing units corresponding to the M sensors for detecting an obstacle in a course of a vehicle, the obstacle detection system including:

an m-th (m is an integer of 1 or more and M or less) sensor that detects an m-th detection range that is a range extending in a traveling direction that is a direction in which the vehicle travels toward the course; and
a controller that controls the m-th sensor and the signal processing unit,
in which the signal processing unit performs signal processing of detecting an obstacle on a basis of an output signal of an m-th sensor,
in which an obstacle detection range that is a range of the entire obstacle detection system is configured to include an m-th detection range,
in which a distance from the (m + 1) th detection range to the vehicle is larger than a distance from an m-th detection range to the vehicle, and
in which, when it is determined that malfunction has occurred in a n-th (n is an integer of 1 or more and M or less) sensor,
on a basis of at least one of an n-th detection range of the n-th sensor, presence or absence of a human substitution means for obstacle detection, a traveling speed of the vehicle, and a traveling speed scheduled after a certain period of time or after a certain distance of travel of the vehicle,
when an end of a p-th detection range of a p-th sensor (p is an integer of 1 or more and M or less other than n) closest to the vehicle in the p-th detection range is defined as a first end, and an end of the p-th detection range farthest from the vehicle is defined as a second end,
the controller performs at least one change processing of first change processing of causing the first end to approach the vehicle and second change processing of causing the second end to move away from the vehicle, and changes at least the p-th detection range to change the obstacle detection range.

(Aspect 2)

The obstacle detection system according to Aspect 1, in which the p-th detection range of the p-th sensor which is subjected to the change processing includes at least a part of the n-th detection range.

(Aspect 3)

The obstacle detection system according to Aspect 1 or 2, in which the obstacle detection range after the change processing has detection ranges arranged continuously.

(Aspect 4)

The obstacle detection system according to any one of Aspects 1 to 3, in which the obstacle detection range after the change processing includes a human detection range by visual observation.

(Aspect 5)

The obstacle detection system according to any one of Aspects 1 to 4, in which the controller sets an upper limit value of a traveling speed of the vehicle on a basis of the obstacle detection range in which the change processing is scheduled to be performed.

(Aspect 6)

The obstacle detection system according to any one of Aspects 1 to 5, in which the controller performs the change processing after an actual speed of the vehicle becomes lower than an upper limit value of the traveling speed.

(Aspect 7)

The obstacle detection system according to any one of Aspects 1 to 6, further including a calculation resource used when the signal processing unit performs the signal processing,
in which the controller allocates the calculation resource allocated to signal processing of the n-th sensor in which malfunction is determined to occur to signal processing of the p-th sensor to be subjected to the change processing.

(Aspect 8)

The obstacle detection system according to any one of Aspects 1 to 7,

in which the sensor on which the change processing is performed includes a stereo camera sensor including left and right cameras, and
in which the controller changes a detection range by performing at least one of changing a magnification of an acquired image of the left and right cameras, changing a search range of stereo matching, and rotating at least one optical axis of the left and right cameras inward in a yaw direction.

(Aspect 9)

The obstacle detection system according to any one of Aspects 1 to 8, in which the controller checks detection performance in the changed p-th detection range using a sensor including the first end changed after the first change processing in a detection range or a sensor including the second end changed after the second change processing in a detection range.

(Aspect 10)

The obstacle detection system according to any one of Aspects 1 to 9,

in which the controller recognizes whether or not an obstacle can be detected by human resources,
when it is determined that at least one of the M sensors is malfunctioning when the obstacle can be detected by the human resources,
the controller outputs information indicating that a handover that passes an operation and a range of obstacle detection to a human resource is performed,
the controller performs a handover after outputting information indicating that the handover is possible by the human resource, or performs handover switching by the human resource and recognizes execution of the handover, and
the controller communicates a human detection range to the human resource via a user interface, such as a monitor or a head mount display.

(Aspect 11)

The obstacle detection system according to any one of Aspects 1 to 10, in which the controller determines that malfunction has occurred when an output signal of the n-th sensor is not updated for a certain period of time.

(Aspect 12)

The obstacle detection system according to any one of Aspects 1 to 11, in which the controller

estimates a cause of malfunction of the n-th sensor on a basis of a difference between statistical information such as a luminance average value, a luminance variance value, and a luminance histogram for an entire image or each area and statistical information for each area, and
instructs a crew member of the vehicle to take a measure to solve the cause.

(Aspect 13)

The obstacle detection system according to any one of Aspects 1 to 12,

in which the n-th sensor and the q-th sensor (q is an integer of 1 or more and M or less other than n) constitute a stereo camera sensor, and
in which, when the signal processing unit receives the n-th sensor as an input and performs obstacle detection using a monocular image, the controller switches to the monocular image having the q-th sensor as an input to perform obstacle detection.

(Aspect 14)

A train including a first obstacle detection system and a second obstacle detection system that are the obstacle detection system according to any one of aspects 1 to 13,

in which the first obstacle detection system is provided in a leading vehicle of the train, and the second obstacle detection system is provided in a last vehicle of the vehicle, and
in which the first obstacle detection system detects an obstacle when the train travels in the traveling direction, and the second obstacle detection system detects an obstacle when the train travels in a backward direction opposite to the traveling direction,
the train including a communication means through which a controller of the first obstacle detection system and a controller of the second obstacle detection system exchange information,
in which, when malfunction is detected in the first obstacle detection system,
the controller of the first obstacle detection system checks soundness of the second obstacle detection system, and determines whether or not to change a traveling direction or a position to change the traveling direction on a basis of at least one of current position information of the vehicle, route map information, presence or absence of a passenger, a position where assistance by human resources is obtained, and an upper limit value of a traveling speed calculated by the controller of the first obstacle detection system, in addition to information on the checked soundness.

Reference Signs List

100, 200 obstacle detection system
111, 211 left camera sensor
112, 212 right camera sensor
113 , 213 LiDAR sensor
114, 214 millimeter wave radar sensor
121 to 124, 221 to 224 signal processing means
130, 230 switch
131, 231 memory
132, 232 computing unit
133, 233 user interface
140, 240 controller
300 vehicle controller
400 vehicle
401 to 404, 401a, 401b, 401c, 402a detection range
1211, 1212 geometric correction means
1213, 1214, 1217, 1218 cutting and compression means
1215, 1219 stereo matching means
1216, 1220 obstacle detection means

Claims

An obstacle detection system including M (M is a positive integer of 2 or more) sensors and signal processing units corresponding to the M sensors for detecting an obstacle in a course of a vehicle, the obstacle detection system comprising:
an m-th (m is an integer of 1 or more and M or less) sensor that detects an m-th detection range that is a range extending in a traveling direction that is a direction in which the vehicle travels toward the course; and

a controller that controls the m-th sensor and the signal processing unit,

wherein the signal processing unit performs signal processing of detecting an obstacle on a basis of an output signal of an m-th sensor,

wherein an obstacle detection range that is a range of the entire obstacle detection system is configured to include an m-th detection range,

wherein a distance from the (m + 1) th detection range to the vehicle is larger than a distance from an m-th detection range to the vehicle, and

wherein, when it is determined that malfunction has occurred in a n-th (n is an integer of 1 or more and M or less) sensor,

on a basis of at least one of an n-th detection range of the n-th sensor, presence or absence of a human substitution means for obstacle detection, a traveling speed of the vehicle, and a traveling speed scheduled after a certain period of time or after a certain distance of travel of the vehicle,

when an end of a p-th detection range of a p-th sensor (p is an integer of 1 or more and M or less other than n) closest to the vehicle in the p-th detection range is defined as a first end, and an end of the p-th detection range farthest from the vehicle is defined as a second end,

the controller performs at least one change processing of first change processing of causing the first end to approach the vehicle and second change processing of causing the second end to move away from the vehicle, and changes at least the p-th detection range to change the obstacle detection range.
The obstacle detection system according to claim 1, wherein the p-th detection range of the p-th sensor which is subjected to the change processing includes at least a part of the n-th detection range.
The obstacle detection system according to claim 1, wherein the obstacle detection range after the change processing has detection ranges arranged continuously.
The obstacle detection system according to claim 1, wherein the obstacle detection range after the change processing includes a human detection range by visual observation.
The obstacle detection system according to claim 1, wherein the controller sets an upper limit value of a traveling speed of the vehicle on a basis of the obstacle detection range in which the change processing is scheduled to be performed.
The obstacle detection system according to claim 5, wherein the controller performs the change processing after an actual speed of the vehicle becomes lower than an upper limit value of the traveling speed.
The obstacle detection system according to claim 1, further comprising a calculation resource used when the signal processing unit performs the signal processing,
wherein the controller allocates the calculation resource allocated to signal processing of the n-th sensor in which malfunction is determined to occur to signal processing of the p-th sensor to be subjected to the change processing.
The obstacle detection system according to claim 1,
wherein the sensor on which the change processing is performed includes a stereo camera sensor including left and right cameras, and

wherein the controller changes a detection range by performing at least one of changing a magnification of an acquired image of the left and right cameras, changing a search range of stereo matching, and rotating at least one optical axis of the left and right cameras inward in a yaw direction.
The obstacle detection system according to claim 1, wherein the controller checks detection performance in the changed p-th detection range using a sensor including the first end changed after the first change processing in a detection range or a sensor including the second end changed after the second change processing in a detection range.
The obstacle detection system according to claim 4,
wherein the controller recognizes whether or not an obstacle can be detected by human resources,

when it is determined that at least one of the M sensors is malfunctioning when the obstacle can be detected by the human resources,

the controller outputs information indicating that a handover that passes an operation and a range of obstacle detection to a human resource is performed,

the controller performs a handover after outputting information indicating that the handover is possible by the human resource, or performs handover switching by the human resource and recognizes execution of the handover, and

the controller communicates a human detection range to the human resource via a user interface, such as a monitor or a head mount display.
The obstacle detection system according to claim 1, wherein the controller determines that malfunction has occurred when an output signal of the n-th sensor is not updated for a certain period of time.
The obstacle detection system according to claim 1, wherein the controller
estimates a cause of malfunction of the n-th sensor on a basis of a difference between statistical information such as a luminance average value, a luminance variance value, and a luminance histogram for an entire image or each area and statistical information for each area, and

instructs a crew member of the vehicle to take a measure to solve the cause.
The obstacle detection system according to claim 1,
wherein the n-th sensor and the q-th sensor (q is an integer of 1 or more and M or less other than n) constitute a stereo camera sensor, and

wherein, when the signal processing unit receives the n-th sensor as an input and performs obstacle detection using a monocular image, the controller switches to the monocular image having the q-th sensor as an input to perform obstacle detection.
A train comprising a first obstacle detection system and a second obstacle detection system that are the obstacle detection system according to claim 5,
wherein the first obstacle detection system is provided in a leading vehicle of the train, and the second obstacle detection system is provided in a last vehicle of the vehicle, and

wherein the first obstacle detection system detects an obstacle when the train travels in the traveling direction, and the second obstacle detection system detects an obstacle when the train travels in a backward direction opposite to the traveling direction,

the train comprising a communication means through which a controller of the first obstacle detection system and a controller of the second obstacle detection system exchange information,

wherein, when malfunction is detected in the first obstacle detection system,

the controller of the first obstacle detection system checks soundness of the second obstacle detection system, and determines whether or not to change a traveling direction or a position to change the traveling direction on a basis of at least one of current position information of the vehicle, route map information, presence or absence of a passenger, a position where assistance by human resources is obtained, and an upper limit value of a traveling speed calculated by the controller of the first obstacle detection system, in addition to information on the checked soundness.