JP2023045830A - Object detection device and object detection method - Google Patents

Abstract

To provide an object detection method capable of detecting an object even when it is hard to distinguish between a background and the object due to low illuminance in surroundings.SOLUTION: An object detection device comprises: an object detection unit which calculates an object existence probability and an object classification probability for each object detection range from a photographed image obtained as a result of imaging by photographing means, and if the object existence probability alone or both the object existence probability and the object classification probability are equal to or greater than prescribed values, determines that an object classified as a highest object classification probability exists in the object detection range; and a three-dimensional sensor which generates point cloud data including at least distance information in a depth direction of the object existing in the detection range, wherein the detection range includes a photographing range of the photographing means. The object detection unit assigns point clouds based on the point cloud data to the object detection range, compares the object existence probability of the object detection range with a point cloud occupancy ratio for the object detection range of the assigned point clouds, and if the point cloud occupancy ratio is greater than object existence probability data, takes the point cloud occupancy ratio as the object existence probability of the object detection range.SELECTED DRAWING: Figure 1

Description

The present invention relates to an object detection device and an object detection method.

Non-Patent Document 1 and Non-Patent Document 2 disclose methods for detecting an object from an image. In the method shown in Non-Patent Document 1, object regions and classification are predicted by deep learning from images of a visible light video camera. In addition, according to the method shown in Non-Patent Document 2, a person, a car, etc. at night are detected using an image of a far-infrared video camera.

A. Bochkovskiy, C. Y. Wang and H. Y. Mark Liao: YOLOv4: Optimal Speed and Accuracy of Object Detection, arXiv:2004.10934, 2020 Masayoshi Aoki: Application of Infrared Image to Human Detection, Information Processing Society of Japan Research Report Computer Vision and Image Media, 2004-CVIM-147, pp.105-113, 2005

In the method of detecting an object only from the image of a visible light video camera, since detection is based on the characteristics of the object in the image, the detection accuracy decreases when the surrounding illumination is low and it is difficult to distinguish between the background and the object. . On the other hand, in the case of the method of detecting an object from the image of an infrared video camera, the object to be detected is limited to an object having heat, although it does not depend on the illuminance because temperature information is imaged.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an object detection method capable of detecting an object even when the ambient illumination is low and it is difficult to distinguish between the background and the object.

The technical means of the present invention for solving this technical problem are characterized by the following points.
The object detection device calculates an object existence probability and an object classification probability for each object detection range from the captured image obtained as a result of imaging by the imaging means. is a detection range that includes an object detection unit in which an object of the classification with the highest object classification probability exists, and the imaging range of the imaging means, and includes at least information on the distance in the depth direction of the object existing in the detection range. a three-dimensional sensor that generates group data, wherein the object detection unit assigns a point group based on point cloud data to the object detection range, and calculates an object existence probability in the object detection range and the value of the assigned point group The point cloud occupancy of the object detection range is compared, and if the point cloud occupancy is greater than the object existence probability data, the point cloud occupancy is used as the object existence probability of the object detection range.

According to the present invention, an object can be detected even when the surrounding illuminance is low.

FIG. 1 is a functional block diagram showing the functional configuration of an object detection system 1 that is an embodiment of the present invention. FIG. 2 is a flowchart for explaining object detection processing at low illumination. FIG. 3 is a diagram showing an image of processing for calculating object existence probability and object classification probability. FIG. 4 is a flow chart showing details of step S2 in FIG. FIG. 5 is a diagram showing an image of the classification result of the object detection range. FIG. 6 is a diagram showing an image of processing for assigning point groups to object detection ranges.

FIG. 1 is a diagram showing the configuration of an object detection system 1 according to this embodiment.

The object detection system 1 is a system that detects an object that can become an obstacle in the running of a railroad vehicle. For example, there are moving objects such as people and vehicles existing on and around a running railroad track, and stationary objects such as trees that have fallen on the railroad track.

The object detection system 1 includes a video camera 11 , a three-dimensional sensor 12 and an object detection device 13 .

The video camera 11 is installed so as to capture an image of a predetermined range in the front direction of the railroad vehicle, and outputs a captured image obtained by capturing to the object detection device 13 .

The three-dimensional sensor 12 is a sensor that detects the position of an object in a three-dimensional space, such as a LiDAR (Laser Imaging Detection and Ranging) sensor or a three-dimensional millimeter wave sensor. A spatial coordinate system that defines the three-dimensional space is an orthogonal three-dimensional coordinate system or a polar coordinate system with a predetermined position (for example, the installation position of the three-dimensional sensor 12) within the range of the railway vehicle as the origin. The three-dimensional sensor 12 is installed on the railway vehicle so as to be able to scan the detection range including the imaging range of the video camera 11 , and outputs a detection signal obtained by scanning the detection range to the object detection device 13 .

The three-dimensional sensor 12 includes a laser light source, scanning unit, light receiving element, and detection circuit.
The laser light source includes, for example, a plurality of laser light emitting portions arranged in a row in one direction, and emits pulsed laser light from each laser light emitting portion.

The scanning unit is a unit that scans each laser beam. The scanning unit comprises an optical element and a scanning actuator. The optical element is an element such as a reflecting mirror that directs each laser beam in a predetermined irradiation direction. The scanning actuator is a device that rotates the optical element forward and backward at a predetermined scanning speed in a direction perpendicular to the arrangement direction of each laser beam. The three-dimensional sensor 12 is installed on the railway vehicle in a posture in which the alignment direction and the scanning direction of the laser beams are aligned with the height direction of the railway vehicle and the direction perpendicular to the height direction (vehicle width direction), respectively. be.

The light-receiving element is an element that receives the reflected light that is the pulse of each laser beam reflected by an object. The detection circuit detects the position of the reflection point on the surface of the object on which each laser beam is reflected based on the result of the reflected light received by the light receiving element, and outputs it to the object detection device 13 as point cloud data.

The point cloud data is data including information on the direction of each reflection point and the distance between each reflection point in spatial coordinates with reference to the railway vehicle. The detection circuit obtains the direction of the reflection point based on the light receiving direction of the reflected light (that is, the irradiation direction of the laser beam that is the source of the reflected light). Further, the detection circuit obtains the distance of the reflection point based on the time from the irradiation of the laser beam to the reception of the reflected light. The direction and distance of these reflection points uniquely identify the position (coordinate point) of the reflection point in the spatial coordinates.

The object detection device 13 includes processors such as CPU (Central Processing Unit) and MPU (Micro-Processing Unit), memory devices (also called main storage devices) such as ROM (Read Only Memory) and RAM (Random Access Memory), HDD A storage device (also called an auxiliary storage device) such as a hard disk drive (hard disk drive) or SSD (Solid State Drive), and a computer equipped with an interface circuit for connecting sensors, peripheral devices, and the like. In the object detection device 13, a processor executes a computer program stored in a memory device or a storage device, thereby realizing various functional configurations for detecting an object.

The object detection device 13 includes an image input section 21 , a point cloud data input section 22 , an object detection section 23 and an output control section 24 . The image input unit 21 acquires a captured image from the video camera 11 and outputs it to the object detection unit 23 . The point cloud data input unit 22 acquires point cloud data from the three-dimensional sensor 12 and outputs it to the object detection unit 23 .

The object detection unit 23 detects an object based on the captured image from the video camera 11 and the point cloud data from the three-dimensional sensor 12, and outputs the detection result to an external device via the output control unit 24. Specifically, the object detection unit 23 detects whether an object exists in the object detection range using the data (hereinafter referred to as learned data) in which the object is learned in advance by machine learning, which is stored in the storage unit 31 . A probability (hereinafter referred to as an object existence probability) and a probability that the object is in a predetermined classification (hereinafter referred to as an object classification probability) are calculated. When the object existence probability or the product of the object existence probability and the object classification probability is larger than the normal determination reference value, the object detection unit 23 determines that an object exists in the detection range and the object is classified into the object with the highest object classification probability. (e.g. people, cars, traffic lights). Hereinafter, this processing is referred to as AI (artificial intelligence) object detection processing. When it is difficult to distinguish between the background and the object in the AI object detection process, the object detection unit 23 calculates the occupancy rate of the point cloud in the object detection range, as will be described later. (hereinafter referred to as point cloud occupancy) is used as the object existence probability to perform object detection. Hereinafter, this process will be referred to as low-illuminance object detection process.

Note that the object detection unit 23 may detect obstacles without using machine learning. For example, the object detection unit 23 can also calculate the object existence probability and the object classification probability based on the captured image using a well-known image recognition technique.

The low-illuminance object detection processing of the object detection unit 23 will be described with reference to the flowchart of FIG.

In step S<b>1 , the object detection unit 23 of the object detection device 13 calculates object existence probability and object classification probability based on the captured image of the video camera 11 input via the image input unit 21 . FIG. 3 is a diagram showing an image of processing for calculating object existence probability and object classification probability. When the image shown in FIG. 3A is input, the object detection unit 23 evenly distributes object detection areas of various sizes (rectangular frames in the figure) in the image as shown in FIG. 3B. Alternatively, it is set randomly, and the object existence probability and the object classification probability are calculated for each object detection range.

Next, in step S<b>2 , the object detection unit 23 uses the point cloud data input via the point cloud data input unit 22 to replace the calculated object existence probability. The details of this processing will be described later.

In step S3, the object detection unit 23 outputs an object detection result based on the object existence probability and the object classification probability for each object detection range. If there is an object detection range that has an object existence probability equal to or higher than the normal judgment reference value or the product of the object existence probability and the object classification probability, the object detection unit 23 determines that an object exists in the object detection range, and the object is , the existence of an object of the classification (for example, human) with the highest object classification probability in the object detection range is output as a detection result.

FIG. 4 is a flow chart showing details of step S2 in FIG. The following processing is performed for each object detection range.

In step S11, the object detection unit 23 classifies objects presumed to exist within the object detection range based on the object classification probabilities. In the object detection range, a plurality of categories are assigned probabilities (for example, people = 0.99, traffic lights = 0.01, dogs = 0, etc.). Specified as a class of objects presumed to be in the detection range.

In step S12, the object detection unit 23 classifies the object detection range according to the size of the object detection range according to the classification of the object in step S11. For example, if the object classification is "people", an object detection range with a size of 50px x 100px or more will be assigned to the object detection range (large), and an object detection range with a size of 30px x 60px to 50px x 100px will be assigned to an object Object detection areas with a size smaller than 30px x 60px are classified as object detection areas (medium) and object detection areas (small), respectively. Also, if the object classification is "truck", the object detection range with a size of 260px x 120px or more is set as the object detection range (large), and the object detection range with a size of 160px x 75px to 260px x 120px is set as the object detection range. Object detection areas with a size smaller than 160px x 75px are classified as object detection areas (small), respectively.

FIG. 5 is a diagram showing an image of the classification result of the object detection range. In the figure, the two-dot chain line indicates the object detection range (large), the one-dot chain line indicates the object detection range (middle), and the solid line indicates the object detection range (small). being classified. The two-dot chain line, the one-dot chain line, etc. in FIG. 5 are shown to indicate the classification of the object detection range.

Returning to FIG. 4, in step S13, the object detection unit 23 assigns a point group to the object detection range according to the classification of the object detection range in step S13. When the classification of the object is "person", for example, in the object detection range classified as the object detection range (large), the point cloud whose distance (hereinafter referred to as depth) from the three-dimensional sensor 12 is less than 100 m, Point clouds with a depth of 100 m or more and less than 200 m are assigned to object detection ranges classified as object detection range (medium), and point clouds with a depth of 200 m or more are assigned to object detection ranges classified as object detection range (small). be done.

FIG. 6 is a diagram showing an image of processing for assigning point groups to object detection ranges. FIG. 6 shows an example in which the classification of the object is "person", and the point group G1 with a depth of 50 mm is assigned to the object detection range W1 classified as the object detection range (large), and the object detection range ( A point group G2 with a depth of 150 mm is assigned to the object detection range W2 classified as medium), and a point group G3 with a depth of 250 mm is assigned to the object detection range W3 classified as the object detection range (small). ing.

A part of the point group G2 exists in both the object detection range W2 and the object detection range W4. In this example, since the depth of the point group G2 is 150 mm, the point group G2 is assigned to the object detection range W2 classified as the object detection range (middle). That is, when a part of the point cloud exists in multiple object detection ranges, it is assigned to the object detection range corresponding to the distance of the point cloud.

Returning to FIG. 4, in step S14, the object detection unit 23 calculates the point cloud occupancy of the object detection range of the point cloud assigned to the object detection range. Specifically, the number of dots in the point cloud divided by the number of pixels within the object detection range is calculated as the point cloud occupancy rate.

Next, in step S15, the object detection unit 23 compares the object existence probability in the object detection range with the point cloud occupancy of the assigned point cloud, and determines whether the point cloud occupancy is greater than the object existence probability. If it is determined that the point cloud occupancy is greater than the object existence probability, then in step S16, the point cloud occupancy is replaced with the object existence probability of the object detection range.

When the point cloud occupancy is replaced with the object existence probability of the object detection range in step S16, or when it is determined in step S15 that the point cloud occupancy is not greater than the object existence probability, the process is as shown in FIG. Return to step S3. In step S3, when there is an object detection range having an object existence probability equal to or higher than the normal determination reference value, the object detection unit 23 determines that "an object exists in the object detection range, and the object is an object in the object detection range." The fact that an object with the highest classification probability (for example, a person) exists" is output as a detection result.

(effect)
An object existence probability and an object classification probability are calculated for each object detection range from the captured image obtained as a result of imaging by the video camera 11, and the object existence probability or the product of the object existence probability and the object classification probability is equal to or greater than a predetermined value. In this case, the object detection range includes the object detection unit 23 which assumes that an object of the classification with the highest object classification probability exists, and the imaging range of the video camera 11. At least the depth direction of the object existing in the detection range is A three-dimensional sensor 12 that generates point cloud data including distance information, and an object detection unit 23 assigns a point cloud based on the point cloud data to the object detection range (step S13 in FIG. 4, FIG. 6) , compare the object existence probability of the object detection range and the point cloud occupancy of the object detection range of the assigned point cloud, and if the point cloud occupancy is greater than the object existence probability data, the point cloud occupancy is assigned to the object detection range (step S16).
Therefore, for example, when the illuminance becomes low in the evening, the object existence probability becomes smaller than the normal determination reference value even though the object exists in the AI object detection process, and the object may not be detected. However, if the point cloud by the three-dimensional sensor 12 that is not affected by the surrounding illumination is assigned to the object detection range, and the point cloud occupancy is greater than the object existence probability, the point cloud occupancy is used as the object existence probability of the object detection range. , a more accurate object existence probability can be set, and as a result, the object can be detected appropriately.

The object detection unit 23 classifies the object detection range according to the size of the object detection range according to the classification of the object (step S12). Assignment: A group of points far from the three-dimensional sensor 12 is assigned to a small object detection range (step S13).
That is, point groups are assigned to object detection ranges corresponding to distances from the three-dimensional sensor 12 . Therefore, the point cloud can be assigned to the object detection range according to the depth of the point cloud. It is expected to be effective in preventing erroneous detection and over-detection.

Also, since the size of the object detection range for a small nearby object (e.g. a person) and a large distant object (e.g. a truck) are about the same, it is recommended to base the classification of the predicted object on the object detection range. , it is possible to more accurately assign the point cloud to the object detection range.

REFERENCE SIGNS LIST 11: video camera 12: three-dimensional sensor 13: object detection device 21: image input unit 22: point cloud data input unit 23: object detection unit

Claims (4)

When the object existence probability and the object classification probability are calculated for each object detection range from the imaged image obtained as a result of imaging by the imaging means, and the object existence probability or the product of the object existence probability and the object classification probability is equal to or greater than a predetermined value. , an object detection unit that determines that an object with the highest object classification probability exists in the object detection range;
a three-dimensional sensor that generates point cloud data including at least distance information in the depth direction of an object existing in the detection range, wherein the detection range includes the imaging range of the imaging means;
The object detection unit assigns a point group based on point cloud data to the object detection range, compares the object existence probability of the object detection range with the point cloud occupancy of the object detection range of the assigned point group, When the point cloud occupancy is greater than the object existence probability, the point cloud occupancy is set as the object existence probability of the object detection range. In the object detection device of claim 1,
The object detection unit classifies the object detection range according to the size of the object detection range according to the classification of the object, assigns a large object detection range to a point group having a short distance from the three-dimensional sensor, An object detection device that allocates a group of points far from the three-dimensional sensor to an object detection range. An object existence probability and an object classification probability are calculated for each object detection range from the captured image obtained as a result of imaging by the imaging means. an object detection step in which an object with the highest classification exists;
an input step of inputting the point cloud data from a three-dimensional sensor that generates point cloud data including at least distance information in the depth direction of an object existing in the detection range, the detection range including the imaging range of the imaging means; with
In the object detection step, a point group based on point cloud data is assigned to the object detection range, and the object existence probability of the object detection range is compared with the point cloud occupancy of the object detection range of the assigned point group, When the point cloud occupancy is greater than the object existence probability, the point cloud occupancy is set as the object existence probability of the object detection range. In the object detection method of claim 3,
The object detection step classifies the object detection range according to the size of the object detection range according to the classification of the object, assigns a point group having a short distance from the three-dimensional sensor to a large object detection range, An object detection method, wherein a group of points far from the three-dimensional sensor is assigned to an object detection range.

Images