JP2023110381A - Moving body detection apparatus, moving body detection method, system, and program - Google Patents

Abstract

To simply and quickly detect a moving body existing around a vehicle.SOLUTION: A moving body detection apparatus includes: an image acquisition unit which acquires image data including a plurality of frames representing surroundings of a mobile body, the image data being captured in time series by a camera mounted on the mobile body; a difference calculation unit which calculates a difference between the frames, and binarizes the difference into a first value and a second value, to calculate a difference image between the frames; a grid extraction unit which extracts, out of multiple grids set in the difference image, a grid in which density of pixels of the first value is equal to or higher than a first threshold; a bounding box setting unit which sets a bounding box which is circumscribed rectangle, on the one or more extracted grids that satisfy a predetermined criterion; and a moving body detection unit which detects, as a moving body, the bounding box when the density of the grids in the set bounding box is equal to or higher than a second threshold.SELECTED DRAWING: Figure 1

Description

The present invention relates to a moving body detection device, a moving body detection method, a system, and a program.

2. Description of the Related Art Conventionally, there is known a technique of detecting a moving object existing around a vehicle based on image data of the front of the vehicle captured by an in-vehicle camera. For example, Patent Literature 1 discloses a technique for outputting identification results of moving objects existing in the image data by performing signal processing on the image data of the surrounding environment of the vehicle based on the results of pre-learning. It is

JP 2021-144689 A

The technique described in Patent Literature 1 uses a deep neural network (DNN) such as a convolutional neural network to detect a moving object existing around a vehicle. However, such machine learning techniques require a large amount of learning data to be prepared in advance, and tend to have a high processing load during execution. As a result, it may not be possible to detect a moving object existing around the vehicle at high speed.

The present invention has been made in consideration of such circumstances, and provides a moving object detection device, a moving object detection method, a system, and a program that can easily and quickly detect a moving object existing around a vehicle. One of the purposes is to

A moving body detection device, a moving body detection method, a system, and a program according to the present invention employ the following configurations.
(1): A moving body detection device according to an aspect of the present invention acquires image data including a plurality of frames representing surrounding conditions of a moving body captured in time series by a camera mounted on the moving body. an acquisition unit; and a difference calculation unit that calculates a difference image between the plurality of frames by calculating the difference between the plurality of frames and binarizing the difference into a first value and a second value. a grid extracting unit for extracting a grid having a pixel density of the first value equal to or higher than a first threshold from among the plurality of grids set in the difference image; a bounding box setting unit that sets a bounding box that is a circumscribing rectangle for the grid of the above; and detects the bounding box as a moving object when the density of the grid in the set bounding box is equal to or greater than a second threshold. and a moving object detection unit.

(2): In the aspect of (1) above, the bounding box setting unit may set the bottom edge of the one or more grids and the height of the one or more grids relative to the bottom edge to each have a constant value. , that the one or more grids satisfy the predetermined criteria.

(3): In the aspect (1) or (2) above, the bounding box setting unit searches for one or more grids satisfying the predetermined criteria in the center of a grid image containing the grids.

(4): In any one of the above aspects (1) to (3), the difference calculation unit calculates the speed of the moving object in the shooting interval at which the plurality of frames were captured. A frame is enlarged, and a difference image between the enlarged frame captured at the previous time and the frame captured at the current time is calculated.

(5): In the aspect of (4) above, the difference calculation unit enlarges the frame captured at the previous time around the vanishing point of the frame captured at the previous time.

(6): In any one of the above aspects (1) to (5), the difference calculation unit calculates the image at the previous time based on the yaw rate of the moving object in the shooting interval at which the plurality of frames were imaged. The frame is corrected, and a difference image between the corrected frame captured at the previous time and the frame captured at the current time is calculated.

(7): In any one of the above (1) to (6), the grid extracting section changes the threshold according to the distance of each of the plurality of grids from the camera.

(8): In any one of the above aspects (1) to (7), the grid extraction unit sets the size of the plurality of grids to the first distance when the distance from the camera is equal to or less than the first distance. size, and if the distance from the camera is greater than a first distance and less than or equal to a second distance, then set the size of the plurality of grids to a second size smaller than the first size; is greater than a second distance, the size of the plurality of grids is set to a third size smaller than the second size.

(9): A system according to an aspect of the present invention includes the moving object detection device according to any one of aspects (1) to (8) above, and based on the detection result obtained by the moving object detection device, and a driving support device that supports driving of the body.

(10): A moving object detection method according to an aspect of the present invention is characterized in that a computer captures image data including a plurality of frames representing surrounding conditions of a moving object in time series by a camera mounted on the moving object. calculating a difference between the plurality of frames; binarizing the difference into a first value and a second value to calculate a difference image between the plurality of frames; extracting a grid image from the difference image by extracting grids in which the density of pixels of the first value is equal to or greater than a first threshold from among a plurality of set grids, and is included in the grid image; A bounding box is set for one or more grids that satisfy predetermined criteria, and the bounding box is detected as a moving object when the density of the grid in the set bounding box is equal to or higher than a second threshold. .

(11): A program according to an aspect of the present invention causes a computer to acquire image data including a plurality of frames representing surrounding conditions of the moving body, which are captured in time series by a camera mounted on the moving body. calculating a difference between the plurality of frames, binarizing the difference into a first value and a second value to calculate a difference image between the plurality of frames, and setting the difference image as the difference image; A grid image is extracted from the difference image by extracting a grid having a density of pixels having the first value equal to or higher than a first threshold from among the plurality of grids, and the grid image is included in the grid image and has a predetermined criterion. A bounding box is set for one or more grids that satisfy the following, and if the grid density in the set bounding box is equal to or higher than a second threshold, the bounding box is detected as a moving object.

According to the aspects (1) to (11), it is possible to easily and quickly detect a moving object existing around the vehicle.

According to the aspect (2), it is possible to more accurately detect a moving object existing around the vehicle.

According to the aspect (3), it is possible to reduce the processing load of searching for a moving object existing around the vehicle.

According to the aspect (4) or (5), it is possible to accurately calculate the difference image between the frame captured at the previous time and the frame captured at the current time.

According to the aspect (6), the difference image can be accurately calculated in consideration of the yaw rate of the moving body.

According to aspect (7) or (8), a moving object can be accurately detected according to the distance from the camera.

According to the aspect (9), the detection result by the moving body detection device can be preferably used for driving assistance.

1A and 1B are diagrams illustrating an example of a configuration and peripheral devices of a moving object detection device 100 according to a first embodiment; FIG. FIG. 2 is a diagram showing an example of a surrounding situation of a vehicle M equipped with the moving object detection device 100; 3 is a diagram showing an example of an image obtained by the camera 10 capturing an image of the front of the vehicle M in the surrounding situation shown in FIG. 2. FIG. FIG. 10 is a diagram showing an example of a vanishing point used as a reference point for enlarging a frame; FIG. 10 is a diagram for explaining a method of correcting a previous frame to calculate a difference image DI; 4 is a diagram showing an example of a difference image DI calculated by a difference calculation unit 120; FIG. 4 is a diagram showing an example of the configuration of a grid G set by a grid extraction unit 130; FIG. FIG. 4 is a diagram showing an example of a method of extracting a grid G by a grid extraction unit 130; 4 is a diagram showing an example of a grid image GI calculated by a grid extraction unit 130; FIG. FIG. 4 is a diagram showing an example of an operation performed by the cruise control device 200; FIG. 2 is a diagram showing another example of the configuration of the moving object detection device 100 and peripheral devices. FIG. FIG. 3 is a diagram showing an example of operations performed by a notification device 210; FIG. 4 is a diagram showing an example of the flow of processing executed by the moving object detection device 100; FIG. FIG. 11 is a diagram for explaining operations performed by a moving object detection unit 140 according to the second embodiment; FIG. FIG. 10 is a diagram showing an example of the flow of processing executed by the moving object detection device 100 according to the second embodiment; FIG. 11 is a diagram showing an example of the configuration and peripheral devices of a moving object detection device 100 according to a third embodiment; FIG. 10 is a diagram showing an example of a grid G search method executed by a bounding box setting unit 132. FIG. 6 is a diagram showing an example of a search range of the grid G by the bounding box setting unit 132; FIG. FIG. 12 is a diagram showing an example of the flow of processing executed by the moving object detection device 100 according to the third embodiment; FIG.

[First embodiment]
Hereinafter, embodiments of a moving body detection device, a moving body detection method, a system, and a program according to the present invention will be described with reference to the drawings. A moving body detection device is mounted on, for example, a moving body. A mobile body is, for example, a four-wheeled vehicle, a two-wheeled vehicle, a micromobility, a robot that moves by itself, or a portable type such as a smartphone that is placed on a mobile body that moves by itself or is carried by a person. It is a device. In the following description, the mobile object is assumed to be a four-wheeled vehicle, and the mobile object is referred to as a "vehicle". The moving object detection device is not limited to being mounted on a moving body, and may be one that performs the processing described below based on an image captured by a camera for fixed-point observation or a camera of a smartphone.

FIG. 1 is a diagram showing an example of the configuration and peripheral devices of a moving object detection device 100. As shown in FIG. The moving object detection device 100 communicates with the camera 10, the travel control device 200, and the like.

The camera 10 is attached to the rear surface of the windshield of the vehicle M or the like, takes images of at least the road in the traveling direction of the vehicle M in time series, and outputs the taken images to the moving object detection device 100 . A sensor fusion device or the like may be interposed between the camera 10 and the moving object detection device 100, but the description thereof will be omitted. The travel control device 200 is, for example, an automatic driving control device that causes the vehicle M to travel autonomously, a driving support device that performs vehicle-to-vehicle distance control, automatic brake control, automatic lane change control, and the like.

The moving object detection device 100 includes, for example, an image acquisition unit 110, a difference calculation unit 120, a grid extraction unit 130, and a moving object detection unit 140. These components are implemented by executing a program (software) by a hardware processor such as a CPU (Central Processing Unit). Some or all of these components are hardware (circuit part; circuitry) or by cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as a HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage such as a DVD or CD-ROM. It may be stored in a medium (non-transitory storage medium) and installed by loading the storage medium into a drive device.

FIG. 2 is a diagram showing an example of a surrounding situation of a vehicle M equipped with the moving object detection device 100. As shown in FIG. FIG. 2 shows, as an example, a scene in which a motorcycle B runs in front of the vehicle M while the vehicle M equipped with the moving object detection device 100 is running on a road. In the following description, a scene in which the moving object detection device 100 detects the motorcycle B as a moving object will be described as an example, but the present invention is not limited to such a scene.

FIG. 3 is a diagram showing an example of an image obtained by imaging the front of the vehicle M with the camera 10 in the surrounding situation shown in FIG. The image acquisition unit 110 acquires image data including a plurality of frames representing the surrounding situation of the vehicle M, captured in time series by the camera 10 mounted on the vehicle M. FIG. More specifically, for example, the image acquisition unit 110 acquires image data from the camera 10 at a frame rate of 30 Hz.

The difference calculation unit 120 calculates differences in pixel values for a plurality of frames acquired by the image acquisition unit 110, and converts the calculated differences into a first value (eg, 1) and a second value (eg, 0). , the difference image DI between the plurality of frames is calculated. More specifically, first, the difference calculation unit 120 performs gray conversion on the plurality of frames acquired by the image acquisition unit 110 to convert the RGB image into a grayscale image.

Next, the difference calculation unit 120 calculates the frame imaged at the previous time (hereinafter sometimes referred to as the “previous frame”) based on the speed of the vehicle M in the imaging interval at which the multiple frames are imaged. By enlarging the image centering on the vanishing point of , alignment with the frame captured at this time (hereinafter sometimes referred to as "current frame") is performed.

FIG. 4 is a diagram showing an example of a vanishing point used as a reference point for enlarging a frame. In FIG. 4, VP indicates the vanishing point of the frame. The vanishing point VP is defined, for example, as an intersection connected by extending both sides of the lane in which the vehicle M travels. The difference calculation unit 120, for example, estimates the moving distance of the vehicle M from the speed (average speed) of the vehicle M measured between the previous time point and the current time point, and calculates the vanishing point by the enlargement rate corresponding to the moving distance. Enlarging the previous frame with the VP as the center.

FIG. 5 is a diagram for explaining a method of correcting the previous frame in order to calculate the difference image DI. In FIG. 5, h indicates the height of the frame and w indicates the width of the frame. As shown in FIG. 5, the difference calculation unit 120 enlarges the previous frame by an enlargement rate corresponding to the travel distance of the vehicle M measured between the previous time and the current time. At this time, since the size of the enlarged previous frame becomes larger than before enlargement, the difference calculation unit 120 trims the ends of the enlarged previous frame to reduce the size of the enlarged previous frame to the original size. return to size.

Note that the difference calculation unit 120 corrects the previous frame in consideration of the yaw rate of the vehicle M in the imaging interval between the previous frame and the current frame in addition to the speed of the vehicle M in the imaging interval between the previous frame and the current frame. may More specifically, the difference calculation unit 120 calculates the difference between the yaw angle of the vehicle M when the previous frame was acquired and the yaw angle of the vehicle M when the current frame was acquired, based on the yaw rate at the shooting interval. may be calculated, and the previous frame and the current frame may be aligned by shifting the previous frame in the yaw direction by an angle corresponding to the difference.

FIG. 6 is a diagram showing an example of the difference image DI calculated by the difference calculator 120. As shown in FIG. After aligning the previous frame with the current frame, the difference calculation unit 120 calculates the difference between the pixel values of the previous frame and the current frame. When the difference value calculated for each pixel is equal to or greater than a specified value, the difference calculation unit 120 assigns the pixel a first value indicating that the pixel is a moving object candidate. On the other hand, if the calculated difference value is less than the specified value, the difference calculation unit 120 assigns the pixel a second value indicating that the pixel is not a moving object candidate. As shown in FIG. 6, it can be seen that the bike B is detected as a moving object in the difference image DI calculated by the difference calculation unit 120 .

The grid extraction unit 130 sets a grid for each of a plurality of pixels in the difference image DI calculated by the difference calculation unit 120, and the density (ratio) of pixels having the first value in each of the set grids is a threshold value. If so, the grid is extracted.

FIG. 7 is a diagram showing an example of the configuration of grids set by the grid extraction unit 130. As shown in FIG. In FIG. 7, G indicates a set of pixels defined as a grid in the difference image DI. As shown in FIG. 7, the grid extracting unit 130 sets the size of the grid G to 10×10 pixels, for example, in the area of the difference image DI whose distance from the camera 10 is equal to or less than a first distance (for example, 10 m). (which is an example of the “first size”), and the distance from the camera 10 is greater than the first distance and less than or equal to the second distance (for example, 20 m), the size of the grid G is set to 8×8 pixels (which is an example of the “second size”), and for the area where the distance from the camera 10 is greater than the second distance, the size of the grid G is set to 5×5 pixels (which is an example of the “third size”) set to This is because the greater the distance from the camera 10, the smaller the change in the area imaged by the camera 10, and the finer the size of the grid G needs to be set in order to detect a moving object. By setting the size of the grid G according to the distance from the camera 10 in the difference image DI, the moving object can be detected more accurately.

FIG. 8 is a diagram showing an example of a grid G extraction method by the grid extraction unit 130. As shown in FIG. The grid extraction unit 130 determines whether or not the density of pixels with the first value is equal to or greater than a threshold (for example, 85%) for each of the plurality of grids G, and determines whether the density of pixels with the first value exceeds the threshold. For the grid G determined as above, as shown in the upper part of FIG. 8, all the pixels forming the grid G are extracted (set to the first value). On the other hand, the grid extracting unit 130 discards all the pixels forming the grid G for which the density of the pixels of the first value is determined to be less than the threshold, as shown in the lower part of FIG. second value).

Note that in the above description, the grid extraction unit 130 determines whether or not the density of pixels with the first value is equal to or greater than a single threshold for each of the plurality of grids G. FIG. However, the present invention is not limited to such a configuration, and the grid extraction section 130 may change the threshold according to the distance from the camera 10 in the difference image DI. For example, in general, the closer the distance from the camera 10 is, the larger the change in the area captured by the camera 10 is, and the more likely an error is to occur. A higher threshold may be set. Furthermore, the grid extracting unit 130 may perform the determination using not only the density of the pixels with the first value but also any statistical value based on the pixels with the first value.

The grid extracting unit 130 performs a process (grid replacement process) of setting all the pixels of the grid whose pixel density of the first value is equal to or higher than the threshold value to the first value on the difference image DI. Calculate the image GI. FIG. 9 is a diagram showing an example of the grid image GI calculated by the grid extracting section 130. As shown in FIG. The components of the difference image DI shown in FIG. 8 are pixels, while the components of the grid image GI shown in FIG. 9 are grids. As shown in FIG. 9, it can be seen that the grid representing the bike B is detected in the grid image GI by performing the grid replacement process on the difference image DI. That is, the moving object detection unit 140 detects the grid represented in the grid image GI as a moving object.

The moving object detection result by the moving object detection unit 140 is transmitted to the running control device 200, and the running control device 200 controls the running of the vehicle M based on the received detection result. FIG. 10 is a diagram showing an example of an operation performed by cruise control device 200. In FIG. FIG. 10 shows, as an example, a situation in which the moving object detection unit 140 detects the motorcycle B as a moving object. At this time, the travel control device 200 generates a trajectory for the vehicle M so as to avoid the moving object detected by the moving object detection unit 140, and causes the vehicle M to travel along the generated trajectory. In the case of FIG. 10, the travel control device 200 controls the vehicle M so that it stops at a point SP immediately before the crosswalk on which the bike B travels. Thereby, the detection result by the moving object detection device 100 can be suitably utilized for automatic driving of the vehicle M or driving assistance. The travel control device 200 is an example of a “driving support device”.

In the above description, the case where the detection result by the moving object detection device 100 is used for automatic driving has been described. However, the present invention is not limited to such a configuration, and the detection result by the moving object detection device 100 can also be used as driving support information provided to a passenger who drives manually, for example.

FIG. 11 is a diagram showing another example of the configuration of the moving object detection device 100 and peripheral devices. In FIG. 11 , moving object detection device 100 is configured to communicate with notification device 210 instead of travel control device 200 . Other configurations are the same as those shown in FIG.

The notification device 210 is a display device, a speaker, a vibrator, a light-emitting device, or the like for outputting information to the occupants of the vehicle M, and notifies the occupants of the vehicle M of information that a moving object exists in front of the vehicle M. do. The notification device 210 is an example of a “driving assistance device”.

FIG. 12 is a diagram showing an example of operations performed by the notification device 210. As shown in FIG. FIG. 12 shows a case where the notification device 210 functions as a navigation device for the vehicle M as an example. As shown in FIG. 12, when the grid extraction unit 130 detects a moving object, the notification device 210 displays the detected moving object on the screen by, for example, surrounding the detected moving object with a bounding box BX.

At this time, the notification device 210 may further display a warning message W to the effect that a moving object exists in front of the vehicle M, or notify information to the effect that a moving object exists in front of the vehicle M by voice. You may Through such processing, useful driving assistance information can be provided to the passenger who drives the vehicle M manually.

Next, with reference to FIG. 13, the flow of processing executed by the moving object detection device 100 will be described. FIG. 13 is a diagram showing an example of the flow of processing executed by the moving object detection device 100. As shown in FIG.

First, the image acquisition unit 110 acquires an image frame representing the surrounding situation of the vehicle M captured by the camera 10 as the current image frame (step S100). Next, the difference calculation unit 120 calculates the vanishing point VP of the previous image frame acquired immediately before the current image frame based on the speed of the vehicle M between the previous image frame and the current image frame. Enlarge as the center and match the size of the current image frame by trimming the edges of the enlarged previous image frame (step S102).

Next, the difference calculator 120 calculates a difference image between the previous image frame and the current image frame (step S104). More specifically, the difference calculation unit 120 calculates a pixel difference value for each pixel between the previous image frame and the current image frame, and if the calculated difference value is equal to or greater than a specified value, the pixel is assigned a first value, while the pixel is assigned a second value if the calculated difference value is less than a specified value.

Next, the grid extraction unit 130 sets a grid G for each of a plurality of pixels in the calculated difference image, and selects a grid G in which the density of pixels having the first value among the pixels in the grid G is equal to or greater than the threshold. By extracting, the grid image GI is calculated (step S106). Next, the moving object detection unit 140 detects the grid G represented in the grid image GI as a moving object (step S108). Next, the travel control device 200 controls travel of the vehicle M so as to avoid the moving object detected by the moving object detection unit 140 (step S110). This completes the processing of this flowchart.

According to the first embodiment described above, a difference image is calculated from time-series image frames captured by a camera, and grids having different sizes are set for each of a plurality of pixels in the calculated difference image. , to detect the presence or absence of a moving object for each set grid. As a result, it is possible to easily and quickly detect a moving object existing around the vehicle.

[Second embodiment]
1st Embodiment detects the grid G shown by the grid image GI calculated from the difference image DI as a moving body. However, the grid G shown in the grid image GI is not necessarily all moving objects, and may include stationary objects such as pedestrian crossings. The moving object detection device 100 according to the second embodiment increases the accuracy of moving object detection by comparing a plurality of calculated grid images GI. The functional configuration of the moving object detection device 100 according to the second embodiment is the same as that of the first embodiment, and thus the description is omitted.

In the second embodiment, the moving object detection unit 140 detects a moving object by comparing multiple grid images obtained at different time points in order to more accurately detect the moving object from the grid images. FIG. 14 is a diagram for explaining operations performed by the moving object detection unit 140 according to the second embodiment. The upper part of FIG. 14 shows the grid image GI1 calculated from the difference image between time t0 and time t1 after time t0, and the lower part of FIG. A grid image GI2 calculated from the difference image between time t2 (for example, the current time) is shown. References G1 and G2 respectively denote grids in which the density of pixels of the first value was determined to be greater than or equal to the threshold. Time t0 is an example of a "first time", time t1 is an example of a "second time", and time t3 is an example of a "third time".

The moving object detection unit 140 detects a moving object existing around the vehicle M by comparing the grid G1 (G2) in the grid image GI1 and the grid G1 (G2) in the grid image GI2. More specifically, moving object detection unit 140 first acquires information about the speed and yaw rate of vehicle M during the period between time t1 and time t2. Next, the moving object detection unit 140 identifies the position of the grid image GI2 corresponding to the grid G in the grid image GI1 based on the acquired information about the speed and yaw rate. Next, the moving object detection unit 140 compares the grid G in the grid image GI1 with the grid G existing at the position of the specified grid image GI2, and the shapes and the pixel densities of the first values match ( or similar), it is determined that these grids G show the same object. FIG. 14 shows an example in which the grid G1 in the grid image GI1 corresponds to the grid G1 in the grid image GI2, and the grid G2 in the grid image GI1 corresponds to the grid G2 in the grid image GI2.

Next, the moving object detection unit 140 determines whether or not the grid G in the grid image GI2 has moved toward the center of the image with reference to the corresponding grid G in the grid image GI1. When the moving object detection unit 140 determines that the grid G in the grid image GI2 has moved toward the center of the image, the moving object detection unit 140 detects the grid G as a moving object. In the example of FIG. 14, the moving object detection unit 140 determines that the grid G1 in the grid image GI2 has moved toward the center of the image (that is, toward the lower right) with respect to the grid G1 in the grid image GI1. is detected as a moving object. On the other hand, as shown in FIG. 14, the grid G2 existing in the center of the image has not moved toward the center of the image, so the moving object detection unit 140 does not detect the grid G2 as a moving object. However, in general, an object existing in the center of the image tends to be underestimated in the apparent amount of movement before and after the frame. may be detected as a certain monitored object.

Next, with reference to FIG. 15, the flow of processing executed by the moving object detection device 100 will be described. FIG. 15 is a diagram showing an example of the flow of processing executed by the moving object detection device 100 according to the second embodiment. The processing up to step S106 is the same as that of the flowchart of the first embodiment shown in FIG. 13, so the description is omitted.

In step S106, when the grid extraction unit 130 calculates the current grid image GI, the moving object detection unit 140 acquires the previous grid image GI calculated one cycle before (step S200). Next, the moving object detection unit 140 identifies the grid G to be compared based on the speed and yaw rate of the vehicle M between the calculation of the previous grid image GI and the calculation of the current grid image GI. (Step S202).

Next, the moving object detection unit 140 determines whether or not the grid G in the current grid image GI has moved toward the center of the image with reference to the corresponding grid G in the previous grid image GI (step S204). When the moving object detection unit 140 determines that the grid G in the current grid image GI has moved toward the center of the image with reference to the corresponding grid G in the previous grid image GI, the moving object detection unit 140 detects the grid G as a moving object ( step S206).

On the other hand, if the moving object detection unit 140 determines that the grid G in the current grid image GI has not moved in the image center direction with respect to the corresponding grid G in the previous grid image GI, It is detected as a moving object (step S208). Next, the travel control device 200 controls travel of the vehicle M so as to avoid the moving object detected by the moving object detection unit 140 (step S210). This completes the processing of this flowchart.

According to the second embodiment described above, the corresponding grid between the grid image calculated at the previous time and the grid image calculated at the current time is specified based on the speed and yaw rate of the vehicle M, If the specified grid moves toward the center of the vehicle with reference to the previous time, the grid is detected as a moving object. Thereby, a moving object can be detected more accurately from the grid image.

[Third Embodiment]
1st Embodiment detects the grid G shown by the grid image GI calculated from the difference image DI as a moving body. However, the grid G shown in the grid image GI is not necessarily all moving objects, and may include stationary objects such as pedestrian crossings. The moving object detection apparatus 100 according to the third embodiment compares the specified size of the object (pedestrian, motorcycle, vehicle, etc.) to be detected with the grid G shown in the grid image GI, thereby increasing the detection accuracy of the moving object. It is intended to increase

FIG. 16 is a diagram showing an example of the configuration and peripheral devices of the moving object detection device 100 according to the third embodiment. As shown in FIG. 16, the moving object detection device 100 includes a bounding box setting unit 132 in addition to the components of the moving object detection device 100 according to the first embodiment. The bounding box setting unit 132 searches for a set of grids G that are extracted by the grid extraction unit 130 and that satisfies a predetermined criterion, and sets a bounding box for the set of grids G found.

FIG. 17 is a diagram showing an example of a grid G search method executed by the bounding box setting unit 132. As shown in FIG. The bounding box setting unit 132 first searches the grid image GI calculated by the grid extracting unit 130 for a set of grids G whose lower ends have a certain length L1 or longer. At this time, as shown in the left part of FIG. 17, in order for the set of grids G to be determined to have a lower end equal to or longer than the constant length L1, the set must necessarily include the grids G without loss. Instead, it may be determined that the lower end has a length L1 or longer on the condition that the density of the grid G included in the lower end is equal to or greater than a reference value.

Next, when the bounding box setting unit 132 identifies a set of grids G having a lower end of a certain length L1 or more, the bounding box setting unit 132 determines whether the set of grids G has a height of a certain length L2 or more. That is, by determining whether or not the set of grids G has a lower end of a certain length L1 or more and a height of a certain length L2 or more, the set of grids G can be regarded as an object such as a pedestrian, a motorcycle, or a vehicle. It is possible to specify whether or not it is applicable. In this case, the combination of the fixed length L1 of the lower end and the fixed length L2 of the height is set as a unique value for each object such as a pedestrian, a motorcycle, and a vehicle.

Next, when the bounding box setting unit 132 identifies a set of grids G having a lower end of a certain length L1 or more and a height of a certain length L2 or more, the bounding box setting unit 132 sets a bounding box for the set of grids G. FIG. Next, the bounding box setting unit 132 determines whether or not the density of the grid G included in the set bounding box is equal to or greater than a threshold. The bounding box setting unit 132 detects the bounding box as a moving object when it is determined that the density of the grid G included in the set bounding box is equal to or greater than the threshold. By performing determinations on bounding box settings and densities, the identified set of grids G is a real object (in other words, even though the set of grids G is not an object, it happens to be defined It is possible to confirm that it does not correspond to the bottom end and height).

In this way, the bounding box setting unit 132 searches for a set of grids G satisfying a predetermined criterion from the grid image GI, but the search for the grids G may increase the processing load. Therefore, the bounding box setting unit 132 may preferentially search for areas important for traveling of the vehicle M in order to reduce the processing load associated with searching the grid G. FIG.

FIG. 18 is a diagram showing an example of the search range of the grid G by the bounding box setting unit 132. As shown in FIG. In FIG. 18, reference R1 indicates the range in which the grid G is searched at a short distance from the vehicle M, reference R2 indicates the range in which the grid G is searched at a medium distance from the vehicle M, and reference R3 indicates the vehicle M. indicates the range over which the grid G is searched at long distances from . As shown in FIG. 18, the bounding box setting unit 132 searches the entire area of the grid image GI in the short distance range R1 from the vehicle M, while in the medium distance range R1 and the long distance range R2 from the vehicle M, Only the central portion of the grid image GI may be searched. As a result, it is possible to preferentially search for areas important for the travel of the vehicle M, and reduce the processing load associated with searching the grid G. FIG.

Note that FIG. 18 describes, as an example, a case where the search range of the grid G is divided into three stages, but the present invention is not limited to such a configuration. For example, the bounding box setting unit 132 may search the grid G in a search range of two stages or four or more stages, or may search the grid G in such a manner that the search width of a single search range is continuously narrowed down to the central part. You may search for G.

Next, with reference to FIG. 19, the flow of processing executed by the moving object detection device 100 will be described. FIG. 19 is a diagram showing an example of the flow of processing executed by the moving object detection device 100 according to the third embodiment. The processing up to step S106 is the same as that of the flowchart of the first embodiment shown in FIG. 13, so the description is omitted.

When the grid extracting unit 130 calculates the grid image GI in step S106, the bounding box setting unit 132 searches the grid image GI for a set of grids G having a lower end of a certain length L1 or longer (step S300). If the grid image GI does not find a set of grids G having lower ends with a length equal to or greater than L1, the bounding box setting unit 132 terminates the processing of this flowchart.

On the other hand, when a set of grids G having a lower end of a certain length L1 or more is searched for in the grid image GI, the bounding box setting unit 132 sets a certain length L2 or more based on the lower end of the search of the set of grids G. (step S304). If it is not determined that the searched set of grids G has a height equal to or greater than the fixed length L2, the bounding box setting unit 132 terminates the processing of this flowchart.

On the other hand, when it is determined that the searched set of grids G has a height equal to or greater than the constant length L2, the bounding box setting unit 132 sets a bounding box surrounding the set of grids G (step S306). Next, the bounding box setting unit 132 determines whether or not the density of the grid G within the set bounding box is equal to or greater than a threshold (step S308). If it is not determined that the density of the grid G within the set bounding box is equal to or greater than the threshold, the bounding box setting unit 132 terminates the processing of this flowchart.

On the other hand, when it is determined that the density of the grid G within the set bounding box is equal to or greater than the threshold, the moving object detection unit 140 detects the bounding box as a moving object (step S310). Next, the travel control device 200 controls travel of the vehicle M so as to avoid the moving object detected by the moving object detection unit 140 (step S210). This completes the processing of this flowchart.

According to the third embodiment described above, a grid image that satisfies a predetermined criterion is searched for, a bounding box is set for the searched grid set, and the density of the set bounding box is a threshold value. A moving object is detected based on whether it is the above. Thereby, a moving object can be detected more accurately from the grid image.

The embodiment described above can be expressed as follows.
a storage device storing a program;
a hardware processor;
By the hardware processor executing the program stored in the storage device,
Acquiring image data including a plurality of frames representing a surrounding situation of the moving object, captured in time series by a camera mounted on the moving object;
calculating a difference image between the plurality of frames by calculating a difference between the plurality of frames and binarizing the difference into a first value and a second value;
Extracting a grid image from the difference image by extracting a grid having a pixel density of the first value equal to or higher than a first threshold from among a plurality of grids set in the difference image;
setting a bounding box for one or more grids included in the grid image and satisfying predetermined criteria;
detecting the bounding box as a moving object when the density of the grid in the set bounding box is equal to or greater than a second threshold;
A moving body detection device configured to:

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

REFERENCE SIGNS LIST 10 camera 100 moving object detection device 110 image acquisition unit 120 difference calculation unit 130 grid extraction unit 132 bounding box setting unit 140 moving object detection unit 200 travel control device 210 notification device

Claims (11)

an image acquisition unit that acquires image data including a plurality of frames representing a surrounding situation of the moving object, which are captured in time series by a camera mounted on the moving object;
a difference calculation unit that calculates a difference image between the plurality of frames by calculating a difference between the plurality of frames and binarizing the difference into a first value and a second value;
a grid extracting unit for extracting a grid having a pixel density of the first value equal to or higher than a first threshold among the plurality of grids set in the difference image;
a bounding box setting unit that sets a bounding box that is a circumscribing rectangle for the one or more grids that are extracted and satisfy a predetermined criterion;
a moving object detection unit that detects the bounding box as a moving object when the density of the grid in the set bounding box is equal to or greater than a second threshold;
Motion detector. The bounding box setting unit is configured such that the one or more grids satisfy the predetermined criterion when the bottom edge of the one or more grids and the height of the one or more grids relative to the bottom edge each have a constant value. determine that
The moving body detection device according to claim 1. The bounding box setting unit searches for one or more grids that satisfy the predetermined criteria in a central portion of a grid image containing the grids.
The moving body detection device according to claim 1 or 2. The difference calculation unit enlarges the frame imaged at the previous time based on the speed of the moving body in the imaging interval at which the plurality of frames are imaged, and compares the enlarged frame imaged at the previous time with the current image. calculating a difference image between the frames captured at the point in time;
The moving body detection device according to any one of claims 1 to 3. The difference calculation unit enlarges the frame captured at the previous time, centering on the vanishing point of the frame captured at the previous time.
The moving body detection device according to claim 4. The difference calculation unit corrects the frame captured at the previous time point based on the yaw rate of the moving body at the shooting interval at which the plurality of frames are captured, and calculates the Calculate the difference image between the frames captured in
The moving body detection device according to any one of claims 1 to 5. The grid extraction unit changes the threshold according to the distance from the camera to each of the plurality of grids.
The moving body detection device according to any one of claims 1 to 6. The grid extraction unit sets the size of the plurality of grids to a first size when the distance from the camera is less than or equal to a first distance, and sets the size of the plurality of grids to a first size, If the distance is less than or equal to the distance, setting the size of the plurality of grids to a second size smaller than the first size, and if the distance from the camera is greater than the second distance, setting the size of the plurality of grids. set to a third size smaller than the second size;
The moving body detection device according to any one of claims 1 to 7. A moving object detection device according to any one of claims 1 to 8;
A driving support device that supports driving of the moving body based on the detection result obtained by the moving body detection device,
system. the computer
Acquiring image data including a plurality of frames representing a surrounding situation of the moving object, captured in time series by a camera mounted on the moving object;
calculating a difference image between the plurality of frames by calculating a difference between the plurality of frames and binarizing the difference into a first value and a second value;
Extracting a grid image from the difference image by extracting a grid having a pixel density of the first value equal to or higher than a first threshold from among a plurality of grids set in the difference image;
setting a bounding box for one or more grids included in the grid image and satisfying predetermined criteria;
detecting the bounding box as a moving object when the density of the grid in the set bounding box is equal to or greater than a second threshold;
Motion detection method. to the computer,
Acquiring image data including a plurality of frames representing a surrounding situation of the moving object, which are captured in time series by a camera mounted on the moving object;
calculating a difference image between the plurality of frames by calculating a difference between the plurality of frames and binarizing the difference into a first value and a second value;
extracting a grid image from the difference image by extracting a grid having a pixel density of the first value equal to or higher than a first threshold from among a plurality of grids set in the difference image;
setting a bounding box for one or more grids included in the grid image and satisfying a predetermined criterion;
detecting the bounding box as a moving object when the density of the grid in the set bounding box is equal to or greater than a second threshold;
program.

Images