JP6593018B2 - Obstacle detection device, obstacle detection method, and obstacle detection program - Google Patents

Description

  The present invention relates to a technique for processing a moving image obtained by imaging a detection target area defined on a road on which a vehicle travels and detecting an obstacle that hinders travel of the vehicle.

  Conventionally, there is a background difference method as one of image processing techniques for processing a frame image obtained by imaging a detection target area and detecting an object (an object imaged in the frame image) located in the detection target area. As is well known, the background subtraction method generates a difference image (background difference image) between a frame image obtained by imaging a detection target area and a background image of the detection target area, thereby generating a frame image obtained by imaging the detection target area. This is an image processing technique for detecting an imaged object (an object located in a detection target area).

  In the background subtraction method, when the detection target area is outdoors, shadows of surrounding structures that appear due to weather changes or lighting changes, puddles of road surfaces due to rainfall, snow accumulation due to snowfall, and the like may be erroneously detected as objects. For example, when the detection target area is on the road on which the vehicle travels, the shadow of surrounding structures, the puddle of the road surface due to rain, the snow accumulation due to snowfall, etc., are erroneously detected as obstacles that hinder the vehicle's travel There is.

  Patent Document 1 proposes a technique for suppressing this erroneous detection. Patent Document 1 is based on the assumption that changes in the shadows of surrounding structures due to weather changes and lighting changes, puddles of road surfaces due to rain, snow accumulation due to snow, etc., occur uniformly in the direction of travel of the vehicle on the road surface. It is a configuration in which a change that occurs uniformly in the traveling direction of the vehicle on the road surface is not detected as an object (obstacle).

JP 2010-191888 A

  However, in Patent Document 1, as described above, changes in shadows of surrounding structures due to weather changes and lighting changes, road surface puddles due to rain, snow accumulation due to snowfall, etc. occur uniformly in the vehicle running direction on the road surface. It is based on the premise of doing. Therefore, Patent Document 1 cannot prevent erroneous detection of changes such as shadows of surrounding structures, puddles of road surfaces due to rain, snow accumulation due to snow, etc., which did not occur uniformly in the traveling direction of the vehicle as obstacles. .

  An object of the present invention is to improve the detection accuracy of an obstacle by suppressing false detection of an obstacle caused by a change in weather or lighting for a detection target area defined on a road on which a vehicle travels. It is to propose a technology that can.

  The obstacle detection device of the present invention is configured as follows in order to solve the above problems.

  A moving image obtained by imaging a detection target area defined on the road on which the vehicle travels is input to the moving image input unit. The foreground pixel determination unit processes the frame image related to the moving image input to the moving image input unit, and uses the feature amount of each pixel of the frame image to determine whether the foreground pixel or the background pixel. Determine if there is. As the feature amount of the pixel, an edge strength, a luminance value, or the like may be used. The foreground pixel determination unit determines, for each pixel of the frame image, a pixel whose absolute value of the difference from the feature amount of the corresponding background pixel is greater than or equal to a predetermined threshold value, and a pixel that is less than this threshold value is a background pixel. do it.

  The specific pixel determination unit uses the imaging timing of the frame image changed from the background pixel to the foreground pixel as a reference timing for a pixel whose determination result from the foreground pixel determination unit has changed from the background pixel to the foreground pixel, and is determined based on this reference timing. It is determined whether the pixel is a specific pixel related to the obstacle using the frame image captured during the determination period.

  The feature amount of the pixel related to the object that suddenly appears in the detection target area changes abruptly, but the feature amount of the pixel related to the shadow of the surrounding structure due to weather change or illumination change changes gradually. Therefore, by determining whether the change in the feature value before and after the reference timing is abrupt or gradual, whether the pixel is applied to the object, whether it is a shadow of a surrounding structure due to weather change or illumination change, etc. It is possible to determine whether the pixel is related to.

  In addition, the feature amount of a pixel related to a puddle whose shape has changed due to traveling of the vehicle changes rapidly. However, the feature amount of the pixel related to the puddle whose shape has changed due to the traveling of the vehicle is not stable over time. Therefore, it is possible to determine whether the pixel is a pixel related to a puddle whose shape has changed due to traveling of the vehicle or a pixel related to the object from the change in the feature amount in the period after the reference timing.

  Even when the object is a traveling vehicle that has passed through the detection target area, the feature amount of the pixel relating to the traveling vehicle is not stable in time.

  The obstacle detection unit detects an object including a pixel determined as a specific pixel by the specific pixel determination unit as an obstacle.

  As a result, it is possible to suppress erroneous detection of shadows of surrounding structures due to weather changes or lighting changes, water pools whose shape has changed due to the vehicle traveling, traveling vehicles, and the like as obstacles that impede vehicle travel.

  The obstacle detection unit may include an image region in which a distribution density of the specific pixels determined by the specific pixel determination unit is higher than a predetermined determination density in a predetermined number of frame images having the reference timing as the center of the imaging timing. For example, it is preferable that the object imaged in this image area is detected as an obstacle. In this way, it is possible to suppress erroneous detection of pixels associated with small dust or noise as an obstacle.

  According to the present invention, since the erroneous detection of the obstacle due to the weather change or the illumination change is suppressed with respect to the detection target area defined on the road on which the vehicle travels, the obstacle detection accuracy is improved. Can do.

It is a schematic block diagram which shows an obstacle detection system. It is a block diagram which shows the structure of the principal part of an obstruction detection apparatus. It is a figure which shows the function structure of an image process part. It is a figure which shows a process target frame image. It is a figure which shows the edge image produced | generated by processing the process target frame image shown in FIG. It is a figure which shows a background edge image. It is a figure which shows the difference image of the edge image shown in FIG. 5, and the background edge image shown in FIG. It is a flowchart which shows a change pixel extraction process. It is a flowchart which shows a 1st specific pixel determination process. FIG. 10A is a diagram illustrating a change in the edge strength of a pixel applied to a falling object, and FIG. 10B is a diagram illustrating a change in the edge strength of the pixel due to a shadow. It is a flowchart which shows a 2nd specific pixel determination process. FIG. 12A is a diagram illustrating a change in the edge strength of the pixel related to the falling object, and FIG. 12B is a diagram illustrating a change in the edge strength of the pixel related to the shape change of the puddle. It is a flowchart which shows an obstruction confirmation process. It is a flowchart which shows a continuation determination process. It is a flowchart which shows a driving | running | working abnormality determination process. It is a flowchart which shows the control notification necessity determination process.

  Embodiments of the present invention will be described below.

  FIG. 1 is a block diagram showing the configuration of the main part of the obstacle detection system according to this example. The obstacle detection system according to this example includes an obstacle detection device 1, a video camera 2, a guidance display board 3, and a control device 4. This obstacle detection system detects an obstacle located in a detection target area defined on a road on which the vehicle travels, and notifies that an obstacle has been detected. An obstacle is an article that hinders the traveling of the vehicle, such as a load (falling object) dropped from the traveling vehicle, a vehicle stopped on the road (stopped vehicle), or the like. Vehicles traveling on the road and structures such as guardrails and center poles installed on the road are not included in the obstacles in this example.

  The video camera 2 is attached so as to image a detection target area defined on the road from several m (about 6 m) above. For example, the video camera 2 is attached to a pole installed on the roadside, a gantry installed so as to straddle the road, and the like. Further, the video camera 2 adjusts the angle and the imaging magnification so that the detection target area falls within the imaging area. The video camera 2 captures a visible light image. The frame rate of the video camera 2 may be about 5 to 30 frames / sec. The video camera 2 inputs a frame image related to the captured moving image of the imaging area to the obstacle detection device 1.

  Note that the video camera 2 may be a far-infrared camera that captures a far-infrared image, a near-infrared camera that captures a near-infrared image, or the like.

  The obstacle detection apparatus 1 processes a frame image related to a moving image input from the video camera 2 and detects an obstacle located in the detection target area. When the obstacle detection device 1 detects an obstacle located in the detection target area, the obstacle detection device 1 outputs a request notification for requesting a notification accompanying the detection of the obstacle to the guidance display board 3 and the control device 4.

  The guidance display board 3 displays guidance messages relating to road conditions and the like to the driver of the traveling vehicle. The guidance display board 3 is installed on the upstream side of the detection target area. The guidance display board 3 is installed so that the driver can confirm the displayed message without hindering the driving of the vehicle. When a request notification requesting notification associated with the detection of an obstacle (corresponding to the first notification referred to in the present invention) is input from the obstacle detection device 1 to the guidance display board 3, the obstacle is displayed downstream. Display a message notifying the driver that there is something.

  The control device 4 is installed in a control center that manages roads. When the control device 4 receives a request notification (corresponding to the second notification, referred to in the present invention) requesting a notification accompanying the detection of an obstacle from the obstacle detection device 1, the traffic control device 4 To prompt the user to remove the obstacle (detected obstacle) on the road.

  Although the details will be described later, the obstacle detection device 1 notifies the guidance display board 3 of the notification requesting the obstacle detection and the control device 4 the notification accompanying the obstacle detection. The timing required is different.

  FIG. 2 is a block diagram illustrating a configuration of a main part of the obstacle detection device. The obstacle detection device 1 includes a control unit 11, an image input unit 12, an image processing unit 13, and an output unit 14.

  The control unit 11 controls the operation of each part of the obstacle detection apparatus 1 main body.

  A moving image captured by the video camera 2 is input to the image input unit 12. The image input unit 12 has an image memory (not shown) that temporarily stores a frame image related to the input moving image. The image input unit 12 corresponds to the moving image input unit referred to in the present invention.

  The image processing unit 13 processes the frame image input to the image input unit 12 (the frame image related to the moving image captured by the video camera 2), and detects an obstacle located in the detection target area. The image processing unit 13 executes the obstacle detection method according to the present invention. Further, the image processing unit 13 has an image processing processor (computer) for executing each step of the obstacle detection program according to the present invention. Details of the image processing unit 13 will be described later.

  The output unit 14 outputs a request notification for requesting notification accompanying detection of an obstacle located in the detection target area to the guidance display board 3 and the control device 4.

  FIG. 3 is a diagram illustrating a functional configuration of the image processing unit. The image processing unit 13 includes a processing target frame image selection function unit 21, an edge image generation function unit 22, a background difference image generation function unit 23, an appearance determination function unit 24, a change determination function unit 25, and an obstacle detection. The function part 26, the obstruction continuation detection function part 27, the vehicle travel condition determination function part 28, and the control notification necessity determination function part 29 are provided.

The processing target frame image selection function unit 21 performs a frame image (processing target frame) for performing image processing, which will be described later, from among the frame images input to the image input unit 12 (frame images related to moving images captured by the video camera 2). Image). The processing target frame image selection function unit 21 may be configured to select all frame images related to the moving image input to the image input unit 12 as processing target frame images, or may be input to the image input unit 12. A configuration may be used in which a processing target frame image is selected at predetermined frame intervals or time intervals from frame images related to a moving image. The relationship between the frame interval of the processing target frame image and the time interval of the processing target frame image is
Frame interval of processing target frame image =
(Frame rate of video camera 2) x (time interval of processing target frame image)
It is.

  In this example, the frame rate of the video camera 2 is 10 frames / sec, and the processing target frame image selection function unit 21 sets all the frame images related to the moving image input to the image input unit 12 as processing target frame images. It will be described as a selection. If the frame rate of the video camera 2 is 30 frames / sec, the processing target frame image selection function unit 21 selects the processing target frame images at intervals of 3 frames or at intervals of 100 ms. A frame image captured at the same timing can be selected as a processing target frame image.

  The edge image generation function unit 22 generates an edge image of the processing target frame image selected by the processing target frame image selection function unit 21. The edge image generation function unit 22 uses a known edge extraction filter (for example, Sobel filter, Prewitt filter, Laplacian filter) to generate an edge image in which the pixel value of each pixel of the processing target frame image is the edge strength. . 4 is a diagram showing a processing target frame image, and FIG. 5 is a diagram showing an edge image generated by processing the processing target frame image shown in FIG. The edge image generated by the edge image generation function unit 22 has, for example, 256 gradations. In this example, the feature amount of the pixel referred to in the present invention is the edge strength.

  The background difference image generation function unit 23 generates a difference image (so-called background difference image) between the edge image generated by the edge image generation function unit 22 and a background edge image stored in an image memory (not shown). . The background edge image is an edge image of the background of the imaging area of the video camera 2. FIG. 6 is a diagram illustrating a background edge image. FIG. 7 is a diagram showing a background difference image generated by the edge image shown in FIG. 5 and the background edge image shown in FIG. The background difference image generation function unit 23 calculates, for each pixel of the edge image generated by the edge image generation function unit 22, the absolute value of the difference between the edge strength of the pixel and the edge strength of the corresponding pixel of the background edge image. A binary image having a foreground pixel (pixel value = 1) if it is equal to or greater than a predetermined threshold and a background pixel (pixel value = 0) if it is less than the threshold is generated as a background difference image. The background difference image generation function unit 23 corresponds to the foreground pixel determination unit referred to in the present invention. Specifically, the function of determining whether each pixel of the edge image generated by the edge image generation function unit 22 is a foreground pixel or a background pixel corresponds to the foreground pixel determination unit referred to in the present invention.

  The background difference image generation function unit 23 obtains a value obtained by subtracting the edge intensity of the corresponding pixel of the background edge image from the edge intensity of each pixel of the edge image generated by the edge image generation function unit 22 in advance. A binary image may be generated as a background difference image if it is a foreground pixel if it is greater than or equal to a predetermined threshold and if it is less than the threshold, a background pixel. In this configuration, a pixel in which an edge appears in the background edge image (a pixel having a relatively high edge strength) and a pixel in which an edge does not appear in the edge image generated by the edge image generation function unit 22 (the edge strength is compared) Smaller pixel) is determined as the background pixel.

  The appearance determination function unit 24 determines whether the pixel determined to be the foreground pixel by the background difference image generation function unit 23 is a pixel related to a shadow of a surrounding structure due to a weather change or a lighting change. The appearance determination function unit 24 performs the first specific pixel determination process referred to in the present invention. Details of the processing in the appearance determination function unit 24 will be described later.

  The change determination function unit 25 applies a pixel that is determined to be a foreground pixel by the background difference image generation function unit 23 to a vehicle that has passed through the detection target area, a puddle whose shape has changed due to traveling of the vehicle, or the like. It is determined whether it is a pixel. The change determination function unit 25 performs the second specific pixel determination process referred to in the present invention. Details of the processing in the change determination function unit 25 will be described later.

  The specific pixel determination unit referred to in the present invention includes an appearance determination function unit 24 and a change determination function unit 25.

  The obstacle detection function unit 26 detects an obstacle located on the road in the detection target area using the determination results of the appearance determination function unit 24 and the change determination function unit 25. The obstacle detection function unit 26 corresponds to the obstacle detection unit referred to in the present invention.

The obstacle continuation detection function unit 27 detects whether the obstacle detected by the obstacle detection function unit 26 continues to be positioned without moving over a predetermined continuation determination time (for example, 1 minute). The obstacle detection function unit 26 also detects a plastic bag that has been blown by the wind, a vehicle that is temporarily stopped for some reason, and the like as an obstacle. The obstacle continuation detection function unit 27 detects whether the obstacle detected by the obstacle detection function unit 26 is a shopping bag that has been blown by the wind or a vehicle that has been temporarily stopped for some reason. The obstacle detection function unit 26 corresponds to a movement detection unit referred to in the present invention.

  Note that a fallen object can be discriminated from a stopped vehicle depending on its size.

  The vehicle traveling state determination function unit 28 compares the traveling pattern (deceleration, acceleration, meandering, etc.) of the vehicle traveling in the detection target area with the traveling pattern of the vehicle when the traffic flow is normal, and detects it. It is determined whether or not the traveling pattern of the vehicle traveling in the target area is unnatural. The vehicle travel state determination function unit 28 corresponds to the travel state determination unit referred to in the present invention.

  The control notification necessity / unnecessity determination function unit 29 requests the control device 4 to be notified of an obstacle based on the detection result of the obstacle continuation detection function unit 27 and the determination result of the vehicle running state determination function unit 28. To determine whether to output a request notification.

  When the obstacle detection function unit 26 detects an obstacle located on the road, the obstacle detection device 1 outputs that fact from the output unit 14 to the guidance display board 3. At this time, the obstacle detection device 1 does not output to the control device 4 that an obstacle located on the road has been detected. When the obstacle detection device 1 determines that the control notification necessity determination function unit 29 notifies the control device 4 of the detection of the obstacle, the obstacle detection device 1 outputs that fact to the control device 4 from the output unit 14.

  Hereinafter, the operation of the obstacle detection apparatus 1 according to this example will be described. The obstacle detection apparatus 1 according to this example includes a change pixel extraction process, a first specific pixel determination process, a second specific pixel determination process, an obstacle determination process, a continuation determination process, a traffic situation determination process, and A control notification necessity determination process is performed.

  FIG. 8 is a flowchart showing the change pixel extraction process. In the obstacle detection device 1, a frame image related to a moving image captured by the video camera 2 is input to the image input unit 12. The obstacle detection apparatus 1 selects a processing target frame image from the frame images related to the moving image input to the image input unit 12 (s1). In s1, a frame number is assigned to the selected processing target frame image. The frame number may be any number that allows the processing target frame images to be arranged in time series at the imaging timing. For example, the frame number may be a serial number or a number generated by time information indicating the imaging timing. The processing target frame image selection function unit 21 performs the process related to s1.

  The image processing unit 13 generates an edge image of the processing target frame image selected in s1 (s2). In s2, the edge image generation function unit 22 uses an edge extraction filter such as a Sobel filter, a Prewitt filter, and a Laplacian filter, and the edge image (see FIG. 5) of the processing target frame image (see FIG. 4) selected this time in s1. Is generated. The pixel value of the edge image generated in s2 is 256 gradations. The pixel value is the edge strength. The edge image generated in s2 is stored in an image memory (not shown) in association with the frame number. This image memory has a capacity capable of storing edge images for several minutes.

  The image processing unit 13 generates a background difference image (see FIG. 7) that is a difference image between the edge image generated in s2 and a background edge image (see FIG. 6) stored in an image memory (not shown). (S3). In s3, a binary image is generated as a background difference image in which the absolute value of the difference in edge strength is a foreground pixel that is a predetermined threshold or more and a background pixel that is less than the threshold. In FIG. 7, pixels shown in white are foreground pixels, and pixels shown in black are background pixels.

  The image processing unit 13 extracts all pixels that are determined as background pixels in the previously selected processing target frame image and are determined as foreground pixels in the processing target frame image selected this time (hereinafter referred to as change pixels). The changed pixels extracted here are stored in a storage unit (not shown) (s4), and the process returns to s1. In s4, the change pixel extracted this time and the frame number of the current processing target frame image are associated with each other and stored in the storage unit. The background difference image generation function unit 23 performs processing relating to s3 and s4.

  The background difference image generation function unit 23 has a function of updating the background edge image stored in the image memory using the edge image generated by the edge image generation function unit 22.

  Next, the first specific pixel determination process will be described. FIG. 9 is a flowchart showing the first specific pixel determination process. The appearance determination function unit 24 of the image processing unit 13 performs a first specific pixel determination process. In the first specific pixel determination process, the edge image stored in the image memory in s2 and the change pixel stored in the storage unit in s4 are used. This appearance determination process is a process for determining whether or not the changed pixel extracted in s4 is a pixel related to a shadow of a surrounding structure that has appeared due to a weather change or a lighting change.

  The appearance determination function unit 24 selects a processing target frame image (s11). In s11, the frame image on which the above-described change pixel extraction processing is performed is selected as a processing target frame image in time series. The processing according to s11 is not performed in the processing target frame image selection function unit 21, but is performed in the appearance determination function unit 24. Further, in the first specific pixel determination process, the processing result of the change pixel extraction process for the frame image captured after the appearance determination time described later is used rather than the processing target frame image selected in s11. That is, the processing result of the first specific pixel determination process is at least delayed in appearance determination time than the processing result of the change pixel extraction process described above.

  The appearance determination function unit 24 extracts an appearance determination target pixel in the processing target frame image selected this time in s11 (s12). In s12, the change pixel extracted by the process concerning s4 mentioned above with respect to the process target frame image selected in s11 is extracted as an appearance determination target pixel. As described above, the storage unit stores the frame number of the processing target frame image and the change pixel extracted from the processing target frame image in association with each other. The number of appearance determination target pixels extracted in s12 may be 0, or may be 1 or more.

  The appearance determination function part 24 performs the process concerning s13-s17 shown below for every appearance determination object pixel extracted by s12.

  The appearance determination function unit 24 calculates, for each appearance determination target pixel, the average edge intensity (the processing target imaged during this period) in the appearance determination time (for example, 2 seconds) immediately before the imaging timing of the processing target frame image selected this time. An average of edge strengths of corresponding pixels in the frame image is calculated (s13). In addition, the appearance determination function unit 24 calculates the average edge intensity (imaged during this period) for each appearance determination target pixel in the appearance determination time (for example, 2 seconds) immediately after the imaging timing of the processing target frame image selected this time. An average of edge strengths of corresponding pixels in the processing target frame image is calculated (s14). The appearance determination time immediately before the imaging timing of the processing target frame image selected this time corresponds to the first period referred to in the present invention. The appearance determination time immediately after the imaging timing of the processing target frame image selected this time corresponds to the second period referred to in the present invention. In this example, the length of the period for calculating the average edge strength at s13 (the first period in the present invention) and the length of the period for calculating the average edge intensity in s14 (the second period in the present invention). Are the same, but may be of different lengths. The imaging timing of the processing target frame image selected this time corresponds to the reference timing in the present invention.

  In s13 and s14, the edge image of the processing target frame image used for calculating the average edge strength is stored in the image memory in s2.

The appearance determination function unit 24 calculates, for each appearance determination target pixel, an average ratio α of the edge intensity immediately before the imaging timing calculated in s13 to the average edge intensity immediately after the imaging timing calculated in s14. It is determined whether it is less than 0.6) (s15). The ratio α is a value obtained by dividing the average edge intensity immediately before the imaging timing calculated in s13 by the average edge intensity immediately after the imaging timing calculated in s14. That is,
Ratio α = (average edge intensity immediately before imaging timing calculated in s13) / (average edge intensity immediately after imaging timing calculated in s14).

  The appearance determination function unit 24 determines that the appearance determination target pixel whose ratio α is less than the appearance determination threshold value in s15 is a specific pixel (s16), and conversely, the ratio α is not less than the appearance determination threshold value in s15 ( The appearance determination target pixel (which is equal to or greater than the appearance determination threshold) is determined as noise (s17).

  The appearance determination function unit 24 stores the specific pixel determined in s16 in a storage unit (not shown) (s18), and returns to s11. In s18, the specific pixel determined this time and the frame number of the processing target frame image selected in s11 are associated with each other and stored in the storage unit.

  As shown in FIG. 10, the edge strength of a pixel related to a fallen object or the like changes abruptly, but the edge strength of a pixel related to a shadow of a surrounding structure that appears due to a weather change or a change in illumination changes gradually. In FIG. 10, t0 is the timing at which the pixel is extracted at s4 (the imaging timing of the processing target frame image selected at s11). Further, the period from t0 to t1 is the appearance determination time immediately after the imaging timing, and the period from t2 to t0 is the appearance determination time immediately before the imaging timing. Therefore, the appearance determination target pixel that is determined to be noise in s17 is a pixel that covers a shadow of a surrounding structure that appears due to a weather change or illumination change, and is not a pixel that covers a vehicle or a fallen object from the vehicle.

  Next, the second specific pixel determination process will be described. FIG. 11 is a flowchart showing the second specific pixel determination process. The change determination function unit 25 of the image processing unit 13 performs the second specific pixel determination process. In the second specific pixel determination process, the edge image stored in the image memory in s2 and the change pixel stored in the storage unit in s4 are used. This second specific pixel determination process is a process for determining whether or not the changed pixel extracted in s4 is a pixel related to a puddle on the road surface due to rain.

  The change determination function unit 25 selects a processing target frame image (s21). In s21, the frame image on which the above-described change pixel extraction processing is performed is selected as a processing target frame image in time series. The processing according to s21 is not performed by the processing target frame image selection function unit 21, but is performed by the change determination function unit 25. The second specific pixel determination process uses the processing result of the change pixel extraction process for the frame image captured after the change determination time described later, rather than the processing target frame image selected in s21. That is, the process result of the second specific pixel determination process is at least delayed by the change determination time than the process result of the above-described change pixel extraction process.

  The change determination function unit 25 extracts a change determination target pixel in the processing target frame image selected in s21 (s22). In s22, the change pixel extracted by the process concerning the above-mentioned s4 with respect to the process target frame image selected in s21 is extracted as a change determination target pixel. As described above, the storage unit stores the frame number of the processing target frame image and the change pixel extracted from the processing target frame image in association with each other. The number of change determination target pixels extracted in s22 may be 0, or may be 1 or more.

  Note that in s22, the configuration may be such that the pixel determined to be an appearance pixel in s16 is extracted as a change determination target pixel in the processing target frame image selected this time in s21.

  The change determination function unit 25 determines a change determination period for the processing target frame image selected in s21 (s23). In this change determination period, the timing when the waiting time (for example, 2 seconds) has elapsed from the imaging timing of the processing target frame image selected in s21 is the start timing, and the timing after the predetermined time (for example, 10 seconds) has elapsed. This change determination period corresponds to the third period referred to in the present invention. The change determination time is a time obtained by combining the above-described waiting time and the change determination period.

  The change determination function unit 25 divides the change determination period determined in s23 into a plurality of determination periods (s24). For example, the change determination period is divided into five determination periods divided at intervals of 2 seconds. This determination period corresponds to a divided period obtained by dividing the third period referred to in the present invention.

  The change determination function unit 25 calculates the standard deviation of the edge intensity of the change determination target pixel extracted in s22 for each determination period divided in s24 (s25). In s25, for each determination period, the edge image of the processing target frame image captured by the video camera 2 during the determination period is used.

  For the change determination target pixel extracted in s22, the change determination function unit 25 determines, for each determination period, the ratio β of the standard deviation of the edge intensity to the average value of the edge intensity in the determination period is a predetermined value (for example, 0.4 ) Is determined (s26).

Standard deviation ratio β
= (Standard deviation of edge strength in judgment period) / (Average value of edge strength in judgment period)
It is.

  The change determination function unit 25 determines, as noise, a change determination target pixel that is determined that the ratio β of the standard deviation of the edge strength to the average value of the edge strength in the determination period is not within a predetermined value in any determination period. (S28). In addition, the change determination function unit 25 sets the determination target pixel that is determined that the ratio β of the standard deviation of the edge intensity to the average value of the edge intensity in the determination period is within a predetermined value as the specific pixel in all the determination periods. Determine (s27). The processing related to s25 to s28 is performed for each change determination target pixel extracted in s22.

  The change determination function unit 25 stores the specific pixel determined in s27 in a storage unit (not shown) (s29), and returns to s21. In s29, the specific pixel determined this time and the frame number of the processing target frame image selected in s21 are associated with each other and stored in the storage unit.

As shown in FIG. 12, the edge strength of a pixel related to a fallen object or the like is stable in the change determination period, but the edge strength of a pixel related to a puddle or the like changes in the change determination period because the shape of the puddle changes . Not stable. In FIG. 12, t0 is the timing extracted as a change pixel at s4. Moreover, the period of t1-t3 is a change determination period. Therefore, the change determination target pixel determined as noise in s28 is a pixel related to a puddle or the like.

In addition, the edge strength of the pixels applied to the vehicle that has traveled without stopping the detection target area is not stable in the change determination period. Therefore, the change determination function unit 25 also determines that the pixel related to the traveling vehicle is noise in s27.

  Next, the obstacle determination process will be described. FIG. 13 is a flowchart showing the obstacle determination process. The obstacle detection function unit 26 of the image processing unit 13 performs this obstacle determination process.

  The obstacle detection function unit 26 selects a processing target frame image (s31). In s31, the frame image that has completed the first specific pixel determination process and the second specific pixel determination process described above is selected as a processing target frame image in time series. The processing according to s31 is not performed by the processing target frame image selection function unit 21, but is performed by the obstacle detection function unit 26.

  The obstacle detection function unit 26 searches for an obstacle detection area for the processing target frame image selected this time in s31 (s32). In s32, a search area that satisfies the following conditions 1 and 2 is searched. The search area is, for example, an area of 5 × 5 pixels.

Condition 1: In the processing target frame image selected this time, a pixel that is determined as a specific pixel in the first specific pixel determination process and that is determined as a specific pixel in the second specific pixel determination process is 1 There are two or more.

Condition 2: In the first and second specific frame determination processing in the first and second frames of the processing target frame image including the processing target frame image selected this time (for example, the processing target frame images of the two frames before and after (a total of five frame images)). The ratio of pixels that are determined to be specific pixels in the second specific pixel determination processing also exceeds a predetermined ratio (for example, 80%).

  Condition 2 is a condition for not detecting small dust or pixels related to noise or the like as an obstacle.

  The obstacle detection function unit 26 returns to s31 if there is no obstacle detection area searched in s32 (search area suitable for condition 1 and condition 2) (s33). When there is an obstacle detection area searched in s32, the obstacle detection function unit 26 determines the area of the foreground pixel including the obstacle detection area searched this time in the background difference image in the processing target frame image. Is detected as an imaged area (s33, s34). Also, the obstacle detection function unit 26 determines that notification is required for the obstacle display to the guidance display board 3 (s35), and returns to s31. In s35, it is not determined that the control device 4 needs to be notified of obstacle detection.

  Based on the determination in s35, the obstacle detection device 1 outputs a request notification for requesting notification related to the detection of the obstacle from the output unit 14 to the guidance display board 3. Thereby, the guidance display board 3 displays a message notifying the driver that there is an obstacle on the downstream side.

  As described above, when the obstacle detection device 1 detects an obstacle, it promptly requests the guidance display board 3 for notification accompanying the detection of the obstacle. Therefore, the guidance display board 3 can promptly display guidance for preventing accidents to the driver.

  When the obstacle detection function unit 26 detects an obstacle, the obstacle detection device 1 determines whether the obstacle continues to be positioned without moving for a predetermined continuation determination time (for example, 1 minute). The continuation determination process for determining FIG. 14 is a flowchart showing this continuation determination process. The obstacle continuation detection function unit 27 performs this continuation determination process.

  The obstacle continuation detection function unit 27 selects, as a processing target frame image, a frame image captured at each detection check waiting time with reference to the imaging timing of the processing target frame image in which the obstacle is detected (s41). For example, a frame image captured every second is sequentially selected as a processing target frame image based on the imaging timing of the processing target frame image in which an obstacle is detected. The processing related to s41 is not performed in the processing target frame image selection function unit 21, but is performed in the obstacle continuation detection function unit 27.

  The obstacle continuation detection function unit 27 calculates the degree of coincidence of edges in the obstacle detection region in the processing target frame image in which the obstacle is detected and the obstacle detection region in the processing target frame image selected in s41 ( s42), it is determined whether or not the calculated degree of coincidence of edges exceeds a predetermined ratio (s43). That is, in s43, it is determined whether or not the obstacle detected by the obstacle detection function unit 26 is in the same position.

  If the obstacle continuation detection function unit 27 determines that the edge coincidence exceeds a predetermined ratio in s42, the obstacle continuation detection function unit 27 increments the continuation count value by one (s44). The obstacle continuation detection function unit 27 does not perform the process related to s44 unless it is determined in s43 that the edge coincidence exceeds a predetermined ratio.

  The obstacle continuation detection function unit 27 increments the number of checks by 1 (s45), and determines whether the number of checks has reached the number of determinations (for example, 10) (s46). If the obstacle continuation detection function unit 27 determines that the number of determinations has not been reached in s46, the obstacle continuation detection function unit 27 returns to s41.

  If the obstacle continuation detection function unit 27 determines that the number of determinations has been reached in s46, the obstacle continuation detection function unit 27 determines whether the continuation count value is a predetermined value (for example, 5) or more (s47). If the obstacle continuation detection function unit 27 determines that the continuation count value is not greater than or equal to the predetermined value, it determines that the obstacle detected this time has moved (s48), and ends the present process. The obstacle that has moved is a shopping bag that has been blown by the wind, a vehicle that has been temporarily stopped for some reason, and the like.

  If the obstacle continuation detection function unit 27 determines that the continuation count value is greater than or equal to a predetermined value, the obstacle count over the continuation determination time is completed after resetting the continuation count value and the number of checks (s49). It is determined whether or not (s50). If the obstacle continuation detection function unit 27 determines in s50 that the obstacle movement confirmation over the continuation determination time is not completed, the process returns to s41.

  If the obstacle continuation detection function unit 27 determines that the obstacle movement confirmation over the continuation determination time has been completed, the obstacle continuation detection function unit 27 determines that the obstacle detected this time has not moved over the continuation determination time (s51), and ends this process. To do.

  The obstacle continuation detection function unit 27 passes the determination result of s48 or s51 to the control notification necessity determination function unit 29. The operation of the control notification necessity determination function unit 29 will be described later.

  In addition, when the obstacle detection function unit 26 detects an obstacle, the vehicle traveling state determination function unit 28 performs a vehicle abnormality determination process. FIG. 15 is a flowchart showing a running abnormality determination process. The vehicle traveling state determination function unit 28 uses the processing target frame image selected by the processing target frame image selection function unit 21 after the obstacle is detected by the obstacle detection function unit 26, and the traveling vehicle in the detection target area. The movement of is acquired (s61). In s61, for example, the movement of the traveling vehicle in the detection target area may be acquired by a known optical flow, or the movement of the traveling vehicle in the detection target area may be acquired by a known tracking technique. In s61, deceleration, acceleration, meandering, etc. of the vehicle are acquired as the movement of the traveling vehicle.

  The vehicle traveling state determination function unit 28 determines whether the traveling pattern based on the movement of the traveling vehicle acquired in s61 is unnatural (s62).

  For example, the traveling pattern based on the movement of the traveling vehicle acquired in s61 is compared with the traveling pattern of the vehicle when the traffic flow is normal, and the traveling pattern of the vehicle traveling in the detection target area is unnatural. Determine if it exists. Specifically, the traveling pattern of the vehicle traveling in the detection target area is determined based on the degree of coincidence between the traveling pattern based on the movement of the traveling vehicle acquired in s61 and the traveling pattern of the vehicle when the traffic flow is normal. Determine if it is unnatural. If there is an obstacle on the road that prevents the vehicle from traveling, the vehicle travels avoiding the obstacle, so the traveling pattern becomes unnatural due to deceleration, acceleration, meandering, and the like. Therefore, when there are obstacles on the road that hinder the traveling of the vehicle, the traveling pattern based on the movement of the traveling vehicle in the detection target area acquired in s61 and the stored traffic flow are normal. The degree of coincidence with the traveling pattern of the vehicle becomes small.

  In this case, the vehicle travel state determination function unit 28 stores a travel pattern of the vehicle when the traffic flow is normal. In addition, the vehicle traveling state determination function unit 28 obtains the traveling pattern of the vehicle when the traffic flow is normal by processing the moving image captured by the video camera 2 when no obstacle is detected. It is preferable that the learning is performed based on the movement of

  If the vehicle traveling state determination function unit 28 determines that the traveling pattern of the vehicle is unnatural in s62, the present processing is terminated. The vehicle travel state determination function unit 28 passes the determination result determined in s62 that the vehicle travel pattern is unnatural to the control notification necessity determination function unit 29.

  On the other hand, if the vehicle travel state determination function unit 28 determines in s62 that the vehicle travel pattern is not unnatural, it determines whether the obstacle continuation detection function unit 27 is executing the continuation determination process described above (s63). . If the vehicle running state determination function unit 28 determines in s63 that the obstacle continuation detection function unit 27 is executing the continuation determination process described above, the process returns to s61. In addition, when the vehicle traveling state determination function unit 28 determines in s63 that the obstacle continuation detection function unit 27 has not performed the above-described continuation determination process, the process ends.

Next, the control notification necessity determination process in the control notification necessity determination function unit 29 will be described. FIG. 16 is a flowchart showing a control notification necessity determination process. When the determination result indicating that the obstacle has moved (the determination result of s48) is passed from the obstacle continuation detection function unit 27 , the control notification necessity determination function unit 29 detects the obstacle with respect to the guidance display board 3. It is determined that the notification release notification is required (s71, s72), and this process is terminated.

Further, the control notification necessity determination function unit 29 receives the determination result (determination result of s51) that the obstacle has not moved for the continuation determination time from the obstacle continuation detection function unit 27, to the control device 4. On the other hand, it is determined that notification is required for the detection of the obstacle (s73, s74), and this process is terminated.

  Further, the control notification necessity determination function unit 29 detects the obstacle to the control device 4 when a determination result that the vehicle traveling pattern is unnatural is passed from the vehicle traveling state determination function unit 28. It is determined that such notification is necessary (s75, s74), and this process is terminated.

  Based on the determination in s72, the obstacle detection device 1 outputs a request notification for requesting the notification display board 3 to cancel the notification related to the detection of the obstacle, from the output unit 14. Thereby, the guidance display board 3 stops displaying the message notifying the driver that there is an obstacle on the downstream side. That is, the guidance display board 3 shifts to a state in which a message notifying the driver that there is an obstacle on the downstream side is not displayed. As a result, the message related to the erroneously detected obstacle is not continuously displayed on the guidance display board 3.

  Also, the obstacle detection device 1 outputs a request notification for requesting notification regarding the detection of the obstacle to the control device 4 from the output unit 14 based on the determination in s74. As a result, the control device 4 notifies the road manager to urge the work such as removing an obstacle (detected obstacle) on the road.

  As described above, the obstacle detection device 1 according to this example performs the first specific pixel determination process, so that it is possible to suppress erroneous detection of a shadow or the like of a surrounding structure due to a weather change or a lighting change as an obstacle. At the same time, by performing the second specific pixel determination process, it is possible to suppress erroneous detection of a puddle or the like whose shape has changed due to the traveling of the vehicle as an obstacle.

  Also, the obstacle detection apparatus 1 according to this example can suppress erroneous detection of a pixel related to small dust or noise as an obstacle by performing the obstacle determination process.

  In addition, the obstacle detection device 1 according to this example performs a continuation determination process, thereby detecting a shopping bag that has been blown by the wind or a vehicle that is temporarily stopped for some reason as an obstacle. 4 can be prevented from outputting a request notification requesting notification related to obstacle detection. Therefore, the control device 4 can suppress wasteful notification that prompts the road manager to perform work by using a shopping bag that has been blown by the wind or a vehicle temporarily stopped for some reason as an obstacle. Furthermore, the obstacle detection apparatus 1 according to this example performs a running abnormality determination process to notify the road administrator of work such as removing an obstacle (detected obstacle) on the road. It can be done quickly.

  In the above example, the edge strength is used as the feature amount of the pixel. However, the luminance value of the pixel may be used as the feature amount. In this case, the edge image generation function unit 22 is. It becomes unnecessary.

  Further, the obstacle continuation detection function unit 27, the vehicle traveling state determination function unit 28, and the control notification necessity determination function unit 29 are not required, and the obstacle detection function unit 26 replaces the guidance display board 3 and the control device 4 with s35. On the other hand, a configuration may be adopted in which it is determined that notification is required for obstacle detection.

DESCRIPTION OF SYMBOLS 1 ... Obstacle detection apparatus 2 ... Video camera 3 ... Guidance display board 4 ... Control apparatus 11 ... Control part 12 ... Image input part 13 ... Image processing part 14 ... Output part 21 ... Process target frame image selection function part 22 ... Edge image Generation function unit 23 ... Background difference image generation function unit 24 ... Appearance determination function unit 25 ... Change determination function unit 26 ... Obstacle detection function unit 27 ... Obstacle continuation detection function unit 28 ... Vehicle travel condition determination function unit 29 ... Control notification Necessity judgment function part

Claims (14)

A moving image input unit for inputting a moving image obtained by imaging a detection target area defined on a road on which the vehicle travels;
A frame image related to a moving image input to the moving image input unit is processed, and for each pixel of the frame image, it is determined whether the pixel is a foreground pixel or a background pixel using a feature amount of the pixel. A foreground pixel determination unit;
For the pixel whose determination result by the foreground pixel determination unit has changed from the background pixel to the foreground pixel, the imaging timing of the frame image that has changed from the background pixel to the foreground pixel is used as the reference timing, and imaging is performed during the determination period determined based on this reference timing. A specific pixel determination unit that determines whether or not the pixel is a specific pixel that covers an obstacle using the frame image that has been generated;
An obstacle detection unit that detects an object including a pixel determined to be the specific pixel by the specific pixel determination unit as an obstacle ,
The specific pixel determination unit determines the first period immediately before the reference timing and the second period immediately after the reference timing for the pixel whose determination result from the foreground pixel determination unit has changed from a background pixel to a foreground pixel. The pixel is determined by the ratio between the feature amount of the pixel in the frame image captured in the first period and the feature amount of the pixel in the frame image captured in the second period. An obstacle detection device that performs a first specific pixel determination process for determining whether or not a pixel is present . A moving image input unit for inputting a moving image obtained by imaging a detection target area defined on a road on which the vehicle travels;
A frame image related to a moving image input to the moving image input unit is processed, and for each pixel of the frame image, it is determined whether the pixel is a foreground pixel or a background pixel using a feature amount of the pixel. A foreground pixel determination unit;
For the pixel whose determination result by the foreground pixel determination unit has changed from the background pixel to the foreground pixel, the imaging timing of the frame image that has changed from the background pixel to the foreground pixel is used as the reference timing, and imaging is performed during the determination period determined based on this reference timing. A specific pixel determination unit that determines whether or not the pixel is a specific pixel related to an obstacle using the frame image that has been generated;
An obstacle detection unit that detects an object including a pixel determined to be the specific pixel by the specific pixel determination unit as an obstacle,
The specific pixel determination unit sets, as the determination period, a third period in which a predetermined time has elapsed from the reference timing as a start timing for a pixel whose determination result from the foreground pixel determination unit has changed from a background pixel to a foreground pixel. An obstacle detection device that performs a second specific pixel determination process that determines and determines whether or not the pixel is a specific pixel from a change in the feature amount of the pixel in the frame image captured in the third period . The specific pixel determination unit determines the first period immediately before the reference timing and the second period immediately after the reference timing for the pixel whose determination result from the foreground pixel determination unit has changed from a background pixel to a foreground pixel. The pixel is identified using the feature amount of the pixel in the frame image captured in the first period and the feature amount of the pixel in the frame image captured in the second period. The obstacle detection device according to claim 2 , wherein a first specific pixel determination process for determining whether or not the pixel is a pixel is performed. The first specific pixel determination process is based on a ratio between a feature amount of the pixel in the frame image captured in the first period and a feature amount of the pixel in the frame image captured in the second period. The obstacle detection device according to claim 3 , wherein the obstacle detection processing is processing for determining whether or not the pixel is a specific pixel. In the second specific pixel determination process, for a pixel whose determination result from the foreground pixel determination unit has changed from a background pixel to a foreground pixel, a variation in the feature amount of the pixel in the frame image captured in the third period The obstacle detection device according to any one of claims 2 to 4 , which is a process of determining whether or not the pixel is a specific pixel. In the second specific pixel determination process, for a pixel whose determination result by the foreground pixel determination unit has changed from a background pixel to a foreground pixel, the pixel is a specific pixel for each divided period obtained by dividing the third period into a plurality of periods. And determining whether the pixel is not a specific pixel when it is determined that the pixel is not a specific pixel in any of the divided periods. The obstacle detection device according to any one of Items 2 to 5 .   The obstacle detection unit includes an image region in which a distribution density of specific pixels determined by the specific pixel determination unit is higher than a predetermined determination density in a predetermined number of frame images having the reference timing as a center of imaging timing. The obstacle detection device according to any one of claims 1 to 6, wherein if there is an object, the object imaged in the image area is detected as an obstacle.   The obstacle detection device according to any one of claims 1 to 7, further comprising an output unit that outputs a first notification related to the detection of an obstacle when the obstacle detection unit detects the obstacle. A movement detection unit for detecting whether the obstacle detected by the obstacle detection unit has moved,
When the movement detection unit does not detect that the obstacle detected by the obstacle detection unit has moved during a predetermined continuation determination time, the output unit applies a second response request for the obstacle. The obstacle detection device according to claim 8, wherein the notification is output. A traveling state determination unit that determines whether or not the vehicle traveling pattern in the detection target area is unnatural;
The said output part outputs the 2nd notification concerning the response | compatibility request | requirement of an obstruction, when the said driving | running state determination part determines with the driving pattern of the vehicle in the said detection target area being unnatural. Obstacle detection device. A frame image related to a moving image obtained by imaging a detection target area defined on a road on which the vehicle is traveling, which is input to the moving image input unit, is processed, and the feature amount of the pixel is used for each pixel of the frame image. A foreground pixel determining step for determining whether the pixel is a foreground pixel or a background pixel;
For a pixel whose determination result in the foreground pixel determination step has changed from a background pixel to a foreground pixel, the imaging timing of the frame image that has changed from the background pixel to the foreground pixel is used as a reference timing, and imaging is performed during the determination period determined based on this reference timing A specific pixel determination step for determining whether or not the pixel is a specific pixel related to an obstacle using the frame image thus obtained;
An obstacle detection step of detecting an object including the pixel determined to be the specific pixel by the specific pixel determination step as an obstacle ,
The specific pixel determining step determines the first period immediately before the reference timing and the second period immediately after the reference timing for the pixel whose determination result from the foreground pixel determining step has changed from a background pixel to a foreground pixel. The pixel is determined by the ratio between the feature amount of the pixel in the frame image captured in the first period and the feature amount of the pixel in the frame image captured in the second period. Performing a first specific pixel determination process for determining whether or not the pixel is a pixel;
Obstacle detection method. A frame image related to a moving image obtained by imaging a detection target area defined on a road on which the vehicle is traveling, which is input to the moving image input unit, is processed, and the feature amount of the pixel is used for each pixel of the frame image. A foreground pixel determining step for determining whether the pixel is a foreground pixel or a background pixel;
For a pixel whose determination result in the foreground pixel determination step has changed from a background pixel to a foreground pixel, the imaging timing of the frame image that has changed from the background pixel to the foreground pixel is used as a reference timing, and imaging is performed during the determination period determined based on this reference timing. A specific pixel determination step for determining whether or not the pixel is a specific pixel related to an obstacle using the frame image thus obtained;
An obstacle detection step of detecting an object including the pixel determined to be the specific pixel by the specific pixel determination step as an obstacle,
In the specific pixel determining step, for a pixel whose determination result in the foreground pixel determining step has changed from a background pixel to a foreground pixel, a third period having a timing after a certain time has elapsed from the reference timing as a start timing is set as the determination period. A step of performing a second specific pixel determination process for determining whether or not the pixel is a specific pixel from a change in a feature amount of the pixel in the frame image captured in the third period,
Obstacle detection method. A frame image related to a moving image obtained by imaging a detection target area defined on a road on which the vehicle is traveling, which is input to the moving image input unit, is processed, and the feature amount of the pixel is used for each pixel of the frame image. A foreground pixel determining step for determining whether the pixel is a foreground pixel or a background pixel;
For a pixel whose determination result in the foreground pixel determination step has changed from a background pixel to a foreground pixel, the imaging timing of the frame image that has changed from the background pixel to the foreground pixel is used as a reference timing, and imaging is performed during the determination period determined based on this reference timing. A specific pixel determination step for determining whether or not the pixel is a specific pixel related to an obstacle using the frame image thus obtained;
Causing the computer to execute an obstacle detection step of detecting an object including the pixel determined to be the specific pixel by the specific pixel determination step as an obstacle ,
The specific pixel determining step determines the first period immediately before the reference timing and the second period immediately after the reference timing for the pixel whose determination result from the foreground pixel determining step has changed from a background pixel to a foreground pixel. The pixel is determined by the ratio between the feature amount of the pixel in the frame image captured in the first period and the feature amount of the pixel in the frame image captured in the second period. Performing a first specific pixel determination process for determining whether or not the pixel is a pixel;
Obstacle detection program. A frame image related to a moving image obtained by imaging a detection target area defined on a road on which the vehicle is traveling, which is input to the moving image input unit, is processed, and the feature amount of the pixel is used for each pixel of the frame image. A foreground pixel determining step for determining whether the pixel is a foreground pixel or a background pixel;
For a pixel whose determination result in the foreground pixel determination step has changed from a background pixel to a foreground pixel, the imaging timing of the frame image that has changed from the background pixel to the foreground pixel is used as a reference timing, and imaging is performed during the determination period determined based on this reference timing. A specific pixel determination step for determining whether or not the pixel is a specific pixel related to an obstacle using the frame image thus obtained;
Causing the computer to execute an obstacle detection step of detecting an object including the pixel determined to be the specific pixel by the specific pixel determination step as an obstacle,
In the specific pixel determining step, for a pixel whose determination result from the foreground pixel determining step is changed from a background pixel to a foreground pixel, a third period having a timing after a certain time has elapsed from the reference timing as a start timing is set as the determination period. A step of performing a second specific pixel determination process for determining whether or not the pixel is a specific pixel from a change in a feature amount of the pixel in the frame image captured in the third period,
Obstacle detection program.