JP2019049918A - Obstacle detection system for heavy equipment - Google Patents

Abstract

To provide an obstacle detection system for heavy equipment that can enhance the detection accuracy while preventing error detection caused by changes of terrain affected by obstacles which are driving or working.SOLUTION: An obstacle detection system comprises a camera 11 for photographing the surrounding of a dump truck 1, an image processing device 12 for detecting obstacles within the image captured by the camera 11, and a display device 13 for displaying an image processed by the image processing device 12. The image processing device 12 comprises: an obstacle position detecting section 21 for calculating the positional detection area of an obstacle by excluding, from difference observed area calculated by a background difference calculating section 17, a traveling route area calculated by a traveling route area calculating section 20; and a composite image generating section 16 for generating a composite image by superimposing, on the current image, an area marker indicating the positional detection area of the obstacle calculated by the obstacle position detecting section 21, and then outputting the composite image to the display device 13.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an obstacle detection system for a working machine that detects an obstacle in an image acquired by a camera.

  Examples of work machines that operate under rough ground environments such as mines include hydraulic shovels excavating the ground, wheel loaders that ground the ground, and dump trucks that transport earth and sand. Among the above-described work machines, particularly a super-large dump truck having a total height of 7 m or more, there are many blind spots where the operator can not directly view. For this reason, there is a possibility that an accident may occur in which the work machine contacts an obstacle around it (specifically, a moving object such as another vehicle or a person).

  Therefore, an obstacle detection system for a working machine has been proposed for alerting an operator of the working machine (see, for example, Patent Document 1). The obstacle detection system of Patent Document 1 includes a camera for photographing the periphery of a work machine, an image processing apparatus for detecting an obstacle in an image acquired by the camera, and a display for displaying an image processed by the image processing apparatus. And an apparatus.

  The image processing apparatus of Patent Document 1 compares, for example, a current image acquired by a camera with a past image to calculate a motion vector (optical flow) between pixels of both images, and an obstacle based on the motion vector. Detect the area of Alternatively, the area of the obstacle is detected based on, for example, the difference between the pixel value of the current image and the pixel value of the background model image created in advance.

  Patent Document 2 discloses another method for detecting an area of an obstacle in an image. In this method, a plurality of difference areas and an outline between the areas are calculated based on the difference between the pixel value of the current image and the pixel value of the past image. Also, a current image and a past image are compared to calculate a motion vector between pixels of both images, and pixels of the current image having motion vectors that can be regarded as the same as each other are collected to obtain a motion vector generation region. Then, among the plurality of contours, the contour corresponding to the motion vector generation region is extracted, and the region surrounded by those contours is detected as the region of the obstacle.

JP, 2013-222254, A

JP, 2015-138319, A

  As described above, in Patent Document 1, a method of calculating a motion vector by comparing a current image and a past image, and detecting a position of an obstacle based on the motion vector is adopted, or A method of detecting the position of the obstacle based on the difference between the pixel value and the pixel value of the background model image is adopted. In the former method, when there is no luminance difference in the whole obstacle, it becomes difficult to calculate a motion vector for the whole obstacle, and the detection accuracy of the position of the obstacle is lowered. Therefore, the latter method can improve the detection accuracy of the position of the obstacle than the former method.

  However, in rough ground environments such as mines, changes in topography occur due to travel or work of obstacles, and the latter method can not cope with changes in topography, which may lead to erroneous detection. . Specifically, when another vehicle which is an obstacle travels, wrinkles occur on the trajectory. In addition, when a hydraulic excavator or a wheel loader, which is an obstacle, works, the topography changes on the trajectory. Then, the difference between the pixel value of the current image and the pixel value of the background model image includes not only the change in the obstacle but also the change in the topography, which may cause erroneous detection.

  Further, even when the method described in Patent Document 2 is adopted, the contour of the obstacle and the contour of the changing portion of the terrain can not be distinguished continuously, which may cause erroneous detection.

  The present invention has been made in view of the above-described matters, and an object thereof is to provide a working machine capable of enhancing detection accuracy while preventing erroneous detection caused by a change in topography due to traveling or work of an obstacle. To provide an obstacle detection system.

  In order to achieve the above object, the present invention provides a camera for photographing the periphery of a work machine, an image processing apparatus for detecting an obstacle in an image acquired by the camera, and an image processed by the image processing apparatus. In the obstacle detection system for a working machine including a display device for displaying the image, the image processing device includes: a first storage unit that stores images acquired by the camera in time series; and the image storage device acquired by the camera A background difference calculation unit that calculates a difference generation area based on the difference between the pixel value of the current image and the pixel value of the background model image created in advance, and comparing the current image with the immediately preceding image A motion vector calculation unit that calculates a motion vector related to the pixels between the pixels, collects pixels of the current image having motion vectors that can be regarded as the same as one another, and acquires a motion vector generation area; At least one obstacle position estimation indicating the area in which the obstacle exists based on the position information of the motion vector generation area when the area of the motion vector generation area acquired by the output unit is equal to or greater than a predetermined threshold value An obstacle position estimation unit that generates a region; a second storage unit that stores the obstacle position estimation region generated by the obstacle position estimation unit in time series; and the second storage unit stores the second storage unit The movement path area calculation unit calculates the movement path area calculation unit that calculates the movement path area of the obstacle based on the history information of the obstacle position estimation area and the difference generation area calculated by the background difference calculation unit An obstacle position detection unit that generates the position detection area of the obstacle by excluding the moving path area, and the obstacle position detection unit generates the current image acquired by the camera. By superimposing a region marker indicating the position detection area of the obstacle to generate a composite image, and a composite image generating unit for outputting the synthesized image to the display device.

  In the present invention, the difference generation area is calculated based on the difference between the pixel value of the current image and the pixel value of the background model image, and the position of the obstacle is detected based on this. As a result, it is possible to calculate the motion vector by comparing the current image and the previous image, and to improve the detection accuracy compared to the case where the position of the obstacle is detected based on this.

  However, the above-described difference generation area may include a change in topography due to travel or work of an obstacle. Therefore, in the present invention, the obstacle position estimation area is generated based on the position information of the motion vector generation area, and the movement path area of the obstacle is calculated based on the history information of the obstacle position estimation area. The movement path area is an area where a change in topography due to travel or work of an obstacle appears. Therefore, by excluding the movement path area from the difference generation area, it is possible to prevent an erroneous detection caused by a change in topography.

It is a side view showing the structure of the dump truck in a 1st embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS It is an external view showing the structure of the obstacle detection system in the 1st Embodiment of this invention. It is a block diagram showing the functional composition of the image processing device in a 1st embodiment of the present invention with related apparatus. It is a figure showing the example of the present image in the 1st Embodiment of this invention, and the image of the last. It is a flowchart showing the processing content of the image processing apparatus in the 1st Embodiment of this invention. It is a figure showing the specific example of the background model image in the 1st Embodiment of this invention. It is a figure which shows the specific example of the difference generation | occurrence | production area | region in the 1st Embodiment of this invention. It is a figure which shows an example of the motion vector in the 1st Embodiment of this invention, and the specific example of a motion vector generation | occurrence | production area | region. It is a figure which shows the specific example of the position estimation area | region of the present obstacle in the 1st Embodiment of this invention. It is a figure for demonstrating the calculation method of the movement path | route area | region of the obstruction in the 1st Embodiment of this invention. It is a figure which shows the specific example of the position detection area | region of the obstruction in the 1st Embodiment of this invention. It is a figure showing the example of the synthetic | combination image in the 1st Embodiment of this invention. It is a block diagram showing the functional composition of the image processing device in a 2nd embodiment of the present invention with related apparatus. It is a flowchart showing the processing content of the image processing apparatus in the 2nd Embodiment of this invention. It is a figure for demonstrating the obstacle of a state of rest in the 2nd embodiment of the present invention, and the obstacle of an operation state. It is a figure showing the specific example of the past image in a 2nd embodiment of the present invention, and shows the specific example of the obstacle position presumed field in the past. It is a figure showing the example of the present picture in a 2nd embodiment of the present invention, and shows the example of the present obstacle position presumed field. It is a figure which shows the specific example of the movement path area | region of the obstruction in the 2nd Embodiment of this invention. It is a figure which shows the specific example of the difference generation | occurrence | production area | region in the 2nd Embodiment of this invention. It is a figure which shows the specific example of the position detection area | region of the obstruction in the 2nd Embodiment of this invention. It is a figure showing the example of the synthetic | combination image in the 2nd Embodiment of this invention. It is a block diagram showing the functional composition of the image processing device in a 3rd embodiment of the present invention with related apparatus. It is a flowchart showing the processing content of the image processing apparatus in the 3rd Embodiment of this invention. It is a figure showing the example of the 1st past and the 2nd past picture in a 3rd embodiment of the present invention, and shows the example of the 1st past and the 2nd past obstacle position presumption field. It is a figure showing the example of the 3rd past picture in a 3rd embodiment of the present invention, and shows the example of the obstacle position presumed field and terrain change field of the 3rd past. It is a figure showing the example of the present picture in a 3rd embodiment of the present invention, and shows the example of the present obstacle position presumed field. It is a figure which shows the specific example of the moving-path area | region of the obstruction in the 3rd Embodiment of this invention with a topography change area. It is a figure which shows the specific example of the difference generation | occurrence | production area | region in the 3rd Embodiment of this invention. It is a figure which shows the specific example of the position detection area | region of the obstruction in the 3rd Embodiment of this invention. It is a figure showing the specific example of the synthetic | combination image in the 3rd Embodiment of this invention. It is a figure showing the example of the synthetic | combination image in one modification of this invention.

  Hereinafter, an embodiment of the present invention will be described by taking a dump truck operating under rough ground environment such as a mine as an example to which the present invention is applied.

  A first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a side view showing the structure of a dump truck in the present embodiment. FIG. 2 is an external view showing a configuration of an obstacle detection system in the present embodiment. FIG. 3 is a block diagram showing the functional configuration of the image processing apparatus in the present embodiment together with related devices.

  The dump truck 1 is disposed at the front of the vehicle body frame 4 with the vehicle body frame 4 having the front wheel 2 and the rear wheel 3, the vessel 6 provided tiltably at the rear of the vehicle body frame 4 via the pin joint 5 and A driver's cab 7 is provided. The vessel 6 is a loading platform for loading soil and the like, and is configured to be undulated by the expansion and contraction of a pair of left and right undulating cylinders 8 (hydraulic cylinders). At the upper front of the vessel 6, a collar 9 is provided to protect the cab 7. As shown in FIG. Although not shown, a shift lever, a steering wheel, an accelerator pedal, a brake pedal and the like are installed inside the cab 7.

  The dump truck 1 is provided with an obstacle detection system for alerting the operator of the dump truck 1. The obstacle detection system according to the present embodiment includes a camera 11 for photographing the periphery of the dump truck 1, an image processing device 12 (controller) for detecting an obstacle in an image acquired by the camera 11, and an image processing device 12. And a display device 13 (monitor) for displaying the processed image.

  For example, as shown in FIG. 1, the camera 11 is attached at the center in the width direction of the front portion of the vehicle body frame 4 at a depression angle (downward angle), and images the front view range AR of the dump truck 1. The camera 11 acquires an image at a predetermined cycle and outputs the image to the image processing device 12.

  The image processing device 12 and the display device 13 are installed in the cab 7. The image processing apparatus 12 includes, for example, a CPU, a ROM, and a RAM, and has a functional configuration including an image storage unit 14 (first storage unit), an obstacle detection unit 15, a composite image generation unit 16, and It has an aircraft stop determination unit (not shown).

  The image storage unit 14 sequentially receives images from the camera 11 and stores them in time series, and when the stored images reach the upper limit number (for example, 30), the oldest image is deleted and the latest (current) ) Is stored. The image storage unit 14 also outputs the current image 31 to the composite image generation unit 16.

  FIG. 4A shows a specific example of the current image 31. As shown in FIG. FIG. 4B is a diagram showing a specific example of the immediately preceding image 32 included in the past image group 33. As shown in FIG. In this specific example, it is assumed that the wheel loader (obstacle) travels and wrinkles occur on the trajectory. In the image 31 etc., a self-machine structure 34 (specifically, the front end of the vehicle body frame 4 of the dump truck 1) is shown.

  The obstacle detection unit 15 includes a background difference calculation unit 17, a motion vector calculation unit 18, an obstacle position estimation unit 19, a movement path area calculation unit 20, an obstacle position detection unit 21, and a storage unit 22 (second storage unit). have. The obstacle detection unit 15 receives the current image 31 and the like from the image storage unit 14 and performs image processing on the current image 31 and the like to generate a position detection area of the obstacle in the current image 31. (Details will be described later). In order to reduce the time, processing is set to be performed on the target range 35 instead of the entire image. However, processing may be performed on the entire image.

  The composite image generation unit 16 superimposes a region marker indicating the position detection region of the obstacle generated by the obstacle detection unit 15 on the current image 31 from the image storage unit 14 to generate a composite image, and this composite image is generated. An image is output to the display device 13 (details will be described later).

  The own-machine stop determination unit inputs, for example, the traveling speed of the own machine (dump truck 1) detected by a speed sensor (not shown) or inputs the operation position of the shift lever, and based on this Determine whether your aircraft is at a standstill. Then, the determination information is output to the image storage unit 14 and the obstacle detection unit 15. The image storage unit 14 counts the number of images stored after the own vehicle stops, and outputs the count information to the background difference calculation unit 17 of the obstacle detection unit 15.

  Next, the processing procedure of the image processing apparatus 12 of the present embodiment will be described. FIG. 5 is a flowchart showing the processing content of the image processing apparatus 12 in the present embodiment.

  First, in step S101, the background difference calculation unit 17 determines whether or not the number of images stored after the own vehicle is stopped is at least a predetermined number (for example, 10) according to the count information from the image storage unit 14 Determine if If the number of images stored after the own vehicle stops is less than the predetermined number, the determination in step S101 is NO, and the determination is repeated. If the number of images stored after the own vehicle is stopped is equal to or more than the predetermined number, the determination in step S101 is YES, and the process proceeds to step S102. In step S102, the background difference calculation unit 17 inputs a predetermined number of images stored after the own machine stops from the image storage unit 14, creates the background model image 36 based on those images, and the image storage unit Store in 14. FIG. 6 is a view showing a specific example of the background model image 36. In this specific example, no obstacle is present in the background model image 36. As shown in FIG.

  Then, the process proceeds to step S103, and the background difference calculation unit 17 determines whether or not the stop condition of the own machine continues since the background model image 36 was created based on the determination information from the own machine stop determination unit. Do. If the own vehicle is not in the stopped state, the determination in step S103 is NO, and the process returns to step S101 to repeat the same procedure as described above. If the stop of the own aircraft continues, the determination in step S103 is YES, and the process proceeds to steps S104 and S105.

  In step S104, the background difference calculation unit 17 receives the current image 31 and the background model image 36 from the image storage unit 14, and calculates a difference generation area based on the difference between the pixel values. Specifically, for the current image 31 and the background model image 36, the difference in luminance value for each pixel in the target range 35 is calculated, and the difference in luminance value is white at a predetermined threshold (for example, 30 gradations) or more, A binarization process is performed to make black below a predetermined threshold. FIG. 7 is a diagram showing a specific example of the difference generation area. In this specific example, a difference generation area 37 calculated based on the difference between the pixel value of the current image 31 of FIG. 4A and the pixel value of the background model image 36 of FIG. 5 is shown. The difference generation area (as well as an obstacle position estimation area, a movement route area, an obstacle position detection area, and the like, which will be described later) includes position information on the image.

  On the other hand, in step S105, the motion vector calculation unit 18 receives the current image 31 and the previous image 32 from the image storage unit 14, compares the images 31 and 32, for example, using the block matching method, , 32 are calculated for the pixel. In detail, a target block including the target pixel 38 (see FIG. 8A) of the immediately preceding image 32 and its surrounding pixels, and the comparison pixel 39 of the current image 31 shifted from the target block (FIG. 8) (A) Set a comparison block including the surrounding pixels and the surrounding pixels, calculate the similarity of the luminance value distribution for the block of interest and the comparison block, and determine the pixel of interest if it is determined that the similarity is high A motion vector 40 (see FIG. 8A) from 38 to the comparison pixel 39 is calculated. In the present embodiment, the direction of the motion vector is roughly divided into any one of the left direction, the right direction, the upper direction, and the lower direction, and the direction of the motion vector 40 in FIG. ing.

  Then, the process proceeds to step S106, and the motion vector calculation unit 18 determines that the motion vector has one of the left direction, the right direction, the upper direction, and the lower direction with respect to the pixels of the current image 31 having the motion vector. If there are things that can be regarded as the same and adjacent to each other, they are collected to obtain a motion occurring vector area. FIG. 8B is a diagram showing a specific example of the motion vector generation area, and for convenience, it is shown superimposed on the wheel loader (obstacle) in the current image 31. In this specific example, two motion vector generation areas 41A and 41B resulting from the travel of one wheel loader are acquired.

  Then, the process proceeds to step S107, and the obstacle position estimation unit 19 determines that the area of the motion vector generation area acquired by the motion vector calculation unit 18 is equal to or larger than a predetermined threshold (for example, 0.5% of the area of the target range 35). It is determined whether or not. If the area of the motion vector generation area is equal to or greater than a predetermined threshold value, it is estimated that an obstacle in the operating state is present in the current image 31. Then, when there is one motion vector generation area acquired by the motion vector calculation unit 18, one obstacle position estimation indicating the existence area of one obstacle is performed based on the position information of the motion vector generation area. Create a region. In detail, a rectangular obstacle position estimation area including one motion vector generation area is set so as to reduce the area as much as possible.

  When there are two motion vector generation areas obtained by the motion vector calculation unit 18, the directions of the motion vectors of the two motion vector generation areas can be regarded as the same and their center distance is predetermined. It is determined whether or not it is less than or equal to a threshold (for example, 30 pixels). If the directions of the motion vectors can be regarded as the same and the center-to-center distance is equal to or less than a predetermined threshold, it is estimated that an obstacle of one motion state exists in the current image 31. Therefore, as shown in FIG. 9, one obstacle position estimation area 42 indicating the existence area of one obstacle is generated based on the position information of the two motion vector generation areas 41A and 41B. In detail, the rectangular obstacle position estimation area 42 including the two motion vector generation areas 41A and 41B is set so as to reduce the area as much as possible.

  If the directions of the motion vectors are different or if the center-to-center distance exceeds a predetermined threshold, it is estimated that there are obstacles of two motion states in the current image 31. Therefore, based on the position information of the two motion vector generation areas, two obstacle position estimation areas respectively indicating the existence areas of the two obstacles are generated. In detail, one rectangular obstacle position estimation area including one motion vector generation area is set to be as small as possible while the other rectangular shape including the other motion vector generation area is set. The obstacle position estimation area is set so as to reduce the area as much as possible. The obstacle position estimation area generated as described above is temporally stored in the storage unit 22 in association with the obstacle being in the operating state and the direction of the motion vector.

  In addition, when the area of the motion vector generation area acquired by the motion vector calculation unit 18 is less than a predetermined threshold, the obstacle position estimation unit 19 determines whether an obstacle in a stationary state exists in the current image 31. Estimate whether or not. Specifically, for example, when the area of the difference generation area calculated by the background difference calculation unit 17 is equal to or more than a predetermined threshold (for example, 0.5% of the area of the target range 35), the still state in the current image 31 It is estimated that the obstacle of Alternatively, for example, the immediately preceding obstacle position estimation area stored in the storage unit 22 (or the obstacle position estimation area including the past several times including this) is in a portion other than the edge side of the target range 35 (in other words, the central portion) If it is located, it is determined that a stationary obstacle is present in the current image 31.

  If it is estimated that a stationary obstacle is present in the current image 31, it is based on the immediately preceding obstacle position estimation area stored in the storage unit 22 (or the obstacle position estimation area for the past several times including this). Generate a current obstacle position estimation area. The obstacle position estimation area generated in this manner is stored in the storage unit 22 in chronological order, in association with the fact that the obstacle is stationary.

  Then, the process proceeds to step S108, and the movement route area calculation unit 20 calculates the movement route area of the obstacle based on the history information of the obstacle position estimation area stored in the storage unit 22. In detail, the moving path of the obstacle is obtained by the difference between the past obstacle position estimation area group 43 (see FIG. 10A) and the present obstacle position estimation area 42 (see FIGS. 9 and 10A). An area 44 (see FIG. 10B) is calculated.

  Then, the process proceeds to step S109, and the topography change difference exclusion unit 23 of the obstacle position detection unit 21 starts from the difference generation area 37 (see FIG. 7) calculated by the background difference calculation unit 17 in step S104 to step S108. By excluding the movement path area 44 (see FIG. 10) of the obstacle calculated by the movement path area calculation unit 20, the position detection area 45 (see FIG. 11) of the obstacle is generated. The position detection area 45 of the obstacle generated in this manner is associated with the stationary state or the operating state of the obstacle similarly to the above-described obstacle position estimation area, and is stored in the storage unit 22 in chronological order It may be stored.

  Then, the process proceeds to step S110, and the composite image generation unit 16 determines that the area of the obstacle position detection area generated by the obstacle position detection unit 21 is equal to or greater than a predetermined threshold (for example, 0.5% of the area of the target range 35). It is determined whether the In the specific example shown in FIG. 11, since the area of the obstacle position detection area 45 is equal to or larger than the predetermined threshold value, the determination in step S110 is YES, and the process proceeds to step S111. In step S111, the composite image generation unit 16 displays an area marker 46 indicating the position detection area 45 of the obstacle generated by the obstacle position detection unit 21 on the current image 31 (original image) from the image storage unit 14. To generate a composite image 47 (see FIG. 12). In detail, the rectangular area marker 46 including the obstacle position detection area 45 is set so as to reduce the area as much as possible. Then, the composite image 47 is output to the display device 13 and displayed.

  If the area of the position detection area of the obstacle calculated by the obstacle position detection unit 21 is smaller than the threshold, the determination in step S110 is NO, and the process proceeds to step S112. In step S112, the composite image generation unit 16 outputs the current image 31 (original image) from the image storage unit 14 to the display device 13 for display as it is.

  In the embodiment configured as described above, the difference generation area 37 is calculated based on the difference between the pixel value of the current image 31 and the pixel value of the background model image 36, and the position of the obstacle is calculated based on this. To detect As a result, it is possible to calculate the motion vector by comparing the current image 31 and the previous image 32, and to improve the detection accuracy compared to the case where the position of the obstacle is detected based on this.

  However, the above-described difference generation area 37 may include a change in topography due to travel or work of an obstacle. Therefore, in the present embodiment, the obstacle position estimation area is generated based on the position information of the motion vector generation area, and the movement path area 44 of the obstacle is calculated based on the history information of the obstacle position estimation area. The movement path area 44 is an area where a change in topography due to travel or work of an obstacle appears. Therefore, by excluding the movement path area 44 from the difference generation area 37, it is possible to prevent an erroneous detection caused by a change in topography. That is, it is possible to prevent the area marker 46 from being superimposed on the change portion of the topography.

  Therefore, the position of the obstacle can be accurately indicated to the operator of the dump truck 1, and the operation of the dump truck 1 can be supported.

  A second embodiment of the present invention will be described with reference to the drawings. FIG. 13 is a block diagram showing the functional configuration of the image processing apparatus according to this embodiment together with related devices. FIG. 14 is a flowchart showing the processing content of the image processing apparatus in the present embodiment. In the present embodiment, parts equivalent to those in the first embodiment are given the same reference numerals, and the description will be omitted as appropriate.

  The obstacle detection unit 15 of the image processing device 12A of the present embodiment includes a background difference calculation unit 17, a motion vector calculation unit 18, an obstacle position estimation unit 19A, a movement path area calculation unit 20, an obstacle position detection unit 21A, A storage unit 22 is provided. The obstacle position estimating unit 19A according to the present embodiment adds or changes the obstacle identification function shown in step S113 of FIG. 14 to the obstacle position estimating unit 19 according to the first embodiment (details will be described later) ). Further, the obstacle position detection unit 21A of the present embodiment adds or changes the generation function of the position detection area of the obstacle shown in step S109A of FIG. 14 to the obstacle position detection unit 21 of the first embodiment. (Details will be described later).

  In the mine, for example, as shown in FIG. 15, the vehicle 51 stops around the dump truck 1 because the operator of the dump truck 1 is replaced, and a person 52 (alternate worker) gets off from the vehicle 51 while stopped. May approach toward dump truck 1 (own machine). That is, the vehicle 51 (first obstacle) in a stationary state and the person 52 (second obstacle) in an operating state may appear on the image acquired by the camera 11. Then, the movement route area of the person 52 may overlap with the current position detection area of the vehicle 51, and the detection accuracy of the position of the vehicle 51 may be impaired. The image processing apparatus 12A of the present embodiment has a function corresponding to such a problem, and the details thereof will be described.

  First, when the person 52 gets off from the stopped vehicle 51, the processing of the image 53 (see FIG. 16) acquired at that time will be described. The motion vector calculation unit 18 of the image processing device 12A acquires a motion vector generation area resulting from the motion of the person 52. When the obstacle position estimation unit 19A of the present embodiment acquires from the storage unit 22 the position detection area of the vehicle 51 immediately before and the fact that the vehicle 51 is stationary, the position detection area of the vehicle 51 immediately before is described above. The overlapping rate with the motion vector generation area is calculated, it is determined whether the overlapping rate is within a predetermined range (for example, 10 to 30%) set in advance, and an obstacle is identified based on the determination result. It is supposed to be.

  If the overlap rate exceeds the upper limit (30%) of the predetermined range, it is estimated that the vehicle 51 in the operating state is present in the image 53. That is, it is estimated that the vehicle 51 has changed from the stationary state to the operating state. Therefore, the position estimation area of the vehicle 51 at that time is generated based on the position information of the motion vector generation area acquired by the motion vector calculation unit 18. If the overlapping rate is less than the lower limit value (10%) of the predetermined range, it is estimated that although the vehicle 51 in the stationary state is present in the image 53, the person 52 in the operating state is not present. Therefore, the position estimation area of the vehicle 51 at that time is generated based on the position estimation area of the vehicle 51 (or the position estimation area of the vehicle 51 several times in the past including this) stored in the storage unit 22.

  When the overlapping rate is within the predetermined range, it is estimated that the vehicle 51 in the stationary state and the person 52 in the operating state exist in the image 53. Therefore, the position estimation area 54 of the vehicle 51 at that time is generated based on the position estimation area of the vehicle 51 immediately before (or the position estimation area of the vehicle 51 including the several times including this) stored in the storage unit 22. . Also, the position estimation area 55 of the person 52 at that time is generated based on the position information of the motion vector generation area acquired by the motion vector calculation unit 18. The obstacle position estimation area generated in this way is given the identification number of the obstacle (for example, vehicle = 1, person = 2), and is associated with the obstacle being stationary or operating. The data are stored in the storage unit 22 in chronological order.

  Next, processing of the image 56 (see FIG. 17) acquired at that time when the person 52 who got off from the vehicle 51 is approaching the own machine will be described. The motion vector calculation unit 18 of the image processing device 12A acquires a motion vector generation area resulting from the motion of the person 52. The obstacle position estimation unit 19A according to the present embodiment has at least the position estimation area of the vehicle 51 stored at least immediately before and the position estimation area of the person 52 stored immediately by the storage unit 22 although the overlapping rate is not within the predetermined range. , It is estimated that the stationary vehicle 51 and the operating person 52 are present in the image 56. Therefore, the position estimation area 54 of the current vehicle 51 is generated based on the position estimation area 54 of the vehicle 51 stored immediately in the storage unit 22 (or the position estimation area 54 of the vehicle 51 including the past several times). Do. Further, the position estimation area 57 of the current person 52 is generated based on the position information of the motion vector generation area acquired by the motion vector calculation unit 18. The obstacle position estimation area generated in this manner is given an identification number of the obstacle, and is stored in the storage unit 22 in chronological order in association with the obstacle being in a stationary state or in an operating state. Ru.

  Then, the movement route area calculation unit 20 calculates the movement route area of the vehicle 51 based on the history information of the position estimation area of the vehicle 51 stored in the storage unit 22. Specifically, the movement route area 58 (see FIG. 18) is calculated by the difference between the position estimation area group (not shown) of the vehicle 51 in the past and the position estimation area 54 of the current vehicle 51. Further, the movement path area of the person 52 is calculated based on the history information of the position estimation area of the person 52 stored in the storage unit 22. Specifically, the movement path area 59 (see FIG. 18) of the person 52 is calculated by the difference between the position estimation area group of the person 52 in the past (including the position estimation area 55) and the position estimation area 57 of the current person 52. Do.

  The background difference calculation unit 17 calculates the difference generation area 60 (see FIG. 19) based on the difference between the pixel value of the image 56 and the pixel value of the background model image 36. Then, the topography change difference exclusion unit 23A of the obstacle position detection unit 21A according to the present embodiment calculates the movement route area 58 of the vehicle 51 and the movement route of the person 52 from the difference generation area 60 calculated by the background difference calculation unit 17. By excluding the part of the area 59 which does not overlap with the position estimation area 54 of the current vehicle 51, the position detection area 61A (see FIG. 20) of the vehicle 51 and the position detection area 61B of the person 52 (see FIG. 20) Generate).

  The composite image generation unit 16 displays area markers 62A and 62B respectively indicating the position detection area 61B of the vehicle 51 and the position detection area 61B of the person 52 generated by the obstacle position detection section 21A in the image 56 from the image storage section 14. To generate a composite image 63 (see FIG. 21), and the composite image 63 is output to the display device 13 for display.

  Therefore, in the present embodiment, even if the movement route area of the person 52 overlaps the position detection area of the vehicle 51, the detection accuracy of the position of the vehicle 51 can be maintained.

  In the second embodiment, the obstacle position detection unit 21A calculates the vehicle 51 (first obstacle) calculated by the movement route area calculation unit 20 from the difference generation area calculated by the background difference calculation unit 17. By excluding the portions that do not overlap with the position estimation area of the current vehicle 51 in the movement path area of the person 52 (second obstacle) calculated by the movement path area calculation unit 20 and the movement path area of Although the case of acquiring the position detection area of the vehicle 51 and the position detection area of the person 52 has been described as an example, the present invention is not limited to this, and modifications are possible without departing from the spirit and technical concept of the present invention.

  For example, the movement route area calculation unit 20 calculates the movement route area of the person 52 based on the history information of the position estimation area of the person 52 (second obstacle), and the current vehicle 51 from the movement route area of the person 52 A portion overlapping with the position estimation area of the (first obstacle) may be deleted to generate a new movement path area, and the new movement path area may be reset as the movement path area of the person 52. Then, the terrain change difference exclusion unit 23A of the obstacle position detection unit 21 moves the movement route area of the vehicle 51 calculated by the movement route area calculation unit 20 from the difference generation area calculated by the background difference calculation unit 17 and The position detection area of the vehicle 51 and the position detection area of the person 52 may be acquired by excluding the movement path area of the person 52 reset by the path area calculation unit 20. Also in this case, the same effect as described above can be obtained.

  A third embodiment of the present invention will be described with reference to the drawings. FIG. 22 is a block diagram showing the functional configuration of the image processing apparatus according to this embodiment together with related devices. FIG. 23 is a flowchart showing the processing content of the image processing apparatus according to this embodiment. In the present embodiment, parts equivalent to those in the first and second embodiments are assigned the same reference numerals and descriptions thereof will be omitted as appropriate.

  The obstacle detection unit 15 of the image processing device 12B according to the present embodiment includes a background difference calculation unit 17, a motion vector calculation unit 18, an obstacle position estimation unit 19B, a movement path area calculation unit 20, an obstacle position detection unit 21B, A storage unit 22 is provided. The obstacle position estimation unit 19B of the present embodiment has the function of estimating the terrain change area shown in step S114 of FIG. 23 added to or changed from the obstacle position estimation unit 19 of the first embodiment (details will be described) Later). Further, the obstacle position detection unit 21B of the present embodiment adds or changes the generation function of the position detection area of the obstacle shown in step S109B of FIG. 23 to the obstacle position detection unit 21 of the first embodiment. (Details will be described later).

  In a mine, the wheel loader may advance while pushing soil, and then the wheel loader may move backward and leave the soil. At this time, since the earth and sand move for a while even if the wheel loader and the earth and sand are separated, there is a possibility that the earth and sand may be erroneously detected as an obstacle. The image processing apparatus 12B has a function corresponding to such a problem, and the details thereof will be described.

  First, at the time when the wheel loader advances while pushing out soil, processing of an image 71A or 71B (see FIG. 24A or 24B) acquired at that time will be described. The motion vector calculation unit 18 of the image processing device 12B acquires a motion vector generation area caused by the movement of the wheel loader and a motion vector generation area caused by the movement of the earth and sand. The obstacle position estimation unit 19B according to the present embodiment, like the obstacle position estimation unit 19 according to the first embodiment, can regard the directions of the motion vectors of the two motion vector generation areas as being the same. It is determined whether the distance between those centers is equal to or less than a predetermined threshold. In this case, since the directions of the motion vectors can be regarded as the same and the center-to-center distance is equal to or less than a predetermined threshold, one obstacle position estimation area 72A or 72B is generated based on two motion vector generation areas. Are stored in the storage unit 22.

  Next, when the wheel loader starts to retract, processing of the image 71C (see FIG. 25) acquired at that time will be described. The motion vector calculation unit 18 of the image processing device 12B acquires a motion vector generation area caused by the movement of the wheel loader and a motion vector generation area caused by the movement of the earth and sand. The obstacle position estimation unit 19B according to the present embodiment regards the directions of the motion vectors of the two motion vector generation areas as being the same as described above, and the distance between their centers is equal to or less than a predetermined threshold. Determine if there is. In this case, two obstacle position estimation areas 72C and 73 are respectively generated based on the two motion vector generation areas because the direction of the motion vector is different or the center-to-center distance exceeds a predetermined threshold. That is, the number of obstacle position estimation areas increases.

  Here, the obstacle position estimation unit 19B of the present embodiment calculates the center-to-center distance between the position estimation area 72B of one obstacle acquired last time and the entire two obstacle position estimation areas 72C and 73 acquired this time. Then, it is determined whether the center-to-center distance is equal to or less than a predetermined threshold (for example, 30 pixels). If the center-to-center distance exceeds a predetermined threshold value, it is estimated that two motion state obstacles exist in the image 71C. However, in this case, since the center-to-center distance is equal to or less than a predetermined threshold, it is estimated that one obstacle position estimation area is divided into two obstacle position estimations. Therefore, of the two position estimation areas 72C and 73, the position estimation area 72C having the largest area is stored in the storage unit 22 as a true position estimation area, and the remaining position estimation area 73 is stored as a topography change area. It memorizes in 22 (the estimation function of the terrain change area).

  Next, when the wheel loader is further retracted, processing of the image 74 (see FIG. 26) acquired at that time will be described. The motion vector calculation unit 18 of the image processing device 12B acquires a motion vector generation area resulting from the motion of the wheel loader. The obstacle position estimation unit 19B of the present embodiment generates a position estimation area 75 of one obstacle based on one motion vector generation area, similarly to the obstacle position estimation unit 19 of the first embodiment. Then, the movement route area calculation unit 20 calculates the movement route area of the obstacle based on the history information of the position estimation area of the obstacle stored in the storage unit 22. More specifically, the moving path area 76 of the obstacle is obtained by the difference between the position estimation area group of obstacles in the past (including the position estimation areas 72A, 72B and 72C) and the position estimation area 75 of the current obstacle (FIG. 27) Calculate (see).

  The background difference calculating unit 17 calculates the difference generation area 77 (see FIG. 28) based on the difference between the pixel value of the image 74 and the pixel value of the background model image 36. Then, the terrain change difference exclusion unit 23B of the obstacle position detection unit 21B according to the present embodiment moves the obstacle calculated by the movement path area calculation unit 20 from the difference generation area 77 calculated by the background difference calculation unit 17. By excluding the route area 76 and the terrain change area 73 stored in the storage unit 22, the position detection area 78 (see FIG. 29) of the obstacle is calculated.

  The composite image generation unit 16 superimposes the area marker 79 indicating the obstacle position detection area 78 generated by the obstacle position detection unit 21B on the image 74 from the image storage unit 14 (see FIG. 30). ) Is output to the display device 13 for display.

  Therefore, in the present embodiment, it is possible to prevent the false detection of soil as an obstacle.

  In the first to third embodiments, when the composite image generation unit 16 of the image processing apparatus generates a composite image, the case where only the area marker indicating the position detection area of the obstacle is superimposed is described as an example. However, the present invention is not limited to this, and modifications can be made without departing from the spirit and technical concept of the present invention. That is, when generating the composite image, the composite image generation unit 16 indicates the motion state and the motion direction of the obstacle (in the present modification, one of the left direction, the right direction, the upper direction, and the lower direction). A marker 81A (see FIG. 31A) and a marker 81B (see FIG. 31B) indicating the stationary state of the obstacle (in other words, no motion direction) may be selectively superimposed. In such a modification, the operator of the dump truck 1 can be notified of the state of the obstacle, and the operation of the dump truck 1 can be further assisted.

  In the first to third embodiments, the camera 11 is provided only on the front side of the dump truck 1 as an example. However, the present invention is not limited thereto, and the scope without departing from the spirit and technical concept of the present invention Modification is possible within. That is, the camera may be provided at at least one of the front side, the rear side, the left side, and the right side of the dump truck 1, for example. For example, when cameras are provided at four places on the front side, rear side, left side, and right side of the dump truck 1, images taken by four cameras may be combined and displayed as one image. The same effects as described above can be obtained also in such a modified example.

  In the first to third embodiments, the own-vehicle-stop determination unit of the image processing apparatus stops the own-vehicle based on the traveling speed of the own-machine detected by the speed sensor or the operation position of the shift lever. Although the case of determining the presence or absence is described as an example, the present invention is not limited to this, and modifications are possible without departing from the spirit and technical concept of the present invention. That is, it may be determined whether the own machine is at a stop by comparing the current image and the past image. The same effects as described above can be obtained also in such a modified example.

  In addition, although the dump truck was demonstrated to the example as a working machine of application object of this invention above, it is not restricted to this, For example, a hydraulic shovel, a wheel loader, etc. may be sufficient.

DESCRIPTION OF SYMBOLS 1 Dump truck 11 Cameras 12, 12A, 12B Image processing apparatus 13 Display apparatus 14 Image storage part (1st storage part)
15 obstacle detection unit 16 composite image generation unit 17 background difference calculation unit 18 motion vector calculation unit 19, 19A, 19B obstacle position estimation unit 20 movement route area calculation unit 21, 21A, 21B obstacle position detection unit 22 storage unit ( Second storage unit)
31 current image 32 previous image 36 background model image 37 difference generation area 40 motion vector 41A, 41B motion vector generation area 42 current obstacle position estimation area 43 past obstacle position estimation area group 44 moving path area of obstacle 45 position detection area of obstacle 46 area marker 47 composite image 51 vehicle (first obstacle)
52 people (second obstacle)
53 past image 54 past and present vehicle position estimation area 55 past person's position estimation area 56 current image 57 current person's position estimation area 58 vehicle's movement path area 59 person's movement path area 60 difference generation area 61A Vehicle position detection area 61B Person position detection area 62A, 62B Area marker 63 Composite image 71A, 71B, 71C Past images 72A, 72B, 72C Past obstacle position estimation area 73 Terrain change area 74 Current image 75 Obstacle position estimation area 76 Moving path area of obstacle 77 Difference generation area 78 Obstacle position detection area 79 Area marker 80 Composite image 81A, 81B Marker

Claims (4)

A work machine comprising: a camera for photographing the surroundings of the work machine, an image processing device for detecting an obstacle in an image acquired by the camera, and a display device for displaying an image processed by the image processing device In the obstacle detection system
The image processing apparatus is
A first storage unit that stores images acquired by the camera in time series;
A background difference calculating unit that calculates a difference generation area based on a difference between a pixel value of a current image acquired by the camera and a pixel value of a background model image created in advance;
The current image and the immediately preceding image are compared to calculate a motion vector of pixels between the two images, and pixels of the current image having motion vectors that can be regarded as the same as each other are collected to obtain a motion vector generation area A motion vector calculation unit,
When the area of the motion vector generation area acquired by the motion vector calculation unit is equal to or larger than a predetermined threshold value, at least one obstacle indicating the area in which the obstacle exists based on the position information of the motion vector generation area An obstacle position estimation unit that generates an object position estimation area;
A second storage unit that stores the obstacle position estimation area generated by the obstacle position estimation unit in time series;
A movement route area calculation unit that calculates a movement route area of the obstacle based on history information of the obstacle position estimation area stored in the second storage unit;
An obstacle position detection unit that generates the position detection area of the obstacle by excluding the movement path area calculated by the movement path area calculation unit from the difference generation area calculated by the background difference calculation unit;
An area marker indicating the position detection area of the obstacle generated by the obstacle position detection unit is superimposed on the current image acquired by the camera to generate a composite image, and the composite image is transmitted to the display device. An obstacle detection system for a working machine, comprising: a composite image generation unit for outputting. In the obstacle detection system according to claim 1,
The second storage unit stores position detection areas of obstacles generated by the obstacle position detection unit in time series in association with whether the obstacle is in the operating state or the stationary state.
The obstacle position estimation unit acquires the position detection area of the immediately preceding first obstacle from the second storage unit, and the position detection area and the motion vector generation area when the first obstacle is stationary. Is determined within a predetermined range set in advance, and if the overlap rate is within the predetermined range, the first obstacle is separated based on the motion vector generation area. Generate an obstacle position estimation area indicating the existence area of the second obstacle of
The movement path area calculation unit calculates the movement path area of the first obstacle and the movement path area of the second obstacle,
The obstacle position detection unit is configured to calculate a movement path area of the first obstacle and a current movement path area of the second obstacle from the difference generation area calculated by the background difference calculation section. (1) A position detection area of the first obstacle and a position detection area of the second obstacle are generated by excluding a portion not overlapping the position estimation area of the obstacle,
The composite image generation unit, when generating the composite image, superimposes a first area marker indicating a position detection area of the first obstacle and a second area marker indicating a position detection area of the second obstacle. An obstacle detection system for a working machine characterized by In the obstacle detection system for a working machine according to claim 1,
When the number of obstacle position estimation areas increases, the obstacle position estimation unit calculates the center-to-center distance between one previously acquired obstacle position estimation area and the plurality of increased obstacle position estimation areas. When the center-to-center distance is equal to or less than a predetermined threshold value, it is estimated that the plurality of increased obstacle position estimation areas are separated from the one previously acquired obstacle position estimation area, Of the plurality of increased obstacle position estimation areas, the one with the largest area is stored in the second storage unit as a true obstacle position estimation area, and the remaining obstacle position estimation areas are regarded as the topography change area. 2 Store in storage unit,
The obstacle position detection unit is configured to exclude the movement path area calculated by the movement path area calculation unit and the topography change area from the difference generation area calculated by the background difference calculation unit. An obstacle detection system for a working machine, characterized by generating an obstacle position detection area. In the obstacle detection system for a working machine according to claim 1,
When generating the composite image, the composite image generation unit superimposes the operation direction information of the obstacle on the obstacle in addition to the area marker indicating the position detection area of the obstacle. system.

Images