JPWO2014073571A1 - Image processing apparatus for self-propelled industrial machine and image processing method for self-propelled industrial machine - Google Patents

Abstract

It is an object of the present invention to automatically set a warning range for detecting an obstacle in the visual field range of the camera and prevent the warning range from being shifted even if the visual field range of the camera is subsequently shifted. A camera (6) that is provided in the dump truck (1) and captures an area in which the front structure (10) of the dump truck (1) is included in the field of view, and image data (P) captured by the camera (6). Based on the structure detection unit for detecting the forward structure (10) in the image data (P) and the range where the obstacle (M) needs to be detected in the image data (P) As (31), a warning range setting unit (26) that sets the warning range (31) in the image data (P) based on the forward structure (19) detected by the structure detection unit, I have.

Description

  The present invention relates to an image processing apparatus for a self-propelled industrial machine that performs predetermined image processing on image data captured by a photographing unit provided in the self-propelled industrial machine, and an image processing method thereof.

  Self-propelled industrial machines include transport machines such as dump trucks and work machines such as hydraulic excavators. Of these, the dump truck has a vessel that can be raised and lowered on the frame of the vehicle body. Excavated material excavated by a hydraulic excavator or the like is loaded on the vessel of the dump truck. After a predetermined amount of excavated material is loaded on the vessel, the dump truck travels to a predetermined accumulation site, and the excavated material loaded in the accumulation site is discharged.

  The dump truck is a large transport vehicle, and there are many blind spots that cannot be seen by an operator boarding the cab. For this purpose, a camera is mounted on the dump truck, and an image captured by the camera is displayed on a monitor in the cab. The camera takes a picture of the blind spot of the operator. Then, by displaying the video captured by the camera on the monitor, the operator can supplementarily obtain a blind spot field of view. Thereby, the operator can recognize the situation around the dump truck in detail.

  A self-propelled industrial machine includes a hydraulic excavator having a lower traveling body. This hydraulic excavator is also equipped with a camera to check the surrounding situation in detail. Japanese Patent Application Laid-Open No. 2004-133867 discloses a technique for recognizing an obstacle by performing predetermined image processing on a video acquired by a camera of a hydraulic excavator. In the technique of Patent Document 1, it is detected by image processing whether an intruding moving body is in the working range of the hydraulic excavator. By performing this image processing, the distance of the invading moving body and the certainty as the human body are evaluated.

JP-A-10-72851

  By the way, the camera has a predetermined visual field range, and an image of the visual field range is displayed on the monitor screen as image data. And like the technique of patent document 1 mentioned above, obstacles, such as a worker, a service car, and another work machine, are detected by performing predetermined image processing with respect to image data. Thereby, the operator boarding a driver's cab can recognize that when an obstacle is detected.

  Here, the range close to the self-propelled industrial machine such as the dump truck needs to be warned, and this range is set as the alert range. This warning area can be set as a range where the dump truck and the obstacle are likely to come into contact with each other. Therefore, the entire alert range needs to be included in the visual field range of the camera. That is, the alert range is a part of the visual field range of the camera. For this reason, a range other than the vigilance range occurs in the visual field range of the camera. The range other than the warning range is a warning-free range, and even if an obstacle exists in this range, no trouble is caused. That is, there is no need to know the operator.

  If an obstacle is detected by performing image processing up to this warning-unnecessary range, the operator is informed of information that does not need to be recognized by the operator, which causes confusion for the operator. Thereby, work efficiency falls. Further, the warning-unnecessary range is a distant range in the visual field range of the camera, and the pixels when this range is imaged become rough. As a result, even objects that are not obstacles may be erroneously detected as obstacles, and normal obstacles cannot be detected. Furthermore, when image processing is performed up to a warning-unnecessary range, there is a problem that the amount of information to be processed increases and the time required for processing increases.

  Therefore, it is important to set a warning area within the visual field range of the camera. However, it is very difficult to manually set the alert range based on the image data acquired from the camera, and the time and effort for setting it are extremely complicated. In addition, since self-propelled industrial machines such as dump trucks are operated in harsh sites, there is a risk that the mounting position and angle of the camera may be shifted due to vibration of the vehicle body. As a result, even if the warning area is initially set, the warning area is shifted afterwards, and the obstacle cannot be normally detected.

  Therefore, the present invention automatically sets a warning range for detecting an obstacle in the camera field of view, and does not cause a shift in the required warning range even if the camera field of view is shifted later. With the goal.

  In order to solve the above-described problems, an image processing apparatus for a self-propelled industrial machine according to the present invention is provided in a self-propelled industrial machine, and includes a part of a structure constituting the self-propelled industrial machine in view. An imaging unit that images a peripheral region, a structure detection unit that detects the structure in the image data based on image data captured by the imaging unit, and a position in the peripheral region of the image data It is necessary to detect an obstacle to be detected, set a predetermined range including the obstacle as a warning area, and set the warning area in the image data based on the structure detected by the structure detection unit. A warning range setting unit.

  Moreover, you may provide the obstruction detection part which performs the process which detects the said obstruction only with respect to the said vigilance range in the said image data.

  In addition, a range other than the alert required range in the image data is set as an unnecessary alert range, and a difference between pixel values in the alert unnecessary range that is temporally changed is calculated. A deviation determination unit may be provided that determines whether or not a deviation greater than an allowable range has occurred in the visual field range of the imaging unit based on the number of pixels that are equal to or greater than a threshold value.

  The self-propelled industrial machine may include a travel speed detection unit that detects a travel speed, and the warning range may be changed based on the travel speed detected by the travel speed detection unit.

  The image picked up by the photographing unit is displayed on the image display means. As the display mode, in addition to those described above, the image of the image pickup unit is subjected to viewpoint conversion by signal processing to obtain an overhead view image as an upper viewpoint. It is good also as a structure.

  Further, the image processing method of the self-propelled industrial machine of the present invention is provided with a photographing unit in the self-propelled industrial machine, photographs a peripheral region including a part of the structure of the self-propelled industrial machine, Based on the image data photographed by the photographing unit, the structure in the image data is detected, an obstacle located in a peripheral region of the image data is detected, and a predetermined range including the obstacle Is set in the image data based on the structure detected by the structure detection unit, and the failure is limited only to the warning range in the image data. Processing to detect objects is performed.

  In the present invention, the photographing unit performs photographing so that the structure of the self-propelled industrial machine is included in the field of view, detects the structure edge, and sets a warning range based on the structure edge. Thereby, it is possible to automatically set a vigilance range for detecting an obstacle in the visual field range of the camera. In addition, even if there is a shift in the visual field range of the camera afterwards, there is no shift in the alert range.

It is a top view of a dump truck. It is a side view of a dump truck. 1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment. 3 is a flowchart illustrating an operation of the image processing apparatus according to the first embodiment. It is a figure which shows an example of image data. It is a figure which shows an example of the image data which performed distortion correction. It is a figure explaining an example of the smoothing process with respect to image data. It is a figure which shows an example of the image data which performed the smoothing process. It is a figure explaining an example of an edge extraction process. It is a figure which shows an example of the image data by which the edge extraction process was carried out, (a) is the image data from which the edge pixel was extracted, (b) is the image data after image conversion processing of the image data shown in (a). . It is a figure which shows an example of a vigilance range and a vigilance unnecessary range, (a) is the image data in which the vigilance range is set based on edge line segment data, (b) is the image data shown in (a). This is image data to which distortion restoration processing is added. It is a figure which shows an example of the display image of 1st Embodiment. It is a block diagram which shows the structure of the image processing apparatus of 2nd Embodiment. It is a flowchart which shows operation | movement of the image processing apparatus of 2nd Embodiment. It is a figure which shows an example of the vigilance range image. It is a figure which shows an example of the detection method of the moving body using the block matching method. It is a figure which shows an example when a comparison range is scanned in a search range. It is the figure explaining the block matching method three-dimensionally. It is a figure which shows an example of the display image of 2nd Embodiment. It is a figure which shows an example for obtaining a difference image from a mask image. It is a figure which shows an example of the display image on which the shift | offset | difference information was superimposed. It is a figure which shows an example which identifies a front structure in case the shift | offset | difference has arisen in the visual field range, (a) is image data from which the edge pixel was extracted, (b) is image conversion of the image data shown in (a) This is image data after processing. It is a figure which shows an example of the warning required range in case the deviation | shift has occurred in the visual field range, and a warning unnecessary range, (a) is the image data by which the required warning range is set based on edge line segment data, (b) is This is image data obtained by applying distortion restoration processing to the image data shown in (a). It is a figure which shows an example of the vigilance range in case the shift | offset | difference has arisen in the visual field range. It is a block diagram which shows the structure of the image processing apparatus of 3rd Embodiment. It is a figure which shows an example of the image data of 4th Embodiment. It is a figure which shows an example of the image which binarized the image data of FIG. It is a figure which shows an example of the display image of 4th Embodiment. It is explanatory drawing which shows the principle which carries out a bird's-eye view imaging process of a camera image. It is a figure which shows an example which carried out the bird's-eye display of the display image of 2nd Embodiment, (a) shows the area | region which looks down at the image data of FIG. 19, (b) is an example of the bird's-eye view image. It is a figure which shows an example of the display image of 3rd Embodiment.

A: First Embodiment of the Present Invention Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. Here, a dump truck as a transport vehicle is applied to a self-propelled industrial machine. Of course, any self-propelled industrial machine other than the dump truck, such as a hydraulic excavator having a lower traveling body, may be applied. There are a rigid type and an articulate type as the dump truck, but any of them may be applied. In short, the present embodiment can be applied to any self-propelled industrial machine that performs predetermined work (transportation, excavation, etc.). In the following description, “left” is the left side when viewed from the cab, and “right” is the right side when viewed from the cab.

  FIG. 1 shows an example of a dump truck 1 according to this embodiment. The dump truck 1 includes a body frame 2, a front wheel 3 (3L and 3R), a rear wheel 4 (4L and 4R), a vessel 5, a camera 6, a cab 7, an image processing device 8, and a monitor 9. Yes. The body frame 2 forms the main body of the dump truck 1, and a front wheel 3 is provided in front of the body frame 2 and a rear wheel 4 is provided in the rear. The front wheel 3R is the right front wheel 3, and the front wheel 3L is the left front wheel 3. The rear wheel 4R is the right rear wheel 4, and the rear wheel 4L is the left rear wheel 4. The vessel 5 is a loading platform on which earth and sand, minerals, and the like are loaded. The vessel 5 is configured to be undulating.

  The dump truck 1 can be provided with a camera 6 as a photographing unit at an arbitrary position. FIG. 2 shows a side view of the dump truck 1, and the camera 6 is attached in front of the cab 7. The camera 6 takes a picture of the peripheral area of the dump truck 1 in a visual field range 6A (a range indicated by a broken line in the figure) in which a diagonally lower portion in front of the dump truck 1 is looked down. In this visual field range 6A, the worker M exists as an obstacle to avoid contact. Other obstacles include work machines and service cars, and these may be included in the visual field range 6A.

  The visual field range 6A of the camera 6 is obliquely below the front of the dump truck 1, and a part of the front structure 10 is included in the visual field range 6A. As shown by a broken line in FIG. 2, a part of the front structure 10 is included in the visual field range 6A. The visual field range 6 </ b> A that is a peripheral region photographed by the camera 6 may be a direction other than the front of the dump truck 1, and an arbitrary direction such as the rear, left, or right of the dump truck 1 may be used as the visual field. By setting the visual field range 6 </ b> A to the rear of the dump truck 1, when the dump truck 1 is moved backward, the rear situation can be displayed as an image. At this time, a part of the rear structure is included in the visual field range 6A.

  The driver's cab 7 is provided with various operation means for operating the dump truck 1 by boarding an operator. For example, a shift lever or the like for moving the dump truck 1 forward or backward is provided as the operation means. The cab 7 is provided with an image processing device 8 and a monitor 9, and image data generated when the camera 6 takes an image is subjected to predetermined image processing by the image processing device 8. The image data that has undergone image processing is displayed on the monitor 9. The monitor 9 is a display device. Basically, the monitor 9 displays an image captured by the camera 6.

  The image processing device 8 is an image processing controller that performs image processing on image data of video captured by the camera 6. Then, the image data subjected to the image processing is displayed on the monitor 9 as a display image. As shown in FIG. 3, the image processing apparatus 8 includes a distortion correction unit 21, a smoothing processing unit 22, an edge extraction unit 23, a structure edge determination unit 24, an edge line segment data determination unit 25, and a warning range setting unit 26. And a distortion restoration unit 27 and a display image creation unit 28.

  The video imaged by the camera 6 is input to the distortion correction unit 21 as image data. The distortion correction unit 21 corrects the distortion of the image data using the lens parameters of the camera 6 (focal length, center of the optical axis of the image, lens distortion coefficient, etc.). The image reflected by the lens of the camera 6 is not distorted at the center, but is distorted as the distance from the center increases. Accordingly, the image data obtained by photographing by the camera 6 is distorted as a whole. On the other hand, image data with high linearity can be obtained by performing correction using the above parameters.

  The smoothing processing unit 22 receives the image data corrected by the distortion correction unit 21 and performs a smoothing process. The smoothing process is a process of generating a smooth image by reducing changes in pixel values between pixels of image data. By performing the smoothing process on the image data, the change of the pixel value can be smoothed, and the image data can be smoothed as a whole. Although any method can be used for the smoothing process, for example, a moving average filter, a median filter, a Gaussian filter, or the like can be used.

  The edge extraction unit 23 performs edge extraction processing on the image data smoothed by the smoothing processing unit 22. For this purpose, a pixel having a large change in pixel value between adjacent pixels is extracted as an edge pixel, and a shape in which the edge pixels are continuously formed is defined as an edge. Thereby, an edge having a large change in pixel value is extracted from the image data. Although any method can be used for the edge extraction process, a method such as a Sobel filter or a Canney algorithm can be used.

  The structure edge determination unit 24 that is a structure detection unit uses, as the edge of the front structure 10, an edge whose shape is close to the boundary of the front structure 10 in the image data among the edges extracted by the edge extraction unit 23. judge. For this purpose, the structure edge determination unit 24 linearizes edges in the image data using a linearization method such as Hough transform. Then, the straightened edge is determined as the edge line segment, and the edge line segment having high connectivity is determined as the boundary (edge) of the front structure 10. The edge line segment can be expressed by using coordinates in the image data. Here, the intersection coordinates of the edge line segments and the edge line segment and the lower end of the image data (of the image data P closest to the dump truck 1). The coordinates of the intersection with the edge portion are stored as edge line segment data.

  The edge line segment data determination unit 25 compares any number of edge line segment data including the latest edge line segment data. Since the camera 6 has a predetermined shooting period and image data is periodically acquired, edge line segment data is also periodically acquired. The edge line segment data determination unit 25 stores past edge line segment data, and calculates an average value with the latest edge line segment data. Then, the average value of the calculated edge line segment data is set as the determined edge line segment data. Each part from the distortion correction unit 21 to the edge line segment data determination unit 25 serves as a structure detection unit for detecting the front structure 10.

  The vigilance range setting unit 26 sets a vigilance range based on the edge line segment data input from the edge line segment data determination unit 25. The relative positional relationship between the warning area and the front structure 10 of the dump truck 1 is constant, and the warning area and the front structure are based on the mounting height and depression angle of the camera 6, the number of pixels of the camera 6, the angle of view, and the like. The relative positional relationship with 10 can be recognized as a known one. Therefore, the alert range setting unit 26 sets the predetermined range as the alert range based on the boundary of the front structure 10 obtained from the edge line segment data.

  The distortion restoration unit 27 restores distortion using the camera lens parameters corrected by the distortion correction unit 21. The image data up to the distortion restoring unit 27 is data obtained by correcting the distortion of the camera lens of the camera 6, and the original image of the camera 6 is distorted. Therefore, when displaying the original video of the camera 6, the distortion restoration unit 27 performs a process of restoring distortion on the image data. Due to the restoration of the distortion, the distortion also occurs in the area requiring caution in the image data.

  The display image creation unit 28 generates a display image to be displayed on the monitor 9. The display image at this time is image data whose distortion has been restored, and a warning area is set therein. The display image creation unit 28 outputs the display image in which the alert range is set to the monitor 9 and displays the display image on the monitor 9. As a result, an operator boarding the cab 7 can recognize the image of the camera 6 by visually recognizing the monitor 9.

  Next, the operation will be described based on the flowchart of FIG. The camera 6 installed in the dump truck 1 captures a visual field range 6A with a visual field in an obliquely downward direction, and performs imaging at a predetermined imaging cycle (for example, frames / second). As a result, the image data for one frame obtained at each photographing cycle is periodically output to the image processing device 8 (step S1). The camera 6 is installed so that the front structure 10 is included in the visual field range 6A, and the front structure 10 is included in the image data.

  An example of image data obtained when the camera 6 captures the visual field range 6A is shown in FIG. The camera 6 has a field of view obliquely below, and as shown in the figure, the image data P shows the ground in the shooting direction of the camera 6 as a background. The site where the dump truck 1 is operated is in a harsh environment, the ground is not leveled, and there are many irregularities. In the image data P, it is projected as an uneven portion R. The field of view of the camera 6 is set so that the front structure 10 of the dump truck 1 is projected in the field of view range 6A. Therefore, the front structure 10 is also included in the image data P.

  The above image data P is output to the distortion correction unit 21 of the image processing apparatus 8 and also to the display image creation unit 28. The image data P input to the distortion correction unit 21 is corrected for distortion based on the lens parameters of the camera 6 (step S2). By performing distortion correction of the image data P using the lens parameters of the camera 6, the distortion of the part separated from the center of the image data P is corrected. As a result, the image data P is corrected to an image having high linearity as shown in FIG.

  The image data P subjected to the distortion correction process is input to the smoothing processing unit 22. The smoothing processing unit 22 performs a smoothing process on the image data P (step S3). As described above, the smoothing processing unit 22 generates smooth image data P by reducing the difference in pixel value between adjacent pixels for each pixel of the image data P. As described above, edge extraction processing is performed by the edge extraction unit 23. At this time, the pixel values of the image data P are sparse due to unevenness of the ground, a difference in illuminance, and the like, and these may be detected as edges. However, these are noise components, and the smoothing processing unit 22 performs smoothing processing in advance to remove the noise components. By performing this smoothing process, sparse pixel values are smoothed and noise components are removed. Here, the smoothing process is performed using the smoothing filter illustrated in FIG. 7, but any technique can be used as long as the difference in pixel values between adjacent pixels can be reduced.

FIG. 7 shows an example in which a Gaussian filter is applied as a smoothing filter. The Gaussian filter is a filter having a large filter coefficient for a pixel close to one pixel (center pixel) in the image data P and a small filter coefficient for a pixel far from the center pixel in consideration of the spatial arrangement of the pixels. The Gaussian filter then takes a weighted average of each filtered pixel. As the filter coefficient, for example, the following Gaussian function (Formula 1) can be used. In Equation 1, σ is a standard deviation.

In the example of FIG. 7, a Gaussian filter having a total of 9 pixels of 3 pixels × 3 pixels is used. Note that the number of pixels of the Gaussian filter is not limited to nine, and may be a total of 25 pixels of 5 pixels × 5 pixels or more. The Gaussian filter using the Gaussian function of the above-described formula 1 is composed of a central pixel and peripheral pixels (8 pixels). The filter coefficient K of the center pixel is the largest, the filter coefficients L of the four pixels above, below, left, and right of the center pixel are the next largest, and the filter coefficients M of the other four pixels are the smallest. That is, “K>L> M”. The filter coefficients are weighted, and the total of the filter coefficients is “1”.

  The smoothing processing unit 22 performs the smoothing process using the Gaussian filter described above on all the pixels of the image data P. Specifically, the pixel value of the center pixel is multiplied by the filter coefficient K, the pixel values of the four pixels above, below, left, and right of the center pixel are multiplied by the filter coefficient L, and the pixel values of the other four pixels are multiplied. The filter coefficient M is multiplied. Then, the pixel values of the surrounding eight pixels multiplied by the filter coefficients L and M are added to the pixel value of the central pixel multiplied by the filter coefficient K. The pixel value of the center pixel when this addition is performed becomes a smoothed value. This process is performed for all the pixels of the image data P.

  By the way, as described above, the image data P shows the uneven portion R of the ground and the front structure 10. The pixel of the uneven portion R has a difference in pixel value from the surrounding pixels. Therefore, when the smoothing processing unit 22 performs the smoothing process on the image data P, the difference in the pixel value between the concavo-convex part R and the surrounding pixels is reduced and smoothed. The values of the filter coefficients K, L, and M described above are values such that the boundary of the front structure 10 in the image data P is detected as an edge, but the other uneven portions R and the like are not detected as edges. Set to. Thereby, as shown in FIG. 8, the uneven | corrugated | grooved part R in the image data P can be reduced significantly. Since the values of the filter coefficients K, L, and M are determined by the standard deviation σ of the Gaussian function, the standard deviation σ is set such that the uneven portion R is not detected.

  Of course, in the uneven portion R, the difference between the pixels of the uneven portion R and the surrounding pixels may be large depending on the size, illuminance conditions, and the like. In this case, even if the smoothing process is performed, the uneven portion R cannot be completely removed from the image data P, and a portion of the uneven portion R remains. However, as shown in FIG. 8, the number can be significantly reduced.

  On the other hand, for the front structure 10, the boundary between the background and the front structure 10 can be clearly distinguished. In general, the dump truck 1 is often colored in a color that can be easily distinguished from the ground or the surrounding environment from the viewpoint of grasping the surrounding situation. Accordingly, the front structure 10 is also clearly different from the background such as the ground and the surrounding environment. For this reason, the difference between the boundary between the background and the front structure 10, that is, the pixel value of the background in the front structure 10 and the pixel value of the front structure 10 is large. Thereby, since the difference of the pixel value in the boundary of the front structure 10 is very large, even if it performs a smoothing process, most remain.

  Next, processing of the edge extraction unit 23 will be described. As shown in FIG. 8, in the image data P that has been subjected to the smoothing process, the uneven part R and the boundary between the front structures 10 whose number has been greatly reduced are reflected. The edge extraction unit 23 extracts, as edge pixels, pixels whose pixel value difference between adjacent pixels is equal to or greater than a predetermined threshold (pixels having a large difference) from the image data P subjected to the smoothing process (step S4). .

  Here, Canny's algorithm is used as edge extraction processing for extracting edge pixels. Of course, the edge pixels may be extracted using other methods. In the Canny algorithm, two types of Sobel filters are used: a Sobel filter in the horizontal direction (lateral direction: X direction) and a Sobel filter in the vertical direction (vertical direction: Y direction) as shown in FIG. The X direction Sobel filter emphasizes the X direction outline of the image data P, and the Y direction Sobel filter emphasizes the Y direction outline. Edge pixels are extracted for all pixels of the image data P using these two types of Sobel filters.

By applying a Sobel filter in the X direction and the Y direction to all the pixels of the image data P, for all the pixels of the image data P, differential values (fx, fy) between the pixel and its surrounding pixels. ). fx is a differential value in the X direction, and fy is a differential value in the Y direction. Then, the edge strength g is calculated as “g = (fx ² + fy ² ) ^1/2 ” based on fx and fy for all pixels of P of the image data. The edge strength g indicates a difference in pixel value between one pixel and surrounding pixels, and an edge pixel is detected based on the calculated edge strength g.

  The edge extraction unit 23 compares the edge intensity g of all the pixels of the image data P to which the Sobel filter has been applied, using the first threshold T1 and the second threshold T2 (where T1> T2). . Focusing on one pixel of the image data P, if the edge intensity g of the pixel is not less than the first threshold T1, the pixel is detected as an edge pixel. On the other hand, if the edge intensity g of the pixel of interest is less than the second threshold T2, the pixel is detected as not being an edge pixel. When the edge intensity of the pixel of interest is greater than or equal to the first threshold value T1 and less than the second threshold value T2, if the edge pixel exists in the vicinity of the pixel (particularly, the pixel adjacent to the pixel), the pixel is edged Detected as pixels, otherwise not detected as edge pixels.

  When the above edge extraction processing is performed, image data P from which edge pixels are extracted as shown in FIG. 10A is generated. A shape in which edge pixels are continuous is extracted as an edge E. By performing the structure edge determination process on the image data P from which the edge E has been extracted, the front structure 10 is determined from the image data P (step S5). In the edge extraction process, pixels with a large difference between adjacent pixels are extracted as edge pixels. Thereby, since the pixel which originally projected the uneven part R is hardly detected as an edge pixel, the accuracy of detecting the front structure 10 is greatly improved.

  On the other hand, the boundary of the front structure 10 is likely to be detected as an edge pixel because of a large difference in pixel values. However, by performing an edge extraction process using a Sobel filter, the edge E may not show linearity and may show irregular changes as shown in FIG. For example, the boundary of the forward structure 10 that originally has linearity may also lose linearity, and a part of the boundary may not be detected.

  The structure edge determination unit 24 extracts a straight line from the edge pixels of the image data P. Arbitrary means can be used as means for extracting a straight line, but here Hough transform is used as a known technique. Of course, any method other than the Hough transform may be used as long as a straight line can be extracted from the edge pixels. By performing Hough transform arithmetic processing on the image data P shown in FIG. 10A, line segments indicating irregular changes are represented by straight lines L, L1 to L4 as shown in FIG. become. Of these, attention is paid to the straight lines L1 and L2 extending in the X direction and the straight lines L3 and L4 extending in the Y direction. And the structure edge determination part 24 selects the longest straight line L1 among the straight lines L1 and L2, and selects two straight lines L3 and L4.

  The structure edge determination unit 24 extends the straight lines L1, L3, and L4 in the image data P. Since the straight line L1 is a straight line extending in the X direction and the straight lines L3 and L4 are straight lines extending in the Y direction, the straight line L1, the straight line L3, and the straight line L4 intersect by extending these. Let the intersection of the straight line L1 and the straight line L3 be C1, and let the intersection of the straight line L1 and the straight line L4 be C2. Further, an intersection between the straight line L3 and the lower end of the image data P is C3, and an intersection between the straight line L4 and the lower end of the image data P is C4. Intersection points C1 to C4 indicate the coordinates of pixels in the two-dimensional image data P.

  When the intersections C1 to C4 are connected by straight lines, an edge line segment of the front structure 10 is formed. Therefore, the information on the intersections C1 to C4 becomes data for specifying the edge line segment of the front structure 10, and this is used as the edge line segment data. A region surrounded by the edge line segment data (coordinate information of the intersections C1 to C4) is the front structure 10. The structure edge determination unit 24 determines the edge of the front structure 10 based on the edge line segment data. The front structure 10 may be specified based on the edge line segment data obtained from one piece of image data P. In order to improve the accuracy of detecting the front structure 10 in the image data P, the edge line 10 The minute data determining unit 25 compares the edge line segment data (step S6).

  As described above, the image data P is input from the camera 6 to the image processing device 8 at a predetermined cycle, and a plurality of pieces of image data P are obtained. The edge line segment data determination unit 25 stores edge line segment data based on the past image data P, and calculates an average value of the edge line segment data based on the latest image data P and the past edge segment data. . The edge line segment data determination unit 25 sets the calculated edge line segment data (average value) as determined edge line segment data. That is, the forward structure 10 in the image data P is specified by the determined edge line segment data.

  Note that when the edge line segment data determination unit 25 calculates the average value of the edge line segment data of the plurality of image data P, the edge line segment data of a predetermined number of pieces of image data P is accumulated in the edge line segment data determination unit 25. Until this is done, the edge line segment data is not confirmed. Also, by calculating the average value of a large amount of edge line segment data, the accuracy of the edge line segment data is improved, but the processing time is lengthened. Therefore, the number of edge line segment data for calculating the average value is determined in consideration of the balance between the accuracy of the edge line segment data and the processing time.

  The edge line segment data determined by the edge line segment data determination unit 25 is output to the alert range setting unit 26. The vigilance range setting unit 26 sets a vigilance range based on the edge line segment data (step S7). The vigilance range is a range in which an obstacle is detected, and is a predetermined range from the dump truck 1. As described above, the relative positional relationship between the front structure 10 of the dump truck 1 and the alert range can be recognized as a known one. Therefore, if the forward structure 10 in the image data P can be specified, the area requiring caution can be automatically recognized.

  FIG. 11A shows a state where the alert range 31 is set based on the edge line segment data. The determined edge line segment data is information on the intersection points C1 to C4, and the front structure 10 is specified by connecting the intersection points C1 to C4. Accordingly, the area surrounded by the intersections C1 to C4 in FIG. Based on the relative positional relationship between the forward structure 10 and the alert range 31, the alert range 31 is set in the image data P.

  On the other hand, in the image data P, an area other than the area requiring caution 31 is a caution-free area 32 that does not require caution. Therefore, when an obstacle is detected from the image data P, the obstacle is detected only for the area requiring caution 31, and the obstacle is not detected for the area 32 not requiring caution. An arbitrary method can be adopted as a method for detecting an obstacle. When the obstacle is detected, the warning-unnecessary range 32 is set as a mask area, and the range can be masked, so that the obstacle can be detected by focusing on the warning range 31.

  The vigilance range 31 is set based on the image data P that has been subjected to distortion correction by the distortion correction unit 21. The display image displayed on the monitor 9 is image data P output from the camera 6 and is an image that has not been subjected to distortion correction. Therefore, the distortion restoration unit 27 restores distortion correction for the image data P in which the caution area 31 is set (step S8). The distortion restoration unit 27 restores distortion correction of the image data P using the lens parameters of the camera 6 used by the distortion correction unit 21. Thereby, the vigilance range 31 deform | transforms like FIG.11 (b). Information on the alert range 31 that has been subjected to the distortion restoration process is output to the display image creation unit 28.

  The display image creation unit 28 generates an image (display image) to be displayed on the monitor 9 (step S9). The display image creation unit 28 periodically inputs image data P from the camera 6. The image data P is an image that is not subjected to distortion correction. In addition, the display image creation unit 28 inputs the information of the caution area 31 that has been subjected to the distortion restoration process from the distortion restoration unit 27. Then, the display image creation unit 28 superimposes the alert range 31 on the image data P input from the camera 6. This state is shown in FIG. That is, in the image data P shown in the figure, the area requiring caution 31 is an area surrounded by virtual lines. The alert area 31 is an area for detecting an obstacle, and it is not necessary to display the alert area 31 as shown in FIG.

  Then, the display image creation unit 28 outputs the image data P input from the camera 6 to the monitor 9. An operator boarding the cab 7 can obtain an auxiliary visual field by visually recognizing the image data P displayed on the monitor 9.

  As described above, in the present embodiment, the front structure 10 of the dump truck 1 is specified based on the image data P, and the alert range 31 is set from the front structure 10. As a result, it is possible to automatically set the caution area 31 for detecting an obstacle without being manually operated, and the labor for setting can be largely omitted. Further, even if the dump truck 1 vibrates vigorously and the visual field range 6A of the camera 6 is shifted, the warning range 31 is not shifted. That is, since the relative positional relationship between the forward structure 10 and the alert range 31 in the image data P is constant when a deviation occurs in the visual field range 6A, the alert range 31 is set based on the forward structure 10. By setting, no shift occurs in the alert range 31 required.

  In the above description, the structure edge determination unit 24 detects the edge line segment of the front structure 10 by detecting the linearity of the edge pixels, but if the boundary between the front structure 10 and the background can be detected. A method other than the method of detecting the edge line segment of the front structure 10 may be used. For example, when the difference in pixel value between the front structure 10 and the background is extremely large, the front structure 10 can be detected based on the difference in pixel value. In this case, it is not necessary to perform distortion correction by the distortion correction unit 21, and it is not necessary to correct distortion recovery by the distortion recovery unit 27. However, since the front structure 10 has peculiar linearity, the accuracy of detecting the boundary between the front structure 10 and the background is improved by adopting the method of detecting the edge line segment of the front structure 10. To do.

  Further, the smoothing processing unit 22 performs a smoothing process on the image data P, thereby reducing the uneven portion R in the image data P. In this regard, when the ground imaged by the camera 6 is leveled to some extent, the uneven portion R in the image data P is reduced. Therefore, the edge E of the front structure 10 can be extracted without performing the smoothing process. However, in order to improve the detection accuracy of the edge E of the front structure 10, it is preferable to perform the smoothing process.

  Further, although the accuracy of the edge line segment data is improved by the edge line segment data determination unit 25, when the edge line segment can be clearly identified based on one image data P, the edge line segment data determination unit The processing may be omitted by 25. However, the accuracy of the edge line segment data is improved by determining the edge line segment data by calculating the average value of the edge line segment data of the plurality of image data P. For this reason, it is desirable to perform processing by the edge line segment data determination unit 25.

B: Second Embodiment of the Invention Next, a second embodiment will be described. FIG. 13 shows the configuration of the image processing apparatus 8 of the second embodiment. As shown in FIG. 13, a vigilance range image generation unit 41, an obstacle detection unit 42, a mask image generation unit 43, and a deviation determination unit 44 are newly added to the first embodiment. Since the configuration is the same as that of the first embodiment, the description thereof is omitted.

  Image data P is periodically output to the image processing device 8 from the camera 6 that captures the visual field range 6A. The image data P is output not only to the distortion correction unit 21 and the display image creation unit 28 but also to the alert range image generation unit 41. The vigilance range image generation unit 41 generates an image of only the vigilance range 31 in the image data P as shown in FIG. For this purpose, the caution area 31 in which the distortion is restored is input from the distortion recovery unit 27, and the image data P is input from the camera 6. And only the image in the area | region of the vigilance range 31 among the image data P is produced | generated as a vigilance range image.

  The vigilance range image is input to the obstacle detection unit 42. The obstacle detection unit 42 detects an obstacle in the alert range image. As obstacles that must be avoided, mainly workers and service cars are assumed, but other work machines and the like are also obstacles. Any method can be used as a method for detecting an obstacle. The obstacle has a stationary state and a moving state. The stationary obstacle is a stationary body, and the moving obstacle is a moving body.

  Although any method can be used as a method for detecting a stationary body, for example, a method for detecting a region of a stationary body by complexly judging an outline, luminance, color, and the like in an image can be used. In the case where the stationary body has a certain area and the outline of the certain area has a predetermined shape, the luminance value has a predetermined value, the color indicates a specific color, etc. The certain area can be detected as a stationary body.

  As a specific method for detecting a stationary object, for example, a pattern matching method can be used in which an obstacle is determined by matching using shape information learned in advance based on information on the contour between the background and the stationary object. . Also, use a method that has specific luminance and color information but extracts only pixels, extracts shape features such as area, perimeter, and moment of the extracted pixel area and identifies the shape as an obstacle. Can do.

  Any method can be used as a method for detecting the moving body. For example, when comparing image data P that are temporally different, the moving body moves when a certain region moves to a different position. Detect that. For example, a block matching method can be applied to the detection of the moving object. This block matching method will be described later. In addition, the gradient method to detect using the relationship between the spatial gradient of brightness and the temporal gradient in each pixel of the image data P, the image data when there is no moving object region, and the latest image data A background difference method for detecting a difference between the two, a frame difference method for detecting a moving body region by generating two difference images using three pieces of image data having different times, and the like can be used.

  The mask image generation unit 43 generates a mask image. As described in the first embodiment, the caution area setting unit 26 sets the caution area 31 in the image data P, and sets other areas as the caution-free area 32. Among these, the mask image generation unit 43 generates a mask image using the warning unnecessary range 32 as a mask area. The mask image is an image having the same size as the image data P, and has the same number of pixels. In the mask image, the mask area (warning-unnecessary range 32) and the other area (warning-required range 31) are clearly distinguished. For example, in the mask image, the mask area can be entirely composed of black (pixel value zero), and the area of the alert range 31 can be composed entirely of white (pixel value 256).

  The deviation determination unit 44 receives the mask image from the mask image generation unit 43 and determines the deviation. The deviation determination unit 44 holds a past mask image, and compares the newly input mask image with the past mask image. Then, the difference calculation between the new mask image and the past mask image is performed for all the pixels, and when there are a predetermined number or more of pixels having a pixel value difference equal to or larger than a predetermined value, the camera 6 is shifted. In other cases, it is determined that no deviation has occurred. The determination result is output to the display image creation unit 28.

  Next, the operation of the second embodiment will be described using the flowchart of FIG. Steps S1 to S8 shown in FIG. 14 are the same as those in the first embodiment, and thus description thereof is omitted. In step S8, a caution area 31 in which the distortion is restored is generated. Information on the alert range 31 is input to the alert range image generation unit 41. The vigilance range image generation unit 41 inputs image data P from the camera 6, and an area other than the vigilance range 31 in the image data P, that is, a vigilance-unnecessary range 32 is set as a non-sensitive area 33. The information of P has been deleted. The dead area 33 has the same function as the mask area described above. Here, the insensitive area 33 is composed of all black pixels (pixel value zero) (for convenience of drawing, the black pixels are shown by shading in FIG. 15).

  As a result, a warning area image P1 as shown in FIG. 15 is generated (step S10). The warning area image P1 includes a worker M as an obstacle. The alert range image P1 has the same number of pixels as the image data P in the X and Y directions, and the image data P is displayed in the portion of the alert range 31 but black in the insensitive region 33 portion. Only the pixels are displayed. In other words, the black pixels in the insensitive area 33 are masks, and an unmasked area in the image data P is the caution area 31. As shown in the figure, there are a large number of irregularities R in the area requiring caution 31 and a worker M as an obstacle. Such a vigilance range image P <b> 1 is output to the obstacle detection unit 42.

  The obstacle detection unit 42 detects an obstacle from the warning range image P1 (step S11). Although any method can be used to detect an obstacle, a case where a block matching method is used will be described here. FIG. 16 shows an example of a moving object detection method using a block matching method. The obstacle detection unit 42 stores the past alert range image P1 and compares the latest alert range image P1 with the past alert range image P1. Here, the alert range image P1 that is one cycle before the shooting period of the camera 6 is set as the past alert range image P1.

  In the example of FIG. 16, a total of 16 pixels of 4 pixels in the X direction and 4 pixels in the Y direction are set as the attention area A1. Of course, the attention area A1 may employ a number of pixels other than 16 pixels. A total of 64 pixels, with 8 pixels in the X direction and 8 pixels in the Y direction, centered on the attention area A1, is set as the search range A2. Of course, the number of pixels in the search range A2 can be arbitrarily set.

  The obstacle detection unit 42 sets the search range A2 to a predetermined position of the image data P, for example, end portions in the X direction and the Y direction. Then, the search range A2 is searched. A search method in the search range A2 will be described with reference to FIG. A comparison area A3 is set in the search range A2. The comparison area A3 has a total of 16 pixels, 4 pixels in the X direction and 4 pixels in the Y direction, like the attention area A1. Then, the comparison area A3 is set at the ends in the X direction and the Y direction.

  The obstacle detection unit 42 calculates a pixel value difference between corresponding pixels in the attention area A1 in the past warning area image P1 and the comparison area A3 in the latest warning area image P1. The attention area A1 and the comparison area A3 are both composed of four pixels in the X direction and four pixels in the Y direction, and one pixel in the attention area A1 and one pixel in the comparison area A3 have a 1: 1 correspondence. doing. Therefore, as described above, the pixel value difference calculation between corresponding pixels is performed. By performing this calculation, a difference value for 16 pixels can be obtained. Then, the calculation is performed by squaring the difference values for 16 pixels and adding the 16 squared difference values. A value obtained by performing this calculation is used as an addition value.

  The obstacle detection unit 42 determines whether or not the added value is equal to or less than a predetermined threshold value. The threshold value at this time is a value with which it can be determined whether or not the attention area A1 and the comparison area A3 can be regarded as the same. The obstacle detection unit 42 determines that the past attention area A1 and the latest comparison area A3 coincide with each other when the added value is equal to or less than a predetermined threshold value. On the other hand, in other cases, it is determined that the past attention area A1 and the latest comparison area A3 do not match.

  If the past attention area A1 and the latest comparison area A3 match and if they do not completely match within the search area A2, the attention area A1 moves within the search area A2. Can be recognized. As shown in FIG. 17, the comparison is performed by shifting the comparison area A3 by one pixel in the X direction. This is a search (scan) in the X direction. When scanning in the X direction is completed, the comparison area A3 is then shifted by one pixel in the Y direction, and scanning in the X direction is performed again. Scanning in the X direction by shifting one pixel in the Y direction is referred to as scanning in the Y direction. Therefore, scanning is performed in the X direction and the Y direction. FIG. 18 shows this scanning method three-dimensionally.

  The scanning in the X direction and the Y direction is performed until it is detected that the past attention area A1 matches the latest comparison area A3. Therefore, if it is detected that the past attention area A1 matches the latest comparison area A3, the scanning is terminated at that time. On the other hand, if the matching past attention area A1 and the latest comparison area A3 are not detected at the time when all the scans in the X direction and the Y direction are completed, a moving object (failure) is included in the search range A2. Object: It is recognized that no worker M) exists.

  The search range A2 is a limit range that can be moved when the worker M normally walks from the time when the previous alert range image P1 is acquired until the latest alert range image P1 is obtained. Determine as a reference. As described above, the past alert range image P1 is an image one cycle before the latest alert range image P1. Therefore, the search range A2 is set based on the limit amount when the worker M normally walks during one shooting period of the camera 6.

  By the way, the camera 6 has a case where the photographing cycle is fast and a case where it is slow. At this time, when the shooting period of the camera 6 is high, even if the worker M has moved, it is recognized that it has hardly moved. On the other hand, when the imaging cycle is low, it is recognized that the worker M is moving at high speed even if the worker M is moving at low speed. Therefore, when the shooting period of the camera 6 is high, the warning area image P1 that is relatively before a plurality of previous periods is set as the past warning area image P1. On the other hand, when the imaging cycle of the camera 6 is low, the most recently required alert range image P1 (for example, one cycle before) is set as the past alert range image P1.

  The obstacle detection unit 42 performs a search based on the search range A2 on all the pixels in the alert range image P1. Therefore, when the search range A2 is at the end in the X direction and the Y direction, when the search is completed, the search range A2 is then shifted in the X direction by the amount of pixels in the X direction (8 pixels), and the same is performed again. Search for. When the search is completed for all the pixels in the X direction of the alert range image P1, the search range A2 is shifted in the Y direction by the pixels in the Y direction (8 pixels), and the same search is performed again. Then, the search within the search range A2 is performed for all the pixels in the alert range image P1.

  Here, as shown in FIG. 16, the warning area image P <b> 1 is divided into a warning area 31 for detecting an obstacle and a dead area 33. The obstacle detection unit 42 recognizes the dead area 33 in the caution area image P1. That is, the pixels included in the dead area 33 are recognized. Therefore, the obstacle detection unit 42 excludes the dead area 33 when performing the search by the search range A2. Thereby, it is possible to detect the worker M as an obstacle limited to the vigilance range 31. Then, the obstacle detection unit 42 outputs information on the worker M as the detected obstacle to the display image creation unit 28.

  The display image creation unit 28 receives the image data P from the camera 6, and inputs information about the worker M as an obstacle from the obstacle detection unit 42. Therefore, the display image creation unit 28 recognizes the worker M within the alert area 31 of the image data P. The display image creation unit 28 superimposes an obstacle marking display M1 around the worker M in order to explicitly display the obstacle. Here, the obstacle marking display M1 is a circumscribed rectangle formed around the worker M, but it need not be a rectangle.

  FIG. 19 shows the image data P on which the obstacle marking display M1 is superimposed. This image data P becomes a display image and is output to the monitor 9. Then, as shown in the figure, the obstacle marking display M1 is superimposed around the worker M as an obstacle, so that the operator can see the presence of the worker M based on the obstacle marking display M1. Can be recognized. Note that, in FIG. 19, the vigilance range 31 is indicated by a virtual line, but this indicates that an obstacle has been detected only within this range. The monitor 9 basically does not display the alert range 31 but may be displayed.

  By the way, the viewpoint of the surroundings is monitored by converting the viewpoint of the camera image acquired by the camera 6 by image processing to form a bird's-eye view image, and a configuration for displaying the surroundings monitoring image on the monitor 9 has been conventionally known. Yes. That is, as shown in FIG. 29, when the camera 6 is arranged at a predetermined height position and the image is taken with the optical axis CA of the objective lens obliquely downward at an angle α, a through image P is obtained. From the through image P having this angle θ, as shown by the phantom line in the figure, an image obtained by coordinate conversion from the virtual viewpoint VF so that the optical axis VA is perpendicular to the through image P is created. As a result, it is possible to obtain a bird's-eye view image obtained by viewing the camera image from above at an angle α. (Note that the image data P is a through image, and an image obtained by converting the viewpoint of the through image data so as to be the upper viewpoint is a bird's-eye view image. (The acquired image data is referred to as a through image P.)

  In each of the embodiments described above, a camera image (through image) acquired by the camera 6 is displayed. However, it is also possible to perform signal processing from this camera image to form an overhead image and display it on the monitor 9. That is, as shown in FIG. 30A, a viewpoint conversion process is performed in which the through image P is the upper viewpoint.

  Then, as shown in FIG. 30 (b), the dump truck 1 is provided with cameras at four locations, the front position, the left and right side positions, and the rear position. The camera images are obtained, and these camera images are subjected to overhead view image processing and displayed on the monitor 9. For the front side camera, the left and right side side cameras and the rear side camera of the dump truck 1, Each overhead image in each image area SPF, SPL, SPR, SPB can be acquired and displayed on the monitor 9. FIG. 30A shows an image area SPF by the front camera. In this display, the icon image Q having the dump truck 1 in a planar shape is arranged in the center of the screen of the monitor 9, and the image areas SPF, SPL, SPR, SPB are arranged in four directions around the icon image Q, respectively. Display the corresponding overhead image.

  In this way, by superimposing the icon image Q around the icon image Q and displaying the bird's eye view of the surrounding situation, a monitoring image displayed so as to look down on the entire periphery of the dump truck 1 can be obtained. In addition, this monitoring image can display the obstacle marking display M1 indicating the worker M in the same manner as the through image P in FIG. 30A, and further, because of the overhead view image, the relative position with respect to the dump truck 1 The display mode can also be grasped.

  Next, a case where a shift occurs in the visual field range 6A of the camera 6 installed in the dump truck 1 will be described. As described above, the visual field range 6 </ b> A of the camera 6 may be displaced due to the vibration of the dump truck 1. Therefore, it is determined whether or not there is a deviation in the visual field range 6A. However, there is an allowable range for the deviation of the visual field range 6A. When the deviation amount exceeds the allowable range, it is determined that there is a deviation, and when it is within the allowable range, it is determined that there is no deviation.

  As shown in FIG. 13, information on the alert range 31 is input from the alert range setting unit 26 to the mask image generation unit 43. The mask image generation unit 43 generates a mask image based on the image data P and the vigilance range 31 (step S12). As shown in FIG. 20, in the mask image, a warning area 31 of the image data P is configured with white pixels (pixel value 256), and a warning-free area 32 is configured with black pixels (pixel value zero).

  The mask image generation unit 43 holds image data P in the past (for example, one cycle before) and information on the caution area 31, and holds a past mask image. Then, when the latest image data P and information on the alert range 31 are input, the latest mask image is generated. The mask image generation unit 43 outputs the past mask image and the latest mask image to the shift determination unit 44.

  The deviation determination unit 44 calculates a deviation amount based on the input past mask image and the latest mask image, and determines whether or not the deviation of the visual field range 6A is within an allowable range based on the calculated deviation amount. Determination is made (step S13). For this purpose, the deviation determination unit 44 calculates the pixel difference between the past mask image and the latest mask image.

  When the difference between the pixels where the vigilance range 31 overlaps between the past mask image and the latest mask image is calculated, the pixel value becomes zero. That is, the pixel becomes black. Further, when a difference between pixels in which the warning-unnecessary range 32 overlaps between the past mask image and the latest mask image is calculated, the pixel value becomes zero. That is, the pixel becomes black. On the other hand, when the difference between the pixels where the alert required range 31 and the alert unnecessary range 32 overlap is calculated, the absolute value of the calculation result is 256. That is, the pixel becomes white. The difference image P2 in FIG. 20 is an image obtained as a result of the calculation of the difference.

  When there is a shift in the visual field range 6A, white pixels are generated in the difference image P2. The number of white pixels indicates the amount of deviation. In other words, if there is no deviation in the visual field range 6A, the past mask image and the latest mask image completely coincide with each other, so that there is no pixel in which the alert range 31 and the alert unnecessary range 32 overlap. For this reason, there is no white pixel. On the other hand, if there is a shift in the visual field range 6A, there is a portion where the alert range 31 and the alert-unnecessary range 32 overlap, so that the pixel of the portion becomes a white pixel.

  A predetermined number of pixels is set as a threshold in the shift determination unit 44, and the number of white pixels in the difference image P2 is compared with the threshold. This threshold indicates an allowable range of deviation of the visual field range 6A. This allowable range can be set arbitrarily, and for example, a deviation that affects the visibility of the operator can be set as the allowable range. Then, as a result of comparing the number of white pixels and the threshold value, the shift determination unit 44 determines that there is no shift in the visual field range 6A if the number of white pixels is less than the threshold value. On the other hand, if the number of white pixels is greater than or equal to the threshold value, it is determined that there is a shift in the visual field range 6A.

  The deviation determination unit 44 outputs information on whether or not a deviation has occurred to the display image creation unit 28. The display image creation unit 28 displays the shift information G in the image data P when information indicating that a shift has occurred is input from the shift determination unit 44. FIG. 21 shows an example of the image data P on which the shift information G is superimposed. The deviation information G is displayed at the corners of the image data P, and is a mark for visually recognizing that the camera 6 is displaced.

  The deviation information G is a notification means that allows the operator to recognize that the camera 6 has a deviation that is greater than or equal to the allowable range. Based on the deviation information G that is displayed on the monitor 9, the operator Can be recognized. The operator who has recognized this can return the camera 6 to the normal position, and can eliminate the shift of the visual field range 6A. Alternatively, the operator can return the camera 6 to the normal position by contacting the maintenance staff.

  As described above, even if a deviation occurs in the visual field range 6A, the relative positional relationship between the front structure 10 and the warning area 31 is constant in the image data P, so the warning area 31 is always set correctly. Is done. FIG. 22A shows the image data P from which the edge extraction unit 23 performs the edge extraction process on the image data P and the edge pixels are extracted. And the structure edge determination part 24 extracts a straight line from the image data P using methods, such as a Hough transform. FIG. 22B shows a state in which a straight line is extracted from the image data P.

  In the first embodiment described above, since there is no shift in the visual field range 6A, the structure edge determination unit 24 detects straight lines in the X direction and the Y direction. However, in the present embodiment, since there is a tilt shift in the visual field range 6A, the front structure 10 in the image data P is also tilted. Therefore, the structure edge determination unit 24 of the present embodiment also extracts straight lines that are inclined at a predetermined angle from the X direction and the Y direction. That is, a predetermined angle is set as the inclination angle θ, and a straight line within the range of the inclination angle θ is extracted from the extracted straight lines.

  Then, the longest straight line among the straight lines inclined within the range of the inclination angle θ from the X direction is determined as the inclined straight line in the X direction of the front structure 10. Here, the straight line L5 is the longest straight line. Next, the inclination angle from the Y direction among the straight lines detected on the lower end (the end closest to the dump truck 1 in the image data P) from the straight line obtained by extending the straight line L5 to both ends of the image data P. A straight line within the range of θ is extracted. Here, straight lines L7 and L8 are extracted. Then, the straight lines L7 and L8 are extended. Thereby, as shown in FIG. 22B, four intersections C5 to C8 of the straight line L5 and the straight lines L7 and L8 are detected. These intersections C5 to C8 become edge line segment data.

  FIG. 23A shows a state in which the alert range 31 is set based on the edge line segment data. Since the inclination range is shifted in the visual field range 6A, the alert range 31 is also inclined. At the same time, the warning-free range 32 is inclined. However, the alert range 31 is all included in the image data P. Then, an image obtained by performing the distortion restoration process on the image data P in FIG. 23A becomes the image data P shown in FIG. The image data P shown in FIG. 23 (b) is output to the vigilance range image generation unit 41, and the vigilance range image as shown in FIG. 24 is generated. The warning range image is output to the obstacle detection unit 42, and the obstacle detection unit 42 detects the worker M as an obstacle.

  Therefore, even if the visual field range 6 </ b> A shifts due to the vibration of the dump truck 1, it is not necessary to set the warning range 31 again, and the correct warning range 31 is set based on the front structure 10. Thereby, even if a deviation occurs in the visual field range 6 </ b> A, the obstacle can be detected only in the alert range 31. However, the visibility of the operator is deteriorated by causing a deviation in inclination in the visual field range 6A. In such a case, the position of the camera 6 may be corrected to eliminate the tilt deviation.

C: Third Embodiment of the Invention Next, a third embodiment will be described. In the present embodiment, the vigilance range 31 is changed according to the traveling speed of the dump truck 1. The range in which the obstacle should be watched also changes depending on whether the dump truck 1 is stopped, traveling at a low speed, or traveling at a high speed. When the dump truck 1 is in a stopped state, it is not necessary to set the warning range 31 as wide as the warning range, but it is necessary to set the warning range 31 as a wide range when traveling at a certain speed.

  Therefore, as shown in FIG. 25, in this embodiment, a traveling speed detection unit 45 is added to the second embodiment. The dump truck 1 is provided with means for detecting the traveling speed, and outputs the detected traveling speed to the alert range setting unit 26. The vigilance range setting unit 26 changes the size of the vigilance range 31 based on the input information on the traveling speed.

  Here, regardless of whether the dump truck 1 is stopped or traveling at a low speed or a high speed, the relative positional relationship between the front structure 10 and the alert range 31 is constant. Therefore, the warning area 31 can be a predetermined area 31 based on the traveling speed. Specifically, as shown in FIG. 31, the size of the area requiring caution 31 is set to a caution area 311 from a stop area to a very low speed area, a caution area 312 at low speed travel, and a caution area at high speed travel It can be changed to 313. Here, the slow speed range is a speed at which the dump truck 1 stops almost at the position when the travel stop operation is performed. It should be noted that the warning range required when the dump truck 1 is in operation is a range that requires warning, not the image display range on the monitor 9.

D: Fourth Embodiment of the Invention In the first embodiment, the second embodiment, and the third embodiment, the dump truck 1 is applied as a self-propelled industrial machine. In the fourth embodiment, A hydraulic excavator is used as a self-propelled industrial machine. Here, the case where the camera 6 is installed in the hydraulic excavator, the rear structure 50 of the hydraulic excavator is extracted, and the alert range 31 is set based on the rear structure 50 will be described.

  FIG. 26 shows the image data P that has been subjected to distortion correction processing by the distortion correction unit 21. The rear structure 50 of the excavator is often rounded, and the boundary between the rear structure 50 and the background in the image data P is curved with a predetermined radius of curvature. The rear structure 50 of the excavator is colored in a color clearly different from the background color. Therefore, the luminance value of the rear structure 50 and the luminance value of the background are greatly different. Therefore, in the present embodiment, pixels having a luminance value equal to or higher than a predetermined luminance value (threshold value) are extracted for all the pixels of the image data P. In addition, the threshold value at this time is set to a luminance value that can extract the color colored in the rear structure 50.

  As a result, the image data P is binarized with the threshold as a boundary, and the pixels constituting the rear structure 50 are extracted as shown in FIG. However, in addition to the rear structure 50, there may be an edge pixel having a luminance value greater than or equal to the threshold value in the concavo-convex portion R. Therefore, the pixel area in contact with the lower end of the screen among the extracted pixels is determined as the rear structure 50. Thereby, the rear structure 50 is specified in the image data P.

  FIG. 28 shows an example of a display image displayed on the monitor 9. The relative positional relationship between the rear structure 50 and the caution area 51 is constant, and by specifying the rear structure 50 in the image data P, the caution area 51 indicated by a virtual line is also set. Then, the worker M can be detected as an obstacle limited to the area requiring caution 51. This figure shows a case where the visual field range is tilted, so that the entire screen is tilted and displayed. Then, the shift information G is superimposed on the image data P.

  In this embodiment, since the rear structure 50 is specified by binarization, each process by the smoothing processing unit 22, the edge extraction unit 23, the structure edge determination unit 24, and the edge line segment data determination unit 25 described above. Is unnecessary. However, if the rear structure 50 having a predetermined curvilinearity can be detected, the rear structure 50 may be specified by the processing of each of these parts. Further, in the embodiment described above, that is, in the dump truck 1, when the front structure 10 is specified, a binarization method may be used as in this embodiment.

  In each of the embodiments described above, the image displayed on the monitor 9 may be a camera image itself, or may be displayed as an overhead image by performing viewpoint conversion processing based on the principle of FIG.

DESCRIPTION OF SYMBOLS 1 Dump truck 6 Camera 6A Field-of-view range 8 Image processing apparatus 9 Monitor 10 Front structure 21 Distortion correction part 22 Smoothing process part 23 Edge extraction part 24 Structure edge determination part 25 Edge line segment data determination part 26 Warning range setting part 27 Distortion Restoration Unit 28 Display Image Creation Units 31, 311, 312, 313 Warning Range 32 Warning Not Required Range 33 Dead Area 41 Warning Range Required Image Generation Unit 42 Obstacle Detection Unit 43 Mask Image Generation Unit 44 Deviation Determination Unit 45 Travel Speed Detection unit G Deviation information M Worker M1 Obstacle marking display P Image data (through image)
P1 Alert range image SPF, SPL, SPR, SPB Image area

Claims (6)

A photographing unit (9) provided in the self-propelled industrial machine (1) for photographing a peripheral region including a part of the structure (10) constituting the self-propelled industrial machine (1) in the field of view;
A structure detection unit (25) for detecting the structure (10) in the image data based on the image data captured by the imaging unit (9);
Among the image data, an obstacle (M) located in the peripheral area is detected, and a predetermined range including the obstacle (M) is set as a warning area (31, 311, 312, 313). A warning range setting unit (26) that sets a warning range (31, 311, 312, 313) in the image data based on the structure (10) detected by the structure detection unit (25);
A self-propelled industrial machine image processing device equipped with The obstacle detection part (42) which performs the process which detects the said obstruction (M) only to the said vigilance range (31,311,312,313) in the said image data is provided. Image processing device for self-propelled industrial machines. In the image data, a range other than the alert range (31, 311, 312, 313) is set as an alert unnecessary range (32), and a difference between pixel values of each pixel in the alert unnecessary range (32) that is temporally changed. A shift determination that determines whether or not a shift exceeding the allowable range has occurred in the field of view of the photographing unit (9) based on the number of pixels having a pixel value difference equal to or greater than a predetermined threshold. The image processing device for a self-propelled industrial machine according to claim 2, further comprising a section (44). The self-propelled industrial machine (1) includes a travel speed detection unit (45) that detects a travel speed, and the warning range (31, 311, 311, 11) is based on the travel speed detected by the travel speed detection unit (45). The image processing device for a self-propelled industrial machine according to claim 2.   The image processing device for a self-propelled industrial machine according to claim 2, wherein the image data is displayed as an overhead image. The self-propelled industrial machine (1) is provided with a photographing unit (9), and the surrounding area including a part of the structure of the self-propelled industrial machine (1) is photographed.
Based on the image data photographed by the photographing unit (9), the structure (10) in the image data is detected,
Detecting an obstacle (M) located in a peripheral region of the image data;
A predetermined range including the obstacle (M) is set as a warning area (31, 311, 312, 313), and the structure detection unit (25) detects the area requiring warning (31, 311, 312, 313). Set in the image data based on the structure (10)
An image processing method for a self-propelled industrial machine that performs a process of detecting the obstacle (M) only for the alert range (31, 311, 312, 313) in the image data.

Images