JP6541734B2 - Shovel - Google Patents

Description

  The present invention relates to a shovel.

  DESCRIPTION OF RELATED ART The circumference | surroundings monitoring apparatus which monitors circumference | surroundings is known conventionally, acquiring the distance and direction with respect to the shovel of the object which exists around a shovel using the laser radar attached to the revolving super structure of a shovel. 1)).

  The surrounding area monitoring device determines that the object is a stationary obstacle when the distance and direction of the object to the shovel do not change for a predetermined time, and determines the object as a moving obstacle when the distance and direction change. Then, the surrounding area monitoring device generates an illustration image of the shovel and its surroundings viewed from above, and displays an illustration representing the obstacle at a position on the illustration image corresponding to the actual position of the obstacle. In addition, the surrounding area monitoring device displays the stationary obstacle and the moving obstacle with illustrations of different modes.

  In this way, the surrounding area monitoring device can communicate the status of obstacles around the shovel to the driver in an easy-to-understand manner.

JP, 2008-163719, A

  However, the laser radar described in Patent Document 1 irradiates one object with laser light and detects the distance and direction of the object from the reflected light. Therefore, the laser radar described in Patent Document 1 only detects the distance between itself and a very limited portion of the object surface, and the detection result roughly determines the position of the illustration representing the object. It is only used for Therefore, the output image presented by the surrounding area monitoring device described in Patent Document 1 is still insufficient for informing the driver of the situation of the obstacle around the shovel in an easy-to-understand manner.

  In view of the above, it would be desirable to provide a shovel that provides the driver with a clear understanding of the status of surrounding obstacles.

  In order to achieve the above object, a shovel according to an embodiment of the present invention is mounted on a lower traveling body which performs a traveling operation, an upper revolving body rotatably mounted on the lower traveling body, and the upper revolving body An excavator including a boom included in an attachment, an arm attached to the boom, an arm included in the attachment, and a sensor mounted on the upper swing body and acquiring distance information for each pixel, A camera is disposed on the upper swing body separately from a sensor that acquires distance information for each pixel.

  According to the above-described means, it is possible to provide a shovel that easily informs the driver of the situation of surrounding obstacles.

It is a block diagram showing roughly an example of composition of an image generation device concerning an example of the present invention. It is a figure which shows the structural example of the shovel by which an image generation apparatus is mounted. It is a figure which shows an example of the space model by which an input image is projected. FIG. 6 is a diagram showing an example of a relationship between a space model and a processing target image plane. It is a figure for demonstrating matching with the coordinate on an input image plane, and the coordinate on a space model. It is a figure for demonstrating the matching between the coordinates by a coordinate matching means. It is a figure for demonstrating the effect | action of a parallel line group. It is a figure for demonstrating the effect | action of an auxiliary line group. It is a flowchart which shows the flow of a process target image generation process and an output image generation process. It is an example of a display of an output picture (the 1). It is a top view (the 1) of a shovel by which an image generating device is carried. It is a figure which shows each input image of three cameras mounted in the shovel, and the output image produced | generated using those input images. It is a figure for demonstrating the image loss prevention process which prevents the loss of the object in the overlap area | region of the imaging range of each of two cameras. FIG. 13 is a contrast diagram showing the difference between the output image shown in FIG. 12 and the output image obtained by applying the image loss prevention process to the output image of FIG. 12. It is a figure which shows the input distance image of each of three stereo cameras mounted in the shovel, and the output distance image produced | generated using those input distance images. FIG. 16 is a contrast diagram showing a difference between the output distance image shown in FIG. 15 and the output distance image obtained by applying the distance image loss prevention process to the output distance image of FIG. 15. It is a figure which shows an example of the after-synthesis | combination output image obtained by synthesize | combining an output distance image with an output image. It is a figure which shows another example of the synthetic | combination output image obtained by synthesize | combining an output distance image with an output image. It is a partial right side view of a shovel carrying an image generation device. FIG. 20 is an enlarged view of a raised portion in FIG. It is a figure which shows the example of an output distance image. It is a figure which shows the example of an output image. It is a flowchart which shows the flow of a 1st pixel extraction process. It is a partial right side view of a shovel carrying an image generation device. It is an enlarged view of the shallow depression part in FIG. It is a figure which shows the example of an output distance image. It is a figure which shows the example of an output image. It is a flowchart which shows the flow of a 2nd pixel extraction process. It is a flow chart which shows a flow of driving support processing. It is a figure which shows the shovel by which an image generation apparatus is mounted. FIG. 6 is a contrast diagram showing the difference between the pre-correction output distance image and the corrected output distance image. It is a contrast diagram (the 1) showing the difference between a pre-correction post-composition output image and a corrected post-composition output image. It is a contrast diagram (the 2) showing the difference between a pre-correction post-composition output image and a corrected post-composition output image.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically showing a configuration example of an image generation apparatus 100 according to an embodiment of the present invention.

  The image generation device 100 is an example of a work machine peripheral monitoring device that monitors the periphery of a work machine, and includes a control unit 1, a camera 2, an input unit 3, a storage unit 4, a display unit 5, and a stereo camera 6. Be done. Specifically, the image generating apparatus 100 generates an output image based on the input image captured by the camera 2 mounted on the work machine and the input distance image output by the stereo camera 6, and outputs the output image to the operator. To present.

  FIG. 2 is a view showing a configuration example of a shovel 60 as a working machine on which the image generating apparatus 100 is mounted. The shovel 60 is an upper portion of a crawler type lower traveling body 61 via a turning mechanism 62. The swing body 63 is rotatably mounted around the swing axis PV.

  The upper swing body 63 also has a cab (driver's cab) 64 on the front left side thereof and an excavating attachment E at the front center thereof, and the camera 2 (right side camera 2R, rear camera 2B on its right side and rear side) And a stereo camera 6 (right side stereo camera 6R, rear stereo camera 6B). In addition, the display part 5 is installed in the position in which the operator in the cab 64 visually recognizes.

  Next, each component of the image generation apparatus 100 will be described.

  The control unit 1 is a computer including a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), a non-volatile random access memory (NVRAM), and the like. Programs corresponding to the setting unit 10, the image generation unit 11, the pixel extraction unit 12, the distance correction unit 13, the distance image combination unit 14, and the driving support unit 15 are stored in the ROM or NVRAM and the RAM as a temporary storage area Make the CPU execute the processing corresponding to each means while using.

  The camera 2 is a device for acquiring an input image that reflects the periphery of the shovel 60. For example, the camera 2 is attached to the right side surface and the rear surface of the upper swing body 63 so as to be able to capture an area which is a blind spot of the operator in the cab 64 They are a right side camera 2R and a rear camera 2B provided with an imaging device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) (see FIG. 2). The camera 2 may be attached to a position other than the right side surface and the rear surface of the upper swing body 63 (for example, the front surface and the left side surface), and a wide angle lens or a fisheye lens is attached to capture a wide range. It may be

  Further, the camera 2 acquires an input image according to a control signal from the control unit 1, and outputs the acquired input image to the control unit 1. When the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the corrected input image corrected for apparent distortion and tilt caused by using these lenses is sent to the control unit 1. An input image that is output but is not corrected for apparent distortion or alias may be output to the control unit 1 as it is. In such a case, the control unit 1 corrects the apparent distortion or tilt.

  The input unit 3 is a device for enabling an operator to input various information to the image generation device 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, or the like.

  The storage unit 4 is a device for storing various information, and is, for example, a hard disk, an optical disk, a semiconductor memory, or the like.

  The display unit 5 is a device for displaying image information, and is, for example, a liquid crystal display or a projector installed in a cab 64 (see FIG. 2) of the shovel 60. Display an image.

  The stereo camera 6 is a device for acquiring a two-dimensional array of distance information of an object present around the shovel 60. For example, the upper swing body 63 can capture an area which is a blind spot of the operator in the cab 64. Mounted on the right and back sides of the (see Figure 2). The stereo camera 6 may be attached to any one of the front, left, right, and rear surfaces of the upper swing body 63, or may be attached to all the surfaces.

  The stereo camera 6 is an apparatus provided with an imaging function of a plurality of cameras. In this embodiment, the stereo camera 6 is an apparatus integrally provided with imaging functions of two cameras, and includes two sets of imaging mechanisms (combination of an imaging element and a lens mechanism) and one shared shutter mechanism. It is provided in one case. However, the stereo camera 6 may be configured by a plurality of independent cameras.

  Further, a plurality of sets of imaging mechanisms in the stereo camera 6 are arranged at predetermined intervals. In this embodiment, the two imaging mechanisms are juxtaposed at predetermined intervals in the longitudinal direction (vertical direction). The two imaging mechanisms may be juxtaposed at predetermined intervals in the horizontal direction (horizontal direction), or at predetermined intervals in the vertical direction (vertical direction) and the horizontal direction (horizontal direction). It may be juxtaposed.

  Further, the stereo camera 6 acquires distance information for each pixel while the camera 2 acquires luminance, hue value, saturation value and the like for each pixel. In the present embodiment, the stereo camera 6 has each pixel of an image (hereinafter referred to as a “basic image”) output by one imaging mechanism based on the parallax between the images output by each of the two sets of imaging mechanisms. The distance between the object shown in and the stereo camera 6 is derived. Then, the stereo camera 6 substitutes the derived distance into the value of each pixel of the basic image to generate distance information of a two-dimensional array, and outputs the distance information to the control unit 1. Therefore, in the following, distance information of a two-dimensional array by the stereo camera 6 is compared with an input image and an output image of the camera 2 and referred to as an input distance image and an output distance image. Further, the resolution (number of pixels) of the input distance image and the output distance image of the stereo camera 6 may be the same as or different from the resolution in the input image and the output image of the camera 2. In addition, one or more pixels in the input image and the output image of the camera 2 may be associated in advance with one or more pixels in the input distance image and the output distance image of the stereo camera 6. Further, the control unit 1 may generate the input distance image based on a plurality of images output by each imaging mechanism of the stereo camera 6.

  Further, the input image planes of the plurality of imaging mechanisms constituting the stereo camera 6 are preferably parallel to each other, and most preferably form the same plane. This is because the process for deriving the distance from the parallax can be easily and accurately performed.

  Further, the stereo camera 6 may be attached to a position other than the right side surface and the rear surface of the upper swing body 63 (for example, the front surface and the left side surface) as the camera 2 and a wide-angle lens so that a wide range can be imaged. Alternatively, a fisheye lens may be attached.

  Further, like the camera 2, the stereo camera 6 acquires an input distance image according to a control signal from the control unit 1, and outputs the acquired input distance image to the control unit 1. In the case where the stereo camera 6 acquires an input distance image using a fisheye lens or a wide-angle lens as with the camera 2, a corrected input distance after correction of apparent distortion and tilt caused by using these lenses An image may be output to the control unit 1, but an input distance image not corrected for apparent distortion or tilt may be output to the control unit 1 as it is. In such a case, the control unit 1 corrects the apparent distortion or tilt.

  Further, the image generation apparatus 100 generates the processing target image based on the input image, and performs image conversion processing on the processing target image so that the positional relationship with surrounding features and the sense of distance can be intuitively grasped. After the output image to be used is generated, the output image may be presented to the operator.

  The image generation apparatus 100 performs the same process on the input distance image. In that case, the processing target image is replaced with the processing target distance image. The same applies to the following description.

  The “processing target image” is an image to be generated on the basis of an input image, which is a target of image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.). When an input image including an image in the horizontal direction (for example, an empty part) by the wide angle of view is used in image conversion processing, the image in the horizontal direction is The input image is projected to a predetermined space model so as not to be displayed unnaturally (for example, the sky part is not treated as being on the ground surface), and the projection image projected onto the space model is another two. It is an image suitable for image conversion processing, which is obtained by reprojection on a dimensional plane. Note that the processing target image may be used as an output image as it is without performing image conversion processing.

  The “spatial model” is at least a plane or a curved surface other than the processing target image plane which is a plane on which the processing target image is located (for example, a plane parallel to the processing target image plane or an angle with the processing target image plane It is a projection object of an input image which is composed of one or a plurality of planes or a curved surface, which is a plane or a curved surface to be formed.

  The image generation apparatus 100 may generate an output image by performing an image conversion process on the projection image projected on the space model without generating the processing target image. In addition, the projection image may be used as an output image as it is without performing image conversion processing.

  FIG. 3 is a view showing an example of a space model MD on which an input image is projected, and FIG. 3 (A) shows the relationship between the shovel 60 and the space model MD when the shovel 60 is viewed from the side. FIG. 3B shows the relationship between the shovel 60 and the space model MD when the shovel 60 is viewed from above.

  As shown in FIG. 3, the space model MD has a semi-cylindrical shape, and has a flat region R1 inside its bottom and a curved region R2 inside its side.

  FIG. 4 is a diagram showing an example of the relationship between the space model MD and the processing target image plane, and the processing target image plane R3 is, for example, a plane including the plane region R1 of the space model MD. Although FIG. 4 shows the space model MD not in a semi-cylindrical shape as shown in FIG. 3 but in a cylindrical shape for clarity, the space model MD has either a semi-cylindrical shape or a cylindrical shape. It may be The same applies to the following figures. Further, as described above, the processing target image plane R3 may be a circular area including the plane area R1 of the space model MD, or may be an annular area not including the plane area R1 of the space model MD.

  Next, various units included in the control unit 1 will be described.

  The coordinate associating means 10 is a means for associating the coordinates on the input image plane where the input image captured by the camera 2 is located, the coordinates on the space model MD, and the coordinates on the processing object image plane R3. For example, various parameters related to the camera 2 such as an optical center of the camera 2, a focal length, a CCD size, an optical axis direction vector, a camera horizontal direction vector, and a projection method which are preset or input through the input unit 3. The coordinates on the input image plane, the coordinates on the space model MD, and the processing object image, based on the mutual positional relationship between the input image plane, the space model MD, and the processing object image plane R3 determined in advance The coordinates on the plane R3 are associated, and the correspondence relationship is stored in the input image / space model correspondence map 40 of the storage unit 4 and the space model / process target image correspondence map 41. .

  Note that, when the coordinate association unit 10 does not generate the processing target image, the association between the coordinates on the space model MD and the coordinates on the processing target image plane R3, and the space model of the corresponding relationship and the processing target The storage in the image correspondence map 41 is omitted.

  Moreover, the coordinate matching means 10 performs the same process also to the input distance image which a stereo camera outputs. In that case, the camera, the input image plane, and the processing target image plane are read again by the stereo camera, the input distance image plane, and the processing target distance image plane. The same applies to the following description.

  The image generation unit 11 is a unit for generating an output image, and for example, coordinates of the processing target image plane R3 and the output image are located by subjecting the processing target image to scale conversion, affine conversion, or distortion conversion. Corresponding to the coordinates on the output image plane to be stored, the correspondence relationship is stored in the processing target image / output image correspondence map 42 of the storage unit 4, and the coordinate correspondence unit 10 stores the value thereof. With reference to the map 40 and the spatial model-processing target image correspondence map 41, the value of each pixel in the output image (for example, luminance value, hue value, saturation value, etc.) and the value of each pixel in the input image To generate an output image.

  In addition, the image generation unit 11 may set an optical center of a virtual camera, a focal length, a CCD size, an optical axis direction vector, a camera horizontal direction vector, a projection method, or the like which is preset or input through the input unit 3. The coordinates on the processing target image plane R3 are associated with the coordinates on the output image plane where the output image is located based on various parameters of the processing target image-output image correspondence map 42 of the storage unit 4 The value (eg, luminance value, etc.) of each pixel in the output image while referring to the input image / space model correspondence map 40 and the space model / processing target image correspondence map 41 in which the values are stored. The hue value, the saturation value, etc. are associated with the value of each pixel in the input image to generate an output image.

  The image generation unit 11 may generate the output image by changing the scale of the processing target image without using the concept of the virtual camera.

  Further, when the image generation means 11 does not generate the processing target image, the coordinates on the space model MD are associated with the coordinates on the output image plane according to the applied image conversion processing, and the input image / space model is supported. While referring to the map 40, the value of each pixel in the output image (for example, luminance value, hue value, saturation value, etc.) is associated with the value of each pixel in the input image to generate an output image. In this case, the image generation unit 11 omits the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane and the storage of the correspondence in the processing target image / output image correspondence map 42. .

  Further, the image generation unit 11 performs the same process on the input distance image or the processing target distance image. In that case, the output image plane is reinterpreted at the output distance image plane. The same applies to the following description.

  The pixel extraction unit 12 is a unit for extracting a pixel satisfying a predetermined condition from the input distance image of the stereo camera 6. For example, the pixel extraction unit 12 extracts a pixel that satisfies a predetermined condition from pixels having a pixel value representing a distance between a feature around the shovel 60 and the stereo camera 6 in the input distance image.

  Specifically, the pixel extracting unit 12 sets the distance between the plane (hereinafter, referred to as "installation surface") on which the shovel 60 is located and the stereo camera 6 (hereinafter, referred to as "installation surface distance"). Pixels having a difference of not less than a predetermined distance are extracted. The installation surface may be an inclined surface. Further, the installation surface distance is set in advance for each pixel in the input distance image of the stereo camera 6 based on the installation position and the installation angle of the stereo camera 6. Then, the pixel extracting unit 12 replaces the pixel value of the pixel not extracted with a predetermined value (for example, zero as the minimum value), and maintains the pixel value of the extracted pixel as it is, thereby removing the input distance image. to correct. The details of the pixel extraction means 12 will be described later.

  The distance correction means 13 is a means for correcting the distance information acquired by the stereo camera. For example, the distance correction unit 13 corrects the value of the pixel extracted by the pixel extraction unit 12. Specifically, the distance correction means 13 sets the pixel value representing the distance between the feature around the shovel 60 and the stereo camera 6 to a value between the reference outside the stereo camera 6 and the feature. Correct to a value representing the distance. Moreover, the distance correction means 13 may correct the value of the pixel in the input distance image output by the stereo camera 6 before the extraction by the pixel extraction means 12. The details of the distance correction means 13 will be described later.

  The distance image combining means 14 is a means for combining an image relating to a camera and an image relating to a stereo camera. For example, the distance image combining unit 14 combines the output image based on the input image of the camera 2 generated by the image generating unit 11 and the output distance image based on the input distance image of the stereo camera 6. The output distance image is an output distance image after at least one of the extraction by the pixel extraction unit 12 and the correction by the distance correction unit 13 is performed. However, the output distance image may be an output distance image before extraction by the pixel extraction unit 12 and correction by the distance correction unit 13. The details of the distance image combining means 14 will be described later.

  The driving support means 15 is a means for supporting the driving of the shovel 60. For example, when it is determined that a predetermined feature exists in the moving direction of the shovel 60, the driving support means 15 executes a function of supporting the driving of the shovel 60.

  The “predetermined feature” is, for example, a person such as a worker who may collide with the shovel 60, an obstacle that prevents the movement of the shovel 60, a cliff or depression where the shovel 60 may slide down, fall over, or be stuck. Including slopes, holes, etc.

  The function to support driving (hereinafter referred to as “driving support function”) is, for example, output of an alarm, display of a warning on the display unit 5, lighting / blinking of a warning light, deceleration of the shovel 60, traveling speed of the shovel 60 Of the vehicle, stopping of the traveling of the shovel 60, and the like.

  Specifically, the driving support means 15 notifies the display unit 5 of the existence of a cliff when it is determined that there is a cliff in the moving direction (rearward) of the shovel 60 while the shovel 60 is moving (backward) Display a warning. In addition, the moving direction of the shovel 60 is detected based on the operation content (for example, operation direction, operation amount, etc.) of the operation lever, for example.

  In addition, the driving support means 15 may decrease the moving (backward) speed of the shovel 60 as the distance between the shovel 60 and the cliff becomes shorter, and the distance between the shovel 60 and the cliff is less than a predetermined distance. The movement (backward movement) of the shovel 60 may be stopped when it has become.

  Further, the driving support means 15 determines, based on the output of the stereo camera 6, whether or not a predetermined feature exists. For example, when a feature having a depth equal to or greater than a predetermined distance from the installation surface is detected by the stereo camera 6, the driving support means 15 determines that there is a depressed portion which may cause the shovel 60 to slide down. Further, the driving support means 15 causes the shovel 60 to slide down only when at least one of the actual depression depth, opening area, width, and depth calculated based on the input distance image satisfies a predetermined condition. It may be determined that there is a feared depression. In addition, the driving support means 15 determines whether or not there is a depression which may cause the shovel 60 to slide based on the increase gradient of the pixel value along the predetermined direction (moving direction of the shovel 60) in the input distance image. You may judge.

  Alternatively, when a feature having a height equal to or greater than a predetermined distance from the installation surface is detected by the stereo camera 6, the driving support means 15 may determine that there is an obstacle that may collide with the shovel 60. . In addition, the driving support means 15 determines whether or not there is an obstacle that may collide with the shovel 60 based on the decreasing gradient of the pixel value along the predetermined direction (moving direction of the shovel 60) in the input distance image. You may judge.

  Next, an example of specific processing by the coordinate matching unit 10 and the image generation unit 11 will be described.

  The coordinate correlating means 10 can associate the coordinates on the input image plane with the coordinates on the space model using, for example, the quaternion of Hamilton.

  FIG. 5 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the space model, and the input image plane of the camera 2 is a UVW Cartesian coordinate whose origin is the optical center C of the camera 2 It is expressed as a plane in the system, and the space model is expressed as a solid plane in the XYZ orthogonal coordinate system.

  First, the coordinate associating unit 10 converts the coordinates on the space model (coordinates on the XYZ coordinate system) into coordinates on the input image plane (coordinates on the UVW coordinate system), so that the origin of the XYZ coordinate system is optically After moving parallel to the center C (the origin of the UVW coordinate system), the X axis is U axis, the Y axis is V axis, and the Z axis is-W axis ("-" indicates that the direction is opposite) This means that the XYZ coordinate system is rotated so that the UVW coordinate system is at the + W direction in front of the camera, and the XYZ coordinate system is at the -Z direction in the vertically downward direction.

  In the case where there are a plurality of cameras 2, each of the cameras 2 has a separate UVW coordinate system, so the coordinate associating unit 10 parallels the XYZ coordinate system to each of the plurality of UVW coordinate systems. It will be moved and rotated.

  The above-mentioned conversion rotates the Z-axis to coincide with the -W axis after parallelly moving the XYZ coordinate system so that the optical center C of the camera 2 becomes the origin of the XYZ coordinate system. Since this is realized by rotating so as to coincide with the axis, the coordinate correlating means 10 may combine those two rotations into one rotation operation by describing this conversion in the quaternion of Hamilton. it can.

  By the way, the rotation for making one vector A coincide with another vector B corresponds to the processing of rotating by an angle formed by the vector A and the vector B around the normal of the surface where the vector A and the vector B extend. The angle θ is given by the inner product of the vector A and the vector B, where θ is the angle

Will be represented by

  Also, the unit vector N of the normal to the surface where the vector A and the vector B extend is the outer product of the vector A and the vector B

Will be represented by

  When quaternions are i, j, k as imaginary units, respectively,

In the present embodiment, the quaternion Q has a real component as t and pure imaginary components as a, b, and c, respectively.

And the conjugated quaternion of quaternion Q is

Shall be represented by

  The quaternion Q can express a three-dimensional vector (a, b, c) with pure imaginary components a, b, c while setting the real component t to 0 (zero), and t, a, b The components c and c can represent rotational motion about an arbitrary vector.

  Further, the quaternion Q can integrate a plurality of consecutive rotation operations and can be expressed as one rotation operation. For example, any point S (sx, sy, sz) can be expressed as an arbitrary unit vector The point D (ex, ey, ez) when rotated by an angle θ with C (l, m, n) as an axis can be expressed as follows.

Here, in the present embodiment, assuming that a quaternion representing a rotation that causes the Z axis to coincide with the -W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X '. X 'is

Will be represented by

  Further, in the present embodiment, assuming that a quaternion representing a rotation that causes the line connecting point X ′ on the X axis and the origin to coincide with the U axis be Qx, “the Z axis coincides with the −W axis. , A quaternion R representing "rotation to align the X axis with the U axis",

Will be represented by

  As described above, coordinates P ′ when arbitrary coordinates P on the space model (XYZ coordinate system) are expressed by coordinates on the input image plane (UVW coordinate system) are:

Since the quaternion number R is invariant in each of the cameras 2, the coordinate associating unit 10 executes the calculation to execute the coordinate in the space model (XYZ coordinate system) thereafter. It can be converted to coordinates on the input image plane (UVW coordinate system).

  After converting the coordinates on the space model (XYZ coordinate system) to coordinates on the input image plane (UVW coordinate system), the coordinate correlating means 10 converts the optical center C of the camera 2 (coordinates on the UVW coordinate system) An incident angle α formed by a line segment CP ′ connecting an arbitrary coordinate P on the model and a coordinate P ′ represented by the UVW coordinate system and the optical axis G of the camera 2 is calculated.

  Further, the coordinate correlating means 10 is a point E parallel to the input image plane R4 (for example, the CCD surface) of the camera 2 and including the coordinate P ', and the coordinate E' with the intersection E of the plane H and the optical axis G. And the length of the line segment EP ′ is calculated. The angle of inclination φ formed by the line segment EP ′ connecting U ′ and the U ′ axis in the plane H is calculated.

  In the optical system of the camera, since the image height h is usually a function of the incident angle α and the focal length f, the coordinate correlating means 10 usually performs projection (h = ftan α), orthographic projection (h = fsin α) Image height by selecting an appropriate projection method such as stereo projection (h = 2 ftan (α / 2)), iso-stereographic projection (h = 2 f sin (α / 2)), equidistant projection (h = fα) Calculate h.

  Thereafter, the coordinate associating unit 10 decomposes the calculated image height h into U component and V component on the UV coordinate system by the argument angle φ, and has a numerical value corresponding to the pixel size per pixel of the input image plane R4. By dividing, coordinates P (P ′) on the space model MD can be associated with coordinates on the input image plane R4.

Incidentally, when the pixel size per one pixel in the U axis direction of the input image plane R4 and a _U, the pixel size per one pixel in the V axis direction of the input image plane R4 and a _V, coordinates P of the space model MD The coordinates (u, v) on the input image plane R4 corresponding to (P ′) are

Will be represented by

  In this way, the coordinate associating unit 10 associates the coordinates on the space model MD with the coordinates on one or more input image plane R4 existing for each camera, and coordinates on the space model MD, a camera identifier , And coordinates on the input image plane R4 are stored in the input image / space model correspondence map 40 in association with each other.

  Further, since the coordinate correlating means 10 calculates transformation of coordinates using quaternions, it has an advantage that gimbal lock is not generated unlike the case of calculating transformation of coordinates using Euler angles. . However, the coordinate correlating means 10 is not limited to one that calculates transformation of coordinates using a quaternion, and may calculate the transformation of coordinates using Euler angles.

  In addition, when the correspondence to the coordinates on the plurality of input image planes R4 is possible, the coordinate associating unit 10 inputs the coordinates P (P ′) on the space model MD with respect to the camera having the smallest incident angle α. It may be made to correspond to the coordinates on the image plane R4, or may be made to correspond to the coordinates on the input image plane R4 selected by the operator.

  Next, among the coordinates on the space model MD, a process of re-projecting coordinates on the curved surface region R2 (coordinates having components in the Z-axis direction) on the processing target image plane R3 on the XY plane will be described.

  FIG. 6 is a diagram for explaining the correspondence between the coordinates by the coordinate correspondence means 10. FIG. 6 (A) is an input image plane R4 of the camera 2 adopting the normal projection (h = ftan α) as an example. FIG. 8 is a diagram showing the correspondence between the upper coordinates and the coordinates on the space model MD, and the coordinate correlating means 10 is a coordinate on the input image plane R4 of the camera 2 and the space model MD corresponding to the coordinates Each of the line segments connecting with the coordinates of is made to pass through the optical center C of the camera 2 and the coordinates are associated with each other.

  In the example of FIG. 6A, the coordinate associating unit 10 associates the coordinate K1 on the input image plane R4 of the camera 2 with the coordinate L1 on the plane region R1 of the space model MD, and the input image plane R4 of the camera 2 The upper coordinate K2 is associated with the coordinate L2 on the curved surface area R2 of the space model MD. At this time, the line segment K <b> 1-L <b> 1 and the line segment K <b> 2-L <b> 2 both pass the optical center C of the camera 2.

  When the camera 2 adopts a projection method other than the normal projection (for example, orthographic projection, stereographic projection, equi-solid-angle projection, equidistant projection, etc.), the coordinate correlating means 10 sets the respective projection methods. Accordingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are made to correspond to the coordinates L1 and L2 on the space model MD.

  Specifically, the coordinate correlating means 10 is configured to set a predetermined function (for example, orthogonal projection (h = fsin α), solid projection (h = 2 ftan (α / 2)), isosolid angle projection (h = 2 f sin (α /) 2) The coordinates on the input image plane are associated with the coordinates on the space model MD based on the equidistant projection (h = fα) and the like. In this case, the line segment K1-L1 and the line segment K2-L2 do not pass through the optical center C of the camera 2.

  FIG. 6B is a diagram showing the correspondence between the coordinates on the curved surface region R2 of the space model MD and the coordinates on the processing object image plane R3, and the coordinate correlating means 10 is positioned on the XZ plane. Parallel line group PL that forms an angle β with the processing target image plane R3 is introduced, and the processing target image corresponding to the coordinates on the curved surface region R2 of the space model MD and the coordinates Both coordinates correspond to each other so that the coordinates on the plane R3 both lie on one of the parallel line group PL.

  In the example of FIG. 6B, the coordinate correlating means 10 sets both coordinates on the assumption that the coordinate L2 on the curved surface area R2 of the space model MD and the coordinate M2 on the processing object image plane R3 lie on a common parallel line. Make it correspond.

  The coordinate associating means 10 can associate the coordinates on the plane region R1 of the space model MD with the coordinates on the processing object image plane R3 using the parallel line group PL in the same manner as the coordinates on the curved region R2. In the example of FIG. 6B, since the plane region R1 and the processing target image plane R3 are common in the example of FIG. 6B, the coordinate L1 on the plane region R1 of the space model MD and the processing target image plane R3 The coordinate value of the coordinate M1 is the same as that of the coordinate M1.

  Thus, the coordinate associating unit 10 associates the coordinates on the space model MD with the coordinates on the processing object image plane R3, and associates the coordinates on the space model MD and the coordinates on the processing object image plane R3. Are stored in the space model / processing target image correspondence map 41.

  FIG. 6C is a diagram showing the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V adopting the normal projection (h = ftan α) as an example. The image generation unit 11 causes each of the line segments connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates to pass through the optical center CV of the virtual camera 2V. In this way, associate the two coordinates.

  In the example of FIG. 6C, the image generation unit 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates M1 on the processing target image plane R3 (the plane region R1 of the space model MD). The coordinate N2 on the output image plane R5 of the virtual camera 2V is associated with the coordinate M2 on the processing object image plane R3. At this time, the line segment M1-N1 and the line segment M2-N2 both pass the optical center CV of the virtual camera 2V.

  When the virtual camera 2V adopts a projection method other than the normal projection (for example, orthographic projection, stereographic projection, iso-stereographic projection, equidistant projection, etc.), the image generation unit 11 selects one of the projection systems. Accordingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are made to correspond to the coordinates M1 and M2 on the processing target image plane R3.

  Specifically, the image generation unit 11 generates a predetermined function (for example, an orthogonal projection (h = fsin α), a solid projection (h = 2 ftan (α / 2)), an isostatic projection (h = 2 f sin (α / 2) The coordinates on the output image plane R5 are associated with the coordinates on the processing object image plane R3 on the basis of equidistant projection (h = fα) etc.)). In this case, the line segment M1-N1 and the line segment M2-N2 do not pass through the optical center CV of the virtual camera 2V.

  Thus, the image generation unit 11 associates the coordinates on the output image plane R5 with the coordinates on the processing object image plane R3, and coordinates on the output image plane R5 and the coordinates on the processing object image plane R3. Each of the output images is stored with reference to the input image / space model correspondence map 40 and the space model / process target image correspondence map 41 stored in the processing object image / output image correspondence map 42 in association with each other and stored An output image is generated by associating the pixel value with the value of each pixel in the input image.

  6D is a diagram combining FIGS. 6A to 6C, and shows the camera 2, the virtual camera 2V, the flat region R1 and the curved region R2 of the space model MD, and the processing target The positional relationship between the image planes R3 is shown.

  Next, with reference to FIG. 7, an operation of the parallel line group PL introduced by the coordinate associating unit 10 in order to associate the coordinates on the space model MD and the coordinates on the processing target image plane R3 will be described.

  FIG. 7A is a diagram in the case where an angle β is formed between the parallel line group PL located on the XZ plane and the processing object image plane R3, and FIG. 7B is a diagram on the XZ plane. It is a figure in the case where angle beta 1 (beta 1> beta) is formed between parallel line group PL and processing object image plane R3 which are located. Further, each of the coordinates La to Ld on the curved surface region R2 of the space model MD in FIGS. 7A and 7B corresponds to each of the coordinates Ma to Md on the processing target image plane R3, The space | interval of each of the coordinates La-Ld in FIG. 7 (A) shall be equal to each space | interval of the coordinates La-Ld in FIG. 7 (B). Although the parallel line group PL is present on the XZ plane for the purpose of explanation, in reality, it is present so as to radially extend from all points on the Z axis toward the processing object image plane R3. It shall be. The Z axis in this case is referred to as a "reprojection axis".

  As shown in FIGS. 7A and 7B, the distance between the parallel line group PL and the processing target image plane R3 is equal to the distance between the coordinates Ma to Md on the processing target image plane R3. It decreases linearly as it increases (the distance decreases uniformly regardless of the distance between the curved region R2 of the space model MD and each of the coordinates Ma to Md). On the other hand, the coordinate group on the plane region R1 of the space model MD is not converted to the coordinate group on the processing object image plane R3 in the example of FIG. 7, so the interval between the coordinate groups does not change. .

  Among the image portions on the output image plane R5 (see FIG. 6), the change in the distance between these coordinate groups is linearly expanded or only the image portion corresponding to the image projected on the curved region R2 of the space model MD. It means to be reduced.

  Next, with reference to FIG. 8, an alternative example of the parallel line group PL introduced by the coordinate associating unit 10 in order to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described.

  FIG. 8A is a diagram in the case where all the auxiliary line groups AL located on the XZ plane extend from the start point T1 on the Z axis toward the processing object image plane R3, and FIG. It is a figure in the case where all the line groups AL extend toward the process target image plane R3 from the start point T2 (T2> T1) on the Z-axis. Further, each of the coordinates La to Ld on the curved surface region R2 of the space model MD in FIGS. 8A and 8B corresponds to each of the coordinates Ma to Md on the processing target image plane R3 In the example of FIG. 8A, the coordinates Mc and Md are not shown because they are out of the region of the processing target image plane R3), and the intervals of the coordinates La to Ld in FIG. It is assumed that they are equal to the intervals of coordinates La to Ld in 8 (B). Although the auxiliary line group AL is present on the XZ plane for the purpose of explanation, in reality, the auxiliary line group AL is present so as to radially extend from any one point on the Z axis toward the processing object image plane R3. It shall be. As in FIG. 7, the Z axis in this case is referred to as “re-projection axis”.

  As shown in FIGS. 8A and 8B, the distance between coordinates Ma to Md on the processing target image plane R3 is the distance between the start point of the auxiliary line group AL and the origin O (high Decreases nonlinearly (the larger the distance between the curved surface region R2 of the space model MD and each of the coordinates Ma to Md, the larger the reduction width of each interval). On the other hand, the coordinate group on the plane region R1 of the space model MD is not converted to the coordinate group on the processing target image plane R3 in the example of FIG. 8, so the interval between the coordinate groups does not change. .

  The change in the distance between these coordinate groups corresponds to the image projected on the curved region R2 of the space model MD in the image portion on the output image plane R5 (see FIG. 6), as in the parallel line group PL. It means that only the image part is expanded or reduced non-linearly.

  In this manner, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the plane region R1 of the space model MD, and the space model MD is generated. Since the image portion (for example, a horizontal image) of the output image corresponding to the image projected on the curved surface area R2 of the road surface area R2 can be expanded or reduced linearly or nonlinearly, a road surface image near the shovel 60 The feature located in the periphery of the shovel 60 (feature in the image when the surroundings are viewed in the horizontal direction from the shovel 60) quickly without affecting the (virtual image when the shovel 60 is viewed from directly above) And it can be expanded or reduced flexibly, and the visibility of the blind spot area of the shovel 60 can be improved.

  Next, referring to FIG. 9, a process in which the image generation apparatus 100 generates a processing target image (hereinafter referred to as “processing target image generation processing”), and an output image using the generated processing target image. A process of generating (hereinafter, referred to as “output image generation process”) will be described. FIG. 9 is a flowchart showing the flow of processing target image generation processing (steps S1 to S3) and output image generation processing (steps S4 to S6). The arrangement of the camera 2 (input image plane R4), the space model (plane area R1 and curved area R2), and the processing target image plane R3 are determined in advance.

  First, the control unit 1 causes the coordinate associating unit 10 to associate the coordinates on the processing target image plane R3 with the coordinates on the space model MD (step S1).

  Specifically, the coordinate associating unit 10 acquires an angle formed between the parallel line group PL and the processing target image plane R3, and the parallel line group PL extending from one coordinate on the processing target image plane R3. A point at which one intersects the curved surface area R2 of the space model MD is calculated, and a coordinate on the curved surface area R2 corresponding to the calculated point is one on the curved surface area R2 corresponding to the one coordinate on the processing target image plane R3. The coordinates are derived, and the correspondence relationship is stored in the spatial model / processing target image correspondence map 41. The angle formed between the parallel line group PL and the processing target image plane R3 may be a value stored in advance in the storage unit 4 or the like, and the operator dynamically uses the input unit 3 via the input unit 3. It may be a value to be input.

  Further, when one coordinate on the processing target image plane R3 coincides with one coordinate on the plane region R1 of the space model MD, the coordinate associating unit 10 processes the one coordinate on the plane region R1 as the processing target image. It is derived as one coordinate corresponding to the one coordinate on the plane R 3, and the correspondence is stored in the spatial model-processing target image correspondence map 41.

  After that, the control unit 1 causes the coordinate associating unit 10 to associate one coordinate on the space model MD derived by the above-described processing with the coordinate on the input image plane R4 (step S2).

  Specifically, the coordinate correlating means 10 acquires the coordinates of the optical center C of the camera 2 adopting normal projection (h = ftan α), and is a line segment extending from one coordinate on the space model MD. A point at which the line segment passing through C intersects the input image plane R4 is calculated, and the coordinates on the input image plane R4 corresponding to the calculated point are on the input image plane R4 corresponding to the one coordinate on the space model MD. The correspondence relationship is stored in the input image / space model correspondence map 40.

  After that, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 are associated with the coordinates on the space model MD and the coordinates on the input image plane R4 (step S3). Are determined not to be associated with each other (NO in step S3), the processing in step S1 and step S2 is repeated.

  On the other hand, when it is determined that all the coordinates are associated (YES in step S3), the control unit 1 ends the processing target image generation processing and then starts the output image generation processing, and the image generation unit 11 Then, the coordinates on the processing target image plane R3 are associated with the coordinates on the output image plane R5 (step S4).

  Specifically, the image generation unit 11 generates an output image by performing scale conversion, affine conversion, or distortion conversion on the processing target image, and is determined by the content of the performed scale conversion, affine conversion, or distortion conversion. The correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 is stored in the processing target image / output image correspondence map 42.

  Alternatively, when generating an output image using the virtual camera 2V, the image generation unit 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the adopted projection method, The correspondence relationship may be stored in the processing target image / output image correspondence map 42.

  Alternatively, in the case of generating an output image using the virtual camera 2V adopting normal projection (h = ftan α), the image generation unit 11 acquires the coordinates of the optical center CV of the virtual camera 2V and then outputs the image. A line segment extending from one coordinate on the image plane R5, and a point at which the line segment passing through the optical center CV intersects the processing target image plane R3, coordinates on the processing target image plane R3 corresponding to the calculated point May be derived as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5, and the correspondence may be stored in the processing target image / output image correspondence map 42.

  Thereafter, the control unit 1 causes the image generation unit 11 to refer to the input image / spatial model correspondence map 40, the space model / process target image correspondence map 41, and the process target image / output image correspondence map 42, and input image plane R4. Correspondence between upper coordinates and coordinates on space model MD, correspondence between coordinates on space model MD and coordinates on processing object image plane R3, and coordinates on processing object image plane R3 and on output image plane R5 To obtain the values (for example, luminance value, hue value, saturation value, etc.) possessed by the coordinates on the input image plane R4 corresponding to the respective coordinates on the output image plane R5. The acquired value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). When a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the image generation unit 11 generates each of the plurality of coordinates on the plurality of input image planes R4. A statistical value (for example, an average value, a maximum value, a minimum value, an intermediate value, etc.) based on a value may be derived, and the statistical value may be adopted as the value of one coordinate on the output image plane R5. .

  Thereafter, the control unit 1 determines whether the values of all the coordinates on the output image plane R5 are associated with the values of the coordinates on the input image plane R4 (step S6), and the values of all the coordinates still correspond If it is determined that it is not attached (NO in step S6), the processing in step S4 and step S5 is repeated.

  On the other hand, when it is determined that the values of all the coordinates are associated (YES in step S6), the control unit 1 generates an output image and terminates this series of processing.

  When the image generation apparatus 100 does not generate the processing target image, the processing target image generation processing is omitted, and “coordinates on the processing target image plane” in step S4 in the output image generation processing is “on the space model”. "Coordinate" shall be read.

  With the above configuration, the image generation device 100 can generate the processing target image and the output image that can make the operator intuitively grasp the positional relationship between the ground object around the shovel 60 and the shovel 60. .

  In addition, the image generation apparatus 100 associates each coordinate on the processing target image plane R3 with the input image plane R3 by executing coordinate association so as to trace back to the input image plane R4 from the processing target image plane R3 through the space model MD. In this case, it is possible to correspond to one or more coordinates on R4 with certainty, and to execute the matching of the coordinates in the order from the input image plane R4 through the space model MD to the processing target image plane R3 (in this case Can reliably make each coordinate on the input image plane R4 correspond to one or more coordinates on the processing object image plane R3, but a part of the coordinates on the processing object image plane R3 corresponds to the input image plane R4. In some cases, it may not be associated with any of the above coordinates, in which case it is necessary to perform interpolation processing etc. on a part of the coordinates on the processing target image plane R3), a better processing target image It is possible to rapidly generate.

  Further, in the case of enlarging or reducing only the image corresponding to the curved surface region R2 of the space model MD, the image generation device 100 changes the angle formed between the parallel line group PL and the processing target image plane R3. Desired enlargement or reduction can be realized without rewriting the content of the input image / space model correspondence map 40 only by rewriting only the part related to the curved region R2 in the space model / processing target image correspondence map 41. .

  Further, when changing the appearance of the output image, the image generation apparatus 100 only changes the values of various parameters related to scale conversion, affine conversion, or distortion conversion to rewrite the processing target image / output image correspondence map 42. A desired output image (scale conversion image, affine conversion image or distortion conversion image) can be generated without rewriting the contents of the input image / space model correspondence map 40 and the space model / processing target image correspondence map 41.

  Similarly, when changing the viewpoint of the output image, the image generation apparatus 100 changes the values of various parameters of the virtual camera 2 V and rewrites the processing object image / output image correspondence map 42 to change the input image / space. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the space model / process target image correspondence map 41.

  FIG. 10 is a display example when an output image generated using input images of two cameras 2 (right-side camera 2R and rear camera 2B) mounted on the shovel 60 is displayed on the display unit 5 .

  The image generation apparatus 100 projects the input image of each of the two cameras 2 onto the plane region R1 and the curved region R2 of the space model MD, and reprojects the image on the processing target image plane R3 to generate the processing target image. An output image is generated by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image, and the vicinity of the shovel 60 is viewed from above An image (an image in the plane region R1) looking down and an image (an image in the processing target image plane R3) viewed from the periphery in the horizontal direction from the shovel 60 are simultaneously displayed.

  The output image is generated by performing image conversion processing (for example, viewpoint conversion processing) on the image projected on the space model MD when the image generation apparatus 100 does not generate the processing target image. It shall be.

  Further, the output image is trimmed in a circular shape so that the image when the shovel 60 performs a turning operation can be displayed without discomfort, and the center CTR of the circle is on the cylinder center axis of the space model MD and the turning of the shovel 60 It is generated so as to be on the axis PV, and displayed so as to rotate about its center CTR in response to the turning operation of the shovel 60. In this case, the cylinder central axis of the space model MD may or may not coincide with the reprojection axis.

  The radius of the space model MD is, for example, 5 meters, and the angle formed by the parallel line group PL with the processing target image plane R3 or the starting point height of the auxiliary line group AL is the turning of the shovel 60. If a feature (for example, a worker) is present at a position away from the center by the maximum reach distance of the drilling attachment E (for example, 12 meters), the feature is sufficiently large (for example, on the display 5) , 7 mm or more) can be set to be displayed.

  Furthermore, the output image may be arranged such that the CG image of the shovel 60 is such that the front of the shovel 60 coincides with the upper side of the screen of the display unit 5 and the turning center coincides with the center CTR. This is to make it easier to understand the positional relationship between the shovel 60 and the features appearing in the output image. Note that the output image may have a frame image including various information such as an orientation disposed around it.

  Next, details of an output image generated by the image generation device 100 will be described with reference to FIGS. 11 to 18.

  FIG. 11 is a top view of a shovel 60 on which the image generating apparatus 100 is mounted. In the embodiment shown in FIG. 11, the shovel 60 has three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) and three stereo cameras 6 (left side stereo camera 6L, right side stereo) And a rear stereo camera 6B). Regions CL, CR, and CB indicated by alternate long and short dash lines in FIG. 11 indicate imaging ranges of the left side camera 2L, the right side camera 2R, and the rear camera 2B, respectively. Further, regions ZL, ZR, and ZB indicated by dotted lines in FIG. 11 indicate imaging ranges of the left side stereo camera 6L, the right side stereo camera 6R, and the rear stereo camera 6B, respectively.

  In this embodiment, although the imaging range of the stereo camera 6 is narrower than the imaging range of the camera 2, the imaging range of the stereo camera 6 may be the same as the imaging range of the camera 2, and is wider than the imaging range of the camera 2. It is also good. Moreover, although the imaging range of the stereo camera 6 is located in the vicinity of the shovel 60 in the imaging range of the camera 2, you may be in the area | region farther from the shovel 60. FIG.

  FIG. 12 shows an input image of each of the three cameras 2 mounted on the shovel 60 and an output image generated using the input images.

  The image generation apparatus 100 projects the input image of each of the three cameras 2 onto the plane region R1 and the curved region R2 of the space model MD, and reprojects the image on the processing target image plane R3 to generate the processing target image. . Further, the image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, and the like) on the generated processing target image. As a result, the image generating apparatus 100 displays an image (image in the flat region R1) looking down near the shovel 60 from above and an image (image in the processing target image plane R3) looking horizontally from the shovel 60 Display at the same time. The image displayed at the center of the output image is a CG image 60 CG of the shovel 60.

  In FIG. 12, the input image of the right side camera 2R and the input image of the rear camera 2B respectively capture a person in the overlapping area of the imaging range of the right side camera 2R and the imaging range of the rear camera 2B (right side A region R10 surrounded by a two-dot chain line in the input image of the two-way camera 2R and a region R11 surrounded by a two-dot chain line in the input image of the rear camera 2B.

  However, assuming that the coordinates on the output image plane correspond to the coordinates on the input image plane related to the camera with the smallest incident angle, the output image causes the person in the overlapping area to disappear (a point in the output image See the region R12 enclosed by a dashed line).

  Therefore, in the output image portion corresponding to the overlapping area, the image generating apparatus 100 associates the area to which the coordinates on the input image plane of the rear camera 2B are associated with the coordinates on the input image plane of the right side camera 2R. The area is mixed, and the object in the overlapping area is prevented from disappearing.

  FIG. 13 is a diagram for explaining an image loss prevention process for preventing the loss of an object in the overlapping area of the imaging ranges of the two cameras.

  FIG. 13A is a diagram showing an output image portion corresponding to an overlapping area of the imaging range of the right side camera 2R and the imaging range of the rear camera 2B, and corresponds to a rectangular area R13 shown by a dotted line in FIG.

  Further, in FIG. 13A, an area PR1 filled with gray is an image area in which the input image portion of the rear camera 2B is disposed, and the rear camera is located at each coordinate on the output image plane corresponding to the area PR1. Coordinates on the 2B input image plane are associated.

  On the other hand, the area PR2 filled with white is an image area where the input image portion of the right side camera 2R is arranged, and each coordinate on the output image plane corresponding to the area PR2 is the input image plane of the right side camera 2R The upper coordinates are associated.

  In the present embodiment, the area PR1 and the area PR2 are arranged to form a stripe pattern, and the boundary between the area PR1 and the area PR2 alternately arranged in a stripe is centered on the turning center of the shovel 60. It is defined by concentric circles on the horizontal plane.

  FIG. 13B is a top view showing the situation of the space area in the diagonally right rear of the shovel 60, and shows the current situation of the space area imaged by both the rear camera 2B and the right side camera 2R. Moreover, FIG. 13 (B) shows that rod-shaped three-dimensional object OB exists in the diagonally right back of the shovel 60. As shown in FIG.

  FIG. 13C shows a part of an output image generated based on an input image obtained by actually imaging the space area shown in FIG. 13B with the rear camera 2B and the right side camera 2R.

  Specifically, in the image OB1, the image of the solid object OB in the input image of the rear camera 2B is extended in the extension direction of the line connecting the rear camera 2B and the solid object OB by viewpoint conversion for generating a road surface image Represents what was That is, the image OB1 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the rear camera 2B.

  In the image OB2, the image of the solid object OB in the input image of the right side camera 2R is expanded in the extension direction of the line connecting the right side camera 2R and the solid object OB by viewpoint conversion for generating a road surface image. Represent the That is, the image OB2 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the right side camera 2R.

  As described above, in the overlapping area, the image generating apparatus 100 combines the area PR1 with which the coordinates on the input image plane of the rear camera 2B are associated and the area PR2 with which the coordinates on the input image plane of the right side camera 2R are associated. Mix As a result, the image generation apparatus 100 displays both the two images OB1 and OB2 of one solid object OB on the output image, and prevents the solid object OB from disappearing from the output image.

  FIG. 14 is a contrast diagram showing the difference between the output image of FIG. 12 and the output image obtained by applying the image loss prevention process to the output image of FIG. 12, and FIG. 14 (A) is an output of FIG. An image is shown and FIG. 14 (B) shows the output image after applying the image loss prevention process. While a person disappears in a region R12 surrounded by an alternate long and short dash line in FIG. 14A, a person is displayed without disappearing in a region R14 enclosed by an alternate long and short dash line in FIG.

  Next, an output distance image generated by the image generation apparatus 100 will be described with reference to FIG. FIG. 15 is a view showing input distance images of the three stereo cameras 6 mounted on the shovel 60 and an output distance image generated using the input distance images.

  The image generation apparatus 100 projects the input distance images of the three stereo cameras 6 on the plane region R1 and the curved region R2 of the space model MD, and reprojects them on the processing object distance image plane R3 to be processed. Generate a distance image. Further, the image generation apparatus 100 generates an output distance image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, and the like) on the generated processing target distance image. Then, the image generating apparatus 100 is a distance image (distance image in the flat region R1) looking down near the shovel 60 from above and a distance image (image in the processing target distance image plane R3) looking horizontally around from the shovel 60 And) simultaneously. The image displayed at the center of the output distance image is a CG image 60 CG of the shovel 60.

  The input distance image and the output distance image in FIG. 15 are displayed as whiter (lighter) as the pixel value (distance from the stereo camera 6) is smaller and as blacker (darker) as the pixel value is larger. Note that the pixel values (distances) of the portions corresponding to the road surface and the upper revolving superstructure 63 are set to the maximum value (infinity) in order to enhance the visibility in the output distance image of the features present around the shovel 60 Be done. That is, portions corresponding to the road surface and the upper swing body 63 are represented in black.

  In FIG. 15, the input distance image of the right side stereo camera 6R and the input distance image of the rear stereo camera 6B fall within the overlapping region of the imaging range of the right side stereo camera 6R and the imaging range of the rear stereo camera 6B. (Refer to a region R15 surrounded by a two-dot chain line in the input distance image of the right side stereo camera 6R and a region R16 surrounded by a two-dot chain line in the input distance image of the rear stereo camera 6B).

  However, assuming that the coordinates on the output distance image plane correspond to the coordinates on the input distance image plane of the stereo camera at the closest position, the output distance image causes the person in the overlapping area to disappear ( Refer to a region R17 surrounded by an alternate long and short dash line in the output distance image.

  Therefore, in the output distance image portion corresponding to the overlapping area, the image generating apparatus 100 includes, among the coordinates on the input distance image plane of the rear stereo camera 6B and the coordinates on the input distance image plane of the right side stereo camera 6R, The coordinates of the smaller pixel value (distance) are made to correspond to the coordinates on the output distance image plane. As a result, the image generation apparatus 100 displays two distance images related to one solid object on the output distance image, and prevents the solid object from disappearing from the output distance image. In the following, this process for preventing the disappearance of an object in the overlapping area of the imaging ranges of the two stereo cameras is referred to as distance image loss prevention process.

  FIG. 16 is a contrast diagram showing the difference between the output distance image of FIG. 15 and the output distance image obtained by applying the distance image loss prevention process to the output distance image of FIG. The output image of FIG. 15 is shown, and FIG. 16 (B) shows the output distance image after applying the distance image loss prevention process. While the person disappears in the region R17 surrounded by the alternate long and short dash line in FIG. 16A, the person is displayed without disappearing in the region R17 surrounded by the alternate long and short dash line in FIG.

  Next, with reference to FIG. 17 and FIG. 18, a process (hereinafter, referred to as “distance image combining process”) in which the image generating apparatus 100 combines the output image with the output image will be described.

  FIG. 17 is a diagram showing an example of the combined output image obtained by combining the output distance image with the output image.

  The distance image combining means 14 of the image generation apparatus 100, for example, as shown in FIG. 16B, measures the pixel value (distance) of the extracted pixel extracted by the pixel extracting means 12 as the luminance value, hue value, saturation value, etc. The extracted pixels are superimposed and displayed on the output image shown in FIG. The extraction pixels displayed in a superimposed manner have, for example, a color according to the distance from the stereo camera 6, and as the distance becomes larger, the colors such as red, yellow, green, and blue gradually or steplessly. Change. The extracted pixels to be superimposed and displayed may have luminance according to the distance from the stereo camera 6, and the luminance may be reduced stepwise or steplessly as the distance increases. Regions EX1 to EX5 in FIG. 17 are regions formed by the extracted pixels, and are hereinafter referred to as feature regions.

  Further, the distance image combining unit 14 may superimpose and display the distance information of the feature regions EX1 to EX5 formed by the predetermined number or more of adjacent extraction pixels on the output image. Specifically, for example, the distance image combining unit 14 determines the minimum value, the maximum value, the average value, the intermediate value, and the like of the pixel values (distances) of the pixels forming the feature region EX1 to the feature region EX1. Is superimposed on the output image as a representative value representing The distance image combining unit 14 may superimpose and display the representative value on the output image only when the representative value is less than the predetermined value. This is to make it easier to recognize features relatively close to the shovel 60, that is, features relatively likely to be touched.

  FIG. 18 is a diagram showing another example of the combined output image obtained by combining the output distance image with the output image.

  In FIG. 18, the distance image combining unit 14 superimposes and displays the frames RF1 to RF5 respectively corresponding to the feature regions EX1 to EX5 of FIG. The line type of the frame is arbitrary including a dotted line, a solid line, a broken line, an alternate long and short dash line and the like, and the shape of the frame is also arbitrary including a rectangle, a circle, an ellipse and a polygon.

  Further, the distance image combining unit 14 may superimpose and display the distance information corresponding to each of the frames RF1 to RF5 on the output image. Specifically, the distance image combining unit 14 may, for example, set the frame RF1 to the minimum value, the maximum value, the average value, the intermediate value, etc. of the pixel values (distances) of the pixels constituting the feature region EX1 corresponding to the frame RF1. Is displayed superimposed on the output image as a representative value of the distance corresponding to. The distance image combining unit 14 may superimpose and display the representative value on the output image only when the representative value is less than the predetermined value. This is to make it easier to recognize features relatively close to the shovel 60, that is, features relatively likely to be touched.

  In addition, the distance image combining unit 14 may change the color of the frame according to the corresponding distance information. Specifically, the distance image combining means 14 changes the color of the frame stepwise or steplessly as red, yellow, green and blue as the distance increases. The distance image combining unit 14 may decrease the luminance of the frame stepwise or steplessly as the distance increases.

  By looking at the combined output image as described above, the operator of the shovel 60 can more easily recognize the position of the feature existing around the shovel 60 and the distance to the feature. In addition, the operator of the shovel 60 can more easily recognize the position of the feature present in the area which is blind to the operator and the distance to the feature.

  With the above configuration, the image generation apparatus 100 combines the output distance image generated based on the input distance image captured by the stereo camera 6 on the output image generated based on the input image captured by the camera 2 . Therefore, the image generating apparatus 100 can measure the distance to each feature present in a relatively wide range around the shovel 60 with relatively high resolution. As a result, the image generation apparatus 100 can generate a highly reliable combined output image.

  Further, the camera 2 and the stereo camera 6 are arranged such that their imaging ranges surround the periphery of the shovel 60. Therefore, the image generation device 100 can generate a combined output image that represents a situation when the surroundings of the shovel 60 are viewed from the sky. As a result, the operator of the shovel 60 can more easily recognize the position of the feature existing around the shovel 60 and the distance to the feature.

  Further, the image generation apparatus 100 performs the same image conversion process as the image conversion process performed on the process target image on the process target distance image. Therefore, the image generating apparatus 100 can easily associate the coordinates on the output image plane with the coordinates on the output distance image plane.

  Further, the image generation device 100 changes the color of the extracted pixel or frame stepwise or steplessly according to the distance from the stereo camera 6, that is, according to the distance from the shovel 60. Therefore, the operator of the shovel 60 can sensuously determine the presence or absence of the danger of contact.

  In addition, the image generating apparatus 100 acquires distance information of a two-dimensional array using the stereo camera 6. Therefore, the image generating apparatus 100 uses a distance image sensor configured of an LED for light emission and a CCD for light reception as compared with the case of acquiring distance information of a two-dimensional array, a heat source, a light source, a light absorbing material, etc. Less susceptible to

  Next, with reference to FIG. 19 to FIG. 23, the image generating apparatus 100 extracts pixels satisfying a predetermined condition from the input distance image and corrects the input distance image (hereinafter referred to as “first pixel extraction processing” Explain). In the present embodiment, the value of the pixel in the input distance image captured by the rear stereo camera 6B is equal to or smaller than the installation surface distance.

  FIG. 19 is a partial right side view of the shovel 60 on which the image generating apparatus 100 is mounted. In the embodiment shown in FIG. 19, the shovel 60 includes one camera 2 (rear camera 2B) and one stereo camera 6 (rear stereo camera 6B). An area ZB indicated by a dotted line in FIG. 19 indicates an imaging range of the rear stereo camera 6B.

  FIG. 19 also shows that behind the shovel 60, an object P3 extending vertically upward from the ground and a raised portion P4 of the ground are present. Further, FIG. 19 shows that the distance E1a to the object P3 is detected by the rear stereo camera 6B, and the distance E2a to the projection P4 is detected.

  FIG. 19 also shows that the rear stereo camera 6B has detected the installation surface distances E1 and E2 instead of the distances E1a and E2a if the object P3 and the raised portion P4 do not exist. .

  Further, in FIG. 19, a difference (E1-E1a) between the installation surface distance E1 and the distance E1a is a distance (hereinafter, a difference when the installation surface distance E1 is larger than the distance E1a is referred to as a "first difference distance" E1b. This indicates that the difference (E2-E2a) between the installation surface distance E2 and the distance E2a is the first difference distance E2b.

  FIG. 20 is an enlarged view of the protruding portion P4 in FIG. As shown in FIGS. 19 and 20, a line connecting a point on the ridge P4 viewed from the rear stereo camera 6B and the rear stereo camera 6B is an angle between the installation surface and the installation surface (hereinafter referred to as “installation surface angle” Let us form γ4. In this case, the height (hereinafter referred to as “estimated height”) H1b from the installation surface of the raised portion P4 is represented by a first difference distance E2b × sin (γ4). Further, the installation surface angle is previously set for each pixel in the input distance image of the rear stereo camera 6B based on the installation position and the installation angle of the rear stereo camera 6B as in the case of the installation surface distance.

  In addition, the value of the pixel in the input distance image captured by the stereo camera 6B represents the distance between the feature and the stereo camera 6B. Therefore, the first difference distance, which is the difference between the value of each pixel in the input distance image and the installation surface distance corresponding to each pixel, is multiplied by the sine of the installation surface angle corresponding to each pixel. Estimated height can be determined. In this embodiment, the estimated height H1b of the raised portion P4 can be obtained by multiplying the first difference distance E2b by sin (γ4).

  By the way, as described above, the values of the pixels in the input distance image captured by the rear stereo camera 6B represent the distance between the feature and the rear stereo camera 6B. Therefore, when an output distance image is generated by subjecting the input distance image to a process of changing color or luminance according to the distance from the rear stereo camera 6B, using the pixel value in the input distance image as it is causes visibility of the output distance image May damage the Specifically, the output distance image represents the ground (installation surface) with a color or luminance that gradually changes as it goes away from the shovel 60, which may make it difficult to understand the presence of the object P3 and the raised portion P4. .

  Therefore, the pixel extraction unit 12 of the image generation apparatus 100 extracts, for example, pixels satisfying a predetermined condition from the input distance image of the rear stereo camera 6B, and corrects the input distance image based on the extraction result. Specifically, the pixel extracting unit 12 extracts, from each pixel in the input distance image, a pixel having a pixel value smaller than the installation surface distance corresponding to each pixel. Then, the pixel extraction unit 12 replaces the pixel value of the pixel having the pixel value equal to the installation surface distance with a predetermined value (for example, zero as the minimum value). On the other hand, the pixel extraction unit 12 maintains the pixel value of the pixel having the pixel value smaller than the installation surface distance as it is. In this case, the pixel value equal to the installation surface distance may be a value having a predetermined width.

  In addition, the pixel extraction unit 12 may extract, from each pixel in the input distance image, a pixel in which the first difference distance is a predetermined distance or more. In this case, the pixel extracting unit 12 replaces the pixel value of the pixel whose first difference distance is less than the predetermined distance with a predetermined value (for example, zero as the minimum value), and the pixel whose first difference distance is the predetermined distance or more Maintain the pixel value of. For example, as shown in FIG. 20, since the first difference distance E2b is less than the predetermined distance E2b1, the pixel extraction unit 12 sets the pixel value of the pixel corresponding to the first difference distance E2b to zero. The predetermined distance E2b1 is a distance corresponding to the predetermined height H1. Alternatively, as illustrated in FIG. 20, the pixel extracting unit 12 determines that the installation surface height H1b is less than the predetermined height H1 and replaces the pixel value of the pixel corresponding to the installation surface height H1b with zero. Good. Alternatively, in the pixel extracting unit 12, a ratio of the first difference distance E2b to the installation surface distance E2 (hereinafter, a ratio when the installation surface distance E2 is larger than the distance E2a is referred to as a "first difference ratio") is a predetermined ratio. It may be determined that it is less and the pixel value of the corresponding pixel may be replaced with zero. The predetermined distance, the predetermined height, the predetermined ratio, and the like are set in advance for each pixel, each pixel row, or each pixel column in the input distance image of the rear stereo camera 6B based on the installation position and the installation angle of the rear stereo camera 6B. Be done. Further, the predetermined height and the predetermined ratio may be common to all the pixels in the input distance image.

  FIG. 21 is a view showing an example of an output distance image of the rear stereo camera 6B. FIG. 21A shows an output distance image when the pixel values of all the pixels in the input distance image are used as they are without performing extraction by the pixel extraction means 12. FIG. 21B shows an output distance image when the pixel extracting unit 12 extracts a pixel having a pixel value smaller than the installation surface distance corresponding to each pixel among the pixels in the input distance image. Further, in FIG. 21C, the pixel extraction unit 12 outputs, among the pixels in the input distance image, pixels whose first difference distance is a predetermined distance or more, or pixels whose estimated height is a predetermined height or more. The output distance image at the time of being extracted is shown.

  As shown in FIG. 21A, when extraction of pixels is not performed, the output distance image represents the installation surface in a manner in which the luminance gradually decreases as the distance from the shovel 60 increases, and the object P3 and the raised portion P4 are displayed. Make the existence unclear.

  On the other hand, as shown to FIG. 21 (B), when extracting the pixel which has a pixel value smaller than installation surface distance among each pixel in an input distance image, an output distance image is a terrestrial feature existing on an installation surface Improve visibility. Specifically, in the output distance image of FIG. 21 (B), the pixel values of all the pixels having pixel values equal to the installation surface distance are replaced with zero, so that regardless of the distance from the shovel 60 Make the color and brightness uniform. As a result, in the output distance image of FIG. 21B, the distance image of the object P3 and the raised portion P4 can be highlighted from the distance image of the installation surface, and the visibility can be improved.

  Furthermore, as shown in FIG. 21C, in the case of extracting pixels in which the first difference distance is a predetermined distance or more among the pixels in the input distance image, the output distance image has an installation surface height of a predetermined height. Improve the visibility of the above features. Specifically, in the output distance image of FIG. 21C, the pixel values of all the pixels whose first difference distance is less than the predetermined distance are set to zero, and therefore, regardless of the distance from the shovel 60, the installation surface Make the color and brightness of features (including the installation surface) whose height is less than a predetermined height uniform. As a result, in the output distance image of FIG. 21C, the distance image of the raised portion P4 whose installation surface height H1b is less than the predetermined height H1 is made into a single color and luminance together with the distance image of the installation surface. Then, in the output distance image of FIG. 21C, only the distance image of the object P3 whose installation surface height is the predetermined height H1 or more can be distinguished from other distance images, and the visibility can be improved.

  In the output distance image of FIG. 21C, the pixel value of the pixel corresponding to the root portion of the object P3 (the portion whose installation surface height is less than the predetermined height H1) is zero and the distance image of that portion is installed. Make a single color and intensity with the distance image of the surface. However, even if the pixel corresponding to the portion determined to be a part of the object P3 has the first difference distance less than the predetermined distance or the installation surface height is less than the predetermined height H1, The pixel values in the input distance image may be maintained as they are. Note that whether or not a specific pixel belongs to a part of the distance image of the object P3 is, for example, a first difference distance or an installation surface height of surrounding pixels and a first difference distance or an installation surface height of the specific pixel. It can be judged based on

  FIG. 22 is a view showing an example of an output image of the rear camera 2B. FIG. 22A shows an output image when the output distance image of FIG. 21A is superimposed. FIG. 22 (B) shows an output image when the output distance image of FIG. 21 (B) is superimposed. 22C shows an output image when the output distance image of FIG. 21C is superimposed.

  As shown in FIG. 22A, the output image obtained by superimposing the output distance image of FIG. 21A represents the installation surface in such a manner that the luminance gradually decreases as the distance from the shovel 60 increases. We can not highlight the existence of P4.

  On the other hand, as shown in FIG. 22 (B), the output image on which the output distance image having the color and the luminance of the installation surface made uniform regardless of the distance from the shovel 60 is the image of the object P3 and the raised portion P4. Make it stand out from the image on the installation surface and improve its visibility. In the output image of FIG. 22B, the value of the distance between each of the object P3 and the raised portion P4 and the rear stereo camera 6B is superimposed and displayed. Thereby, the presence of the object P3 and the raised portion P4 can be further highlighted. As the value of the distance between the object P3 and the rear stereo camera 6B, for example, a value corrected by the distance correction means 13 is adopted. In addition, the value of the distance between the object P3 and the rear stereo camera 6B may be superimposed and displayed only when the value is equal to or less than a predetermined value. This is to further highlight the presence of a feature near the shovel 60. Further, the brightness or color of the value of the distance (pixel value) is determined according to the size of the distance (pixel value). For example, the color of the value of the distance (pixel value) changes to purple, blue, green, yellow, red as the distance (pixel value) decreases. Further, the value of the distance may be the minimum value among the pixel values of the pixels included in the distance image area of the object P3 in the input distance image, which are extracted by the pixel extraction means 12. The same applies to the value of the distance between the protuberance P4 and the rear stereo camera 6B.

  Furthermore, as shown in FIG. 22C, regardless of the distance from the shovel 60, the output distance in which the color and brightness of the feature (including the installation surface) whose installation surface height is less than the predetermined height is uniformed The output image on which the image is superimposed improves the visibility of the object P3. Specifically, the output image highlights only the upper part of the image of the object P3 whose installation surface height is a predetermined height or more from the other images, thereby improving the visibility. In the output image of FIG. 22C, only the value of the distance between the object P3 whose installation surface height is a predetermined height or more and the rear stereo camera 6B is superimposed and displayed. Thereby, only the presence of the object P3 whose installation surface height is equal to or more than the predetermined height can be further highlighted.

  Further, in the above, the image generation apparatus 100 superimposes the output distance image of the rear stereo camera 6B on the output image of the rear camera 2B and displays the image on the display unit 5. However, the image generation apparatus 100 may display only the output distance image on the display unit 5 or may display the output distance image and the output image on the display unit 5 side by side.

  Also, instead of displaying the value of the distance between each of the object P3 and the protuberance P4 and the rear stereo camera 6B, the image generating apparatus 100 is shown in FIG. 18 instead of or in addition to the display of the value of the distance. Such a frame may be displayed.

  In addition, the image generation apparatus 100 displays an indicator (not shown) such as a rod-shaped indicator with a distance scale indicating the relationship between the distance (pixel value) and the color or luminance at the end of the output distance image or the output image. May be

  Next, the flow of the first pixel extraction process will be described with reference to FIG. FIG. 23 is a flowchart showing the flow of the first pixel extraction process. The pixel extraction unit 12 of the image generation apparatus 100 executes this first pixel extraction process, for example, every time an input distance image is acquired.

  First, the pixel extraction unit 12 reads out and acquires the installation surface distance of one pixel (hereinafter, referred to as “target pixel”) in the input distance image from the ROM or the like (step S11). The pixel extraction unit 12 may read and acquire the installation surface angle from the ROM or the like.

  Thereafter, the pixel extraction unit 12 calculates the first difference distance by subtracting the pixel value of the target pixel from the installation surface distance (step S12). The pixel extracting unit 12 may calculate the first difference ratio based on the calculated first difference distance and the installation surface distance, and the estimated height based on the calculated first difference distance and the installation surface angle. May be calculated.

  Thereafter, the pixel extracting unit 12 compares the calculated first difference distance with the predetermined distance stored in the ROM or the like (step S13). The pixel extracting unit 12 may compare the calculated first difference ratio with the predetermined ratio stored in the ROM or the like, or even if the calculated estimated height is compared with the predetermined height stored in the ROM or the like. Good.

  If the first difference distance is equal to or greater than the predetermined distance (YES in step S13), the pixel extraction unit 12 extracts the target pixel as a pixel maintaining the current pixel value as it is. The same applies to the case where the first difference ratio is equal to or more than the predetermined ratio, or the case where the estimated height is equal to or more than the predetermined height.

  On the other hand, when the first difference distance is less than the predetermined distance (NO in step S13), the pixel extracting unit 12 does not extract the target pixel as a pixel maintaining the current pixel value as it is. Specifically, the pixel extracting unit 12 replaces the pixel value of the target pixel with a predetermined value stored in the ROM or the like (step S14). The same applies to the case where the first difference ratio is less than the predetermined ratio or the case where the estimated height is less than the predetermined height.

  In the above-described embodiment, the process of replacing the pixel value of the target pixel not extracted with a predetermined value is performed when it is determined whether or not the extraction is necessary for each target pixel. It may be performed collectively after determining the necessity of extraction.

  Thereafter, the pixel extraction unit 12 determines whether or not the determination of extraction necessity has been performed for all the pixels in the acquired input distance image (step S15).

  If it is determined that the extraction necessity determination has not been performed for all the pixels (NO in step S15), the pixel extraction unit 12 executes the processes in step S11 and subsequent steps for another target pixel.

  If it is determined that the extraction necessity determination has been performed for all the pixels (YES in step S15), the pixel extraction unit 12 completes the current first pixel extraction process.

  With the above-described configuration, the image generating apparatus 100 can improve the visibility of the distance image on the feature that may be in contact with the shovel 60 by omitting the detailed display of the distance image on the installation surface.

  Further, the image generating apparatus 100 omits the detailed display of the features whose installation surface height is less than the predetermined height, so that the visibility of the features (for example, workers) whose installation surface height is the predetermined height or more Can be improved.

  As a result, the image generating apparatus 100 can more reliably prevent the shovel 60 from colliding with a feature.

  Next, referring to FIGS. 24 to 28, another example of processing for the image generation device 100 to extract pixels satisfying a predetermined condition from the input distance image and correct the input distance image (hereinafter referred to as “second pixel The extraction process will be described. In the present embodiment, it is assumed that the values of the pixels in the input distance image captured by the rear stereo camera 6B are equal to or larger than the installation surface distance.

  FIG. 24 is a partial right side view of the shovel 60 on which the image generating apparatus 100 is mounted, and corresponds to FIG. In the embodiment shown in FIG. 24, the shovel 60 includes one camera 2 (rear camera 2B) and one stereo camera 6 (rear stereo camera 6B). An area ZB indicated by a dotted line in FIG. 24 indicates an imaging range of the rear stereo camera 6B.

  Moreover, FIG. 24 shows that the shallow depression part PK1 and deep depression part (cliff) PK2 of the ground exist behind the shovel 60. As shown in FIG. Further, FIG. 24 shows that the distance E3a to the bottom of the shallow recess portion PK1 is detected by the rear stereo camera 6B, and the distance E4a to the bottom of the deep recess portion PK2 is detected.

  Further, in FIG. 24, if the shallow depression portion PK1 and the deep depression portion PK2 do not exist, the rear stereo camera 6B detects the installation surface distances E3 and E4 instead of the distances E3a and E4a. Indicates that.

  Further, in FIG. 24, a difference (E3a-E3) between the distance E3a and the installation surface distance E3 is a distance (hereinafter, the difference when the installation surface distance E3 is smaller than the distance E3a is referred to as a "second difference distance" E3b. This indicates that the difference (E4a-E4) between the distance E4a and the installation surface distance E4 is the second difference distance E4b.

  FIG. 25 is an enlarged view of the depression part PK1 in FIG. As shown in FIGS. 24 and 25, the line connecting the point of the bottom of the shallow depression portion PK1 viewed from the rear stereo camera 6B and the stereo camera 6B forms an installation surface angle γ5 with the installation surface. In this case, the depth (hereinafter referred to as “estimated depth”) DP1 b from the installation surface of the shallow depression portion PK1 is represented by a second difference distance E3 b × sin (γ5).

  In addition, the value of the pixel in the input distance image captured by the stereo camera 6B represents the distance between the feature and the stereo camera 6B. Therefore, by multiplying the second difference distance, which is the difference between the value of each pixel in the input distance image and the installation surface distance corresponding to each pixel, by the sine of the installation surface angle corresponding to each pixel, Estimated depth can be determined. In this embodiment, the estimated depth DP1b of the shallow recess portion PK1 can be obtained by multiplying the second difference distance E3b by sin (γ5).

  By the way, as described above, the values of the pixels in the input distance image captured by the rear stereo camera 6B represent the distance between the feature and the rear stereo camera 6B. Therefore, when an output distance image is generated by subjecting the input distance image to a process of changing color or luminance according to the distance from the rear stereo camera 6B, using the pixel value in the input distance image as it is causes visibility of the output distance image May damage the Specifically, the output distance image represents the ground (installation surface) with a color or luminance that gradually changes as it goes away from the shovel 60, making it difficult to understand the presence of the depression PK1 and the depression PK2. There is a fear.

  Therefore, the pixel extraction unit 12 of the image generation apparatus 100 extracts, for example, pixels satisfying a predetermined condition from the input distance image of the rear stereo camera 6B, and corrects the input distance image based on the extraction result. Specifically, the pixel extraction unit 12 extracts, from each pixel in the input distance image, a pixel having a pixel value larger than the installation surface distance corresponding to each pixel. Then, the pixel extraction unit 12 replaces the pixel value of the pixel having the pixel value equal to the installation surface distance with a predetermined value (for example, zero as the minimum value). On the other hand, the pixel extraction unit 12 maintains the pixel value of the pixel having the pixel value larger than the installation surface distance as it is. In this case, the pixel value equal to the installation surface distance may be a value having a predetermined width.

  In addition, the pixel extraction unit 12 may extract, from each pixel in the input distance image, a pixel in which the second difference distance is a predetermined distance or more. In this case, the pixel extraction unit 12 replaces the pixel value of the pixel whose second difference distance is less than the predetermined distance with a predetermined value (for example, zero as the minimum value), and the pixel whose second difference distance is the predetermined distance or more Maintain the pixel value of. For example, as shown in FIG. 25, since the second difference distance E3b is less than the predetermined distance E3b1, the pixel extracting unit 12 sets the pixel value of the pixel corresponding to the second difference distance E3b to zero. The predetermined distance E3b1 is a distance corresponding to the predetermined depth DP1. Alternatively, as shown in FIG. 25, the pixel extracting unit 12 determines that the installation surface depth DP1b is less than the predetermined depth DP1, and replaces the pixel value of the pixel corresponding to the installation surface depth DP1b with zero. Good. Alternatively, in the pixel extracting unit 12, the ratio of the second difference distance E3b to the distance E3a (hereinafter, the ratio when the installation surface distance E3 is smaller than the distance E3a is referred to as "second difference ratio") is less than a predetermined ratio. It may be determined that there is, and the pixel value of the corresponding pixel may be replaced with zero. The predetermined distance, predetermined depth, predetermined ratio, etc. are set in advance for each pixel, each pixel row, or each pixel column in the input distance image of the rear stereo camera 6B based on the installation position and the installation angle of the rear stereo camera 6B. Be done. Also, the predetermined depth and the predetermined ratio may be common to all the pixels in the input distance image.

  FIG. 26 is a view showing an example of an output distance image of the rear stereo camera 6B. FIG. 26A shows an output distance image when the pixel values of all the pixels in the input distance image are used as they are without performing extraction by the pixel extraction means 12. FIG. 26B shows an output distance image when the pixel extracting unit 12 extracts a pixel having a pixel value larger than the installation surface distance corresponding to each pixel among the pixels in the input distance image. Further, in FIG. 26C, the pixel extraction unit 12 outputs, among the pixels in the input distance image, pixels whose second difference distance is a predetermined distance or more, or pixels whose estimated depth is a predetermined depth or more. The output distance image at the time of being extracted is shown.

  As shown in FIG. 26A, when extraction of pixels is not performed, the output distance image shows the installation surface in a mode in which the luminance decreases gradually as it gets away from the shovel 60, and the depression PK1 and the depression Make the existence of department PK2 unclear.

  On the other hand, as shown in FIG. 26B, when extracting pixels having a pixel value larger than the installation surface distance among the pixels in the input distance image, the output distance image is a depression located in the lower part of the installation surface. Improve visibility. Specifically, in the output distance image of FIG. 26B, the pixel values of all the pixels having pixel values equal to the installation surface distance are replaced with zero, so that regardless of the distance from the shovel 60, Make the color and brightness uniform. As a result, in the output distance image of FIG. 26B, the distance images of the shallow depression portion PK1 and the deep depression portion PK2 can be highlighted from the distance image of the installation surface, and the visibility thereof can be improved.

  Further, as shown in FIG. 26C, in the case of extracting the pixels in which the second difference distance is a predetermined distance or more among the pixels in the input distance image, the output distance image has an installation surface depth of a predetermined depth. Improve the visibility of the above features. Specifically, in the output distance image of FIG. 26C, the pixel values of all the pixels whose second difference distance is less than the predetermined distance are set to zero, and therefore, regardless of the distance from the shovel 60, the installation surface Make the color and brightness of features (including the installation surface) whose depth is less than a predetermined depth uniform. As a result, in the output distance image of FIG. 26C, the distance image of the shallow depression portion PK1 whose installation surface depth DP1b is less than the predetermined depth DP1 is made a single color and luminance together with the distance image of the installation surface. Then, in the output distance image of FIG. 26C, only the distance image of the deep depression portion PK2 whose installation surface depth is a predetermined depth DP1 or more is made to stand out from the other distance images, and the visibility thereof is improved. it can.

  In the output distance image of FIG. 26C, the pixel value of the pixel corresponding to the edge portion of the deep depression portion PK2 (a portion where the installation surface depth is less than the predetermined depth DP1) is zero and the distance image of that portion Into a single color and intensity with the distance image of the installation surface. However, even if the second difference distance is less than the predetermined distance, or the pixel corresponding to the portion determined to be a part of the deep depression portion PK2, the installation surface depth is less than the predetermined depth DP1. Also, the pixel values in the input distance image may be maintained as they are. Note that whether or not a specific pixel belongs to a part of the distance image of the deep depression portion PK2 is, for example, the second difference distance or the installation surface depth of surrounding pixels and the second difference distance or the installation surface of the specific pixel It can be judged based on the depth.

  FIG. 27 is a view showing an example of an output image of the rear camera 2B. FIG. 27A shows an output image when the output distance image of FIG. 26A is superimposed. FIG. 27 (B) shows an output image when the output distance image of FIG. 26 (B) is superimposed. 27C shows an output image when the output distance image of FIG. 26C is superimposed.

  As shown in FIG. 27A, an output image obtained by superimposing the output distance image of FIG. 26A represents the installation surface in a mode in which the luminance gradually decreases as the distance from the shovel 60 increases. It is not possible to highlight the existence of the deep depression portion PK2.

  On the other hand, as shown in FIG. 27 (B), the output image on which the output distance image in which the color and the luminance of the installation surface are uniformed is superimposed regardless of the distance from the shovel 60 is the depression PK1 and the depression PK2. Of the image from the image of the installation surface to improve its visibility. Further, the output image in FIG. 27B displays the value of the distance between each of the shallow depression portion PK1 and the deep depression portion PK2 and the rear stereo camera 6B in a superimposed manner. Thereby, the presence of the shallow depression portion PK1 and the deep depression portion PK2 can be further highlighted. Note that, for example, a value corrected by the distance correction unit 13 is adopted as the value of the distance between the shallow depression portion PK1 and the rear stereo camera 6B. In addition, the value of the distance between the shallow depression portion PK1 and the rear stereo camera 6B may be superimposed and displayed only when it is less than a predetermined value. This is to further emphasize the presence of the depression near the shovel 60. Further, the brightness or color of the value of the distance (pixel value) is determined according to the size of the distance (pixel value). For example, the color of the value of the distance (pixel value) changes to purple, blue, green, yellow, red as the distance (pixel value) decreases. Further, the value of the distance may be the minimum value among the pixel values of the pixels included in the distance image area of the shallow depression portion PK1 in the input distance image, which is extracted by the pixel extraction unit 12. The same applies to the value of the distance between the deep depression portion PK2 and the rear stereo camera 6B.

  Furthermore, as shown in FIG. 27C, regardless of the distance from the shovel 60, the output distance in which the color and brightness of the feature (including the installation surface) whose installation surface depth is less than the predetermined depth is uniformed The output image on which the image is superimposed improves the visibility of the deep recess portion PK2. Specifically, as the output image, only the image of the deep depression portion PK2 whose installation surface depth is equal to or more than a predetermined depth is made to stand out from the other images, and the visibility thereof is improved. Further, in the output image of FIG. 27C, only the value of the distance between the deep depression portion PK2 whose installation surface depth is a predetermined depth or more and the rear stereo camera 6B is superimposed and displayed. As a result, it is possible to further emphasize only the presence of the deep depression portion PK2 whose installation surface depth is equal to or more than the predetermined depth.

  Further, in the above, the image generation apparatus 100 superimposes the output distance image of the rear stereo camera 6B on the output image of the rear camera 2B and displays the image on the display unit 5. However, the image generation apparatus 100 may display only the output distance image on the display unit 5 or may display the output distance image and the output image on the display unit 5 side by side.

  Further, instead of displaying the value of the distance between each of the shallow depression portion PK1 and the deep depression portion PK2 and the rear stereo camera 6B, the image generation apparatus 100 is not limited to or in addition to the display of the value of the distance. A frame as shown in 18 may be displayed.

  Next, the flow of the second pixel extraction process will be described with reference to FIG. FIG. 28 is a flowchart showing the flow of the second pixel extraction process. The pixel extraction unit 12 of the image generation apparatus 100 executes this second pixel extraction process, for example, every time an input distance image is acquired.

  First, the pixel extracting unit 12 reads and acquires the installation surface distance of one pixel (hereinafter, referred to as “target pixel”) in the input distance image from the ROM or the like (step S21). The pixel extraction unit 12 may read and acquire the installation surface angle from the ROM or the like.

  Thereafter, the pixel extraction unit 12 calculates the second difference distance by subtracting the installation surface distance from the pixel value of the target pixel (step S22). The pixel extraction unit 12 may calculate the second difference ratio based on the calculated second difference distance and the pixel value of the target pixel, and based on the calculated second difference distance and the installation plane angle. The estimated depth may be calculated.

  Thereafter, the pixel extracting unit 12 compares the calculated second difference distance with the predetermined distance stored in the ROM or the like (step S23). Note that the pixel extraction unit 12 may compare the calculated second difference ratio with the predetermined ratio stored in the ROM or the like, or even if the calculated estimated depth is compared with the predetermined depth stored in the ROM or the like. Good.

  If the second difference distance is equal to or greater than the predetermined distance (YES in step S23), the pixel extraction unit 12 extracts the target pixel as a pixel maintaining the current pixel value as it is. The same applies to the case where the second difference ratio is equal to or more than the predetermined ratio, or the case where the estimated depth is equal to or more than the predetermined depth.

  On the other hand, when the second difference distance is less than the predetermined distance (NO in step S23), the pixel extracting unit 12 does not extract the target pixel as a pixel maintaining the current pixel value as it is. Specifically, the pixel extracting unit 12 replaces the pixel value of the target pixel with a predetermined value stored in the ROM or the like (step S24). The same applies to the case where the second difference ratio is less than the predetermined ratio or the case where the estimated depth is less than the predetermined depth.

  The process of replacing the pixel value of the target pixel that has not been extracted with a predetermined value may be performed collectively after determining the necessity of extraction for a plurality of or all of the target pixels.

  Thereafter, the pixel extraction unit 12 determines whether or not the determination of extraction necessity has been performed for all the pixels in the acquired input distance image (step S25).

  If it is determined that the extraction necessity determination has not been performed for all the pixels (NO in step S25), the pixel extraction unit 12 executes the processes in step S21 and subsequent steps for another target pixel.

  If it is determined that the extraction necessity determination has been performed for all the pixels (YES in step S25), the pixel extraction unit 12 completes the current second pixel extraction process.

  With the above configuration, the image generating apparatus 100 omits the detailed display of the distance image on the installation surface, so that the distance image of a depression (for example, a road shoulder, a cliff, etc.) that may cause the shovel 60 to slide down. Visibility can be improved.

  In addition, the image generating apparatus 100 can improve the visibility of the depression having the installation surface depth equal to or more than the predetermined depth by omitting the detailed display of the depression having the installation surface depth less than the predetermined depth. .

  As a result, the image generation device 100 can more reliably prevent the shovel 60 from sliding into the depression.

  In addition, although the first pixel extraction process and the second pixel extraction process are described separately in the above, two pixel extraction processes may be combined.

  Next, with reference to FIG. 29, a process (hereinafter, referred to as “drive support process”) in which the drive support means 15 executes the drive support function will be described. FIG. 29 is a flowchart showing the flow of the driving support processing, and the driving support means 15 repeatedly executes the driving support processing at a predetermined cycle while the shovel 60 is traveling.

  First, the driving support means 15 detects the moving direction of the shovel 60 based on the operation direction of the traveling control lever (not shown) of the shovel 60 (step S31).

  Thereafter, the driving support means 15 determines whether or not a predetermined feature exists in the moving direction of the shovel 60 (step S32). In the present embodiment, based on the input distance image acquired by the rear stereo camera 6B, the driving support means 15 determines whether or not there is a depression which may cause the shovel 60 to slide down behind the shovel 60 moving backward. judge.

  If it is determined that there is a depressed portion which may cause the shovel 60 to slide down (YES in step S32), the driving support means 15 executes the driving support function (step S33). In the present embodiment, the driving support means 15 makes the alarm sound and displays on the display unit 5 a warning message notifying that there is a depression. Further, the driving support means 15 may reduce the retreating speed of the shovel 60 as the distance to the depression becomes shorter, and may stop the shovel 60 when the distance to the depression becomes smaller than a predetermined value. .

  On the other hand, when it is determined that there is no depression that may cause the shovel 60 to slide down (NO in step S32), the driving support unit 15 ends the current driving support processing without executing the driving support function.

  With the above configuration, the image generating apparatus 100 automatically determines the presence or absence of a predetermined feature based on the output of the stereo camera 6, and executes the driving support function as needed, to operate the shovel 60. Safety can be further improved.

  Next, a process of correcting the pixel value (distance) of the input distance image by the image generating apparatus 100 (hereinafter, referred to as “distance correction process”) will be described with reference to FIGS.

  FIG. 30 is a view showing a shovel 60 on which the image generating apparatus 100 is mounted, FIG. 30 (A) is a top view thereof, and FIG. 30 (B) is a partial right side view thereof. In the embodiment shown in FIG. 30, the shovel 60 includes one camera 2 (rear camera 2B) and one stereo camera 6 (rear stereo camera 6B). An area CB indicated by an alternate long and short dash line in FIG. 30 indicates an imaging range of the rear camera 2B. Further, a region ZB indicated by a dotted line in FIG. 30 indicates the imaging range of the rear stereo camera 6B.

  As shown in FIG. 30, behind the shovel 60, there are two objects P1 and P2 extending vertically upward from the ground. The two objects P1 and P2 are rod-like objects with different heights, and exist in different directions as viewed from the rear stereo camera 6B. Specifically, as shown in FIG. 30 (B), the height of the object P1 is lower than the height of the object P2. However, the shortest distances between each of the two objects P1 and P2 and the rear stereo camera 6B are all equal at the distance D. This is because the lower object P1 is closer to the shovel 60 than the higher object P2.

  Further, as shown in FIG. 30B, a line segment connecting the closest point on the object P1 viewed from the stereo camera 6B to the stereo camera 6B forms an angle γ1 with the horizontal plane. Similarly, a line segment connecting the closest point on the object P2 viewed from the stereo camera 6B to the stereo camera 6B forms an angle γ2 with the horizontal plane. In this case, the horizontal distance D1 between the object P1 and the plane including the rear surface of the shovel 60 is represented by D × cos (γ1), and the horizontal distance D2 between the object P2 and the plane including the rear surface of the shovel 60 is , D × cos (γ2). The horizontal distance D2 is larger than the horizontal distance D1.

  The values of the pixels in the input distance image captured by the stereo camera 6B represent the distance between the feature and the stereo camera 6B. Therefore, when the pixel value in the input distance image is used as it is, in the processing of changing the display color or the luminance according to the distance from the stereo camera 6B, the top of the object P1 and the top of the object P2 are displayed with the same color or the same luminance. It will be done. Even if the display is such that the object P1 is closer to the shovel 60 than the object P2 as described above, the operator misunderstands that the object P1 and the object P2 exist at the same distance from the rear surface of the shovel 60 There is a risk of

  Such a situation arises from the fact that the rear stereo camera 6 </ b> B captures an image obliquely downward while being attached to the top of the upper swing body 63. That is, while the distance recognized by the operator is the horizontal distance from the shovel 60, the distance acquired by the stereo camera 6 is due to the linear distance from the stereo camera 6. The mounting position of the stereo camera 6B is an upper portion of the upper swing body 63 so that the sensor can be protected from contact with surrounding features.

  Therefore, the distance correction means 13 of the image generation device 100 is a plane including, for example, the pixel values representing the distance between each of the objects P1 and P2 and the stereo camera 6B, each of the objects P1 and P2 and the rear surface of the shovel 60. To the pixel value representing the horizontal distance between

  Specifically, the distance correction means 13 connects the stereo camera 6B and an arbitrary point on the object P1 from the coordinates of the pixels in the input distance image corresponding to the arbitrary point on the object P1 viewed from the stereo camera 6B. The angle γ that the line segment forms with the horizontal plane is derived. Since the mounting position and mounting angle of the stereo camera 6B are known, the distance correction means 13 can derive the angle γ from the coordinates of the pixel.

  After that, the distance correction means 13 sets the pixel value of that pixel, that is, on the stereo camera 6B and the object P1, to the cosine cos (γ) of the angle γ derived from the coordinates of the pixel corresponding to the arbitrary point Multiply the distance between any point of In this manner, the distance correction means 13 calculates the horizontal distance between the stereo camera 6B and an arbitrary point on the object P1, and calculates the pixel value of the pixel corresponding to the arbitrary point by the calculated horizontal distance. replace.

  Similarly, the distance correction means 13 derives an angle γ which a line segment connecting the stereo camera 6B and an arbitrary point on the object P2 forms with the horizontal plane. Then, the cosine cos (γ) of the angle γ is multiplied by the pixel value, that is, the distance between the stereo camera 6B and an arbitrary point on the object P2. Then, the horizontal distance between the stereo camera 6B and an arbitrary point on the object P2 is calculated, and the pixel value of the pixel corresponding to the arbitrary point is replaced with the calculated horizontal distance.

  Here, the effect of the correction by the distance correction means 13 will be described with reference to FIGS. 31 and 32. FIG.

  FIG. 31 shows an output distance image based on an input distance image not subjected to distance correction processing (hereinafter, referred to as “pre-correction output distance image”) and an output distance based on an input distance image subjected to distance correction processing. FIG. 31 (A) shows an output distance image before correction, and FIG. 31 (B) shows a corrected output distance image, which are differences from the image (hereinafter referred to as “corrected output distance image”). Indicates In the present embodiment, pixel extraction processing has already been performed on the input distance image.

  Further, FIG. 32 is an output image after composition obtained by combining the output image before correction with the output image generated based on the input image of the rear camera 2B (hereinafter, referred to as “preset output image after correction”). FIG. 32A is a contrast diagram showing a difference from a post-composition output image (hereinafter referred to as “corrected post-composition output image”) obtained by combining the corrected output distance image with the output image, and FIG. The post-output image is shown, and FIG. 32 (B) shows the corrected post-composed output image.

  As shown in FIGS. 31A and 32A, in the output distance image before correction and the output image after combination before correction, the object P1 and the object P2 are expressed in the same color arrangement. Specifically, both the object P1 and the object P2 are displayed white (thin) near the top and black (dark) near the bottom. Therefore, it is difficult for the operator to determine which object is closer to the shovel 60.

  On the other hand, as shown in FIGS. 31B and 32B, in the corrected output distance image and the corrected combined output image, the object P1 and the object P2 are expressed in different colors. Specifically, the object P1 is generally displayed light and the object P2 is generally displayed dark. Therefore, the operator can easily recognize that the object P1 is closer to the shovel 60 than the object P2.

  FIG. 33 is another example of a contrast diagram showing the difference between the pre-correction post-composition output image and the corrected post-composition output image, and shows an example where a frame is displayed in a superimposed manner.

  As shown in FIG. 33A, in the combined output image before correction, when the minimum value of the pixel values of the pixels included in the feature region is adopted as a representative value of the feature region, the object P1 and the object P2 Are displayed as being at a distance of 1 meter from the shovel 60.

  On the other hand, as shown in FIG. 33B, in the corrected combined output image, horizontal distances D1 and D2 between the objects P1 and P2 and a plane including the rear surface of the shovel 60 are displayed. That is, it is displayed that the object P1 is at a distance of 1 meter from the rear surface of the shovel 60, and the object P2 is at a distance of 1.5 meters from the rear surface of the shovel 60. Therefore, the operator can easily recognize that the object P1 is closer to the shovel 60 than the object P2.

  With the above configuration, the image generation device 100 corrects the pixel value such that the pixel value in the input distance image captured by the stereo camera 6 represents the horizontal distance from the shovel 60. As a result, the image generating apparatus 100 can convey the status of surrounding features to the operator more easily.

  Further, in the above-described embodiment, the distance correction unit 13 individually corrects the pixel values of the pixels extracted by the pixel extraction unit 12. However, the present invention is not limited to this. For example, the distance correction means 13 does not calculate the horizontal distance using cosine as described above, but collects predetermined corrections for each pixel row, each pixel column, or each region divided in advance in the input distance image plane. May be performed. This is to obtain the same effect as in the case of calculating the horizontal distance using cosines by simple correction while suppressing the calculation load. Specifically, for example, the distance correction means 13 decreases the distance so that the increase correction of the distance becomes larger as the pixel row is above the input distance image plane, or as the pixel row is below the input distance image plane. The correction by a predetermined correction amount may be performed collectively in order to increase the correction.

  Further, in the above-described embodiment, the distance correction means 13 executes the correction in the case where the stereo camera 6B is attached to the upper portion of the rear surface of the upper swing body 63 and images the obliquely lower side. However, the present invention is not limited to this. For example, the distance correction unit 13 may perform correction when the stereo camera captures an image in the horizontal direction, or may perform correction when the stereo camera captures an image obliquely upward.

  Further, in the above embodiment, the distance correction means 13 sets the pixel value of the pixel included in the feature region to the plane including the feature and the rear surface of the shovel 60 based on the plane including the rear surface of the shovel 60 To the pixel value representing the horizontal distance between However, the present invention is not limited to this. For example, the distance correction means 13 is a pixel value representing the horizontal distance between the feature and the turning axis PV of the shovel 60 with reference to the turning axis PV of the shovel 60 and the pixel value of the pixel included in the feature area. It may be corrected to

  Further, the distance correction means 13 may correct the pixel values of the pixels included in the feature area to pixel values representing the shortest horizontal distance between the feature and the shovel 60. In this case, the distance correction means 13 selects the reference from the plane including the right side of the shovel 60, the plane including the left side of the shovel 60, the plane including the rear side of the shovel 60, etc. according to the actual position of the feature. .

  The shortest horizontal distance may be the shortest horizontal distance in the movement direction of the shovel 60, that is, in the direction parallel to the front-rear direction of the lower traveling body 61. In this case, even if the feature is in the vicinity of the shovel 60, the pixel value of the feature that is not in the moving direction of the shovel 60 is corrected to the maximum value (infinity). As a result, the operator can more easily recognize the feature in the moving direction of the shovel 60.

  Although the preferred embodiments of the present invention have been described above in detail, the present invention is not limited to the above-described embodiments, and various modifications and substitutions may be made to the above-described embodiments without departing from the scope of the present invention. It can be added.

  For example, in the above-described embodiment, the image generation apparatus 100 adopts a cylindrical space model MD as a space model, but may adopt a space model having another columnar shape such as a polygonal column, etc. A space model composed of two sides may be adopted, or a space model having only sides may be adopted.

  In addition, the image generating apparatus 100 is mounted on a self-propelled shovel with movable members such as a bucket, an arm, a boom, a turning mechanism, etc., together with a camera and a stereo camera. An operation support system is configured to support movement and operation of the movable members. However, the image generating apparatus 100 may be a working machine that does not have a turning mechanism such as a forklift, an asphalt finisher, or a working machine that has movable members but does not self-propelled, such as an industrial machine or a fixed crane. You may comprise the operation assistance system which is mounted with a camera and a stereo camera, and assists operation of those working machines.

  Further, in the above-described embodiment, the distance image combining unit 14 generates the output distance image, and then combines the generated output distance image with the output image. However, the present invention is not limited to this. For example, the distance image combining unit 14 combines the input image and the input distance image to generate a post-combination input image, and generates a combination post-processing target image on which information on the distance image is superimposed and displayed. An output image may be generated. In addition, the distance image combining unit 14 may, for example, combine the processing target image and the processing target distance image to generate a combined post-processing target image, and then generate a combined output image.

  Moreover, in the above-mentioned embodiment, the image generation device 100 is provided with the camera 2 and the stereo camera 6 separately. Then, the image generation apparatus 100 generates one output image (viewpoint conversion image) based on the plurality of input images from the camera 2 and one output based on the plurality of input distance images from the stereo camera 6 A distance image (viewpoint conversion distance image) is generated. Then, the image generation apparatus 100 combines the output distance image with the output image. However, the present invention is not limited to this configuration. For example, the image generation device 100 generates an output image (viewpoint conversion image) based on three images output by one imaging mechanism in each of the left side stereo camera 6L, the rear stereo camera 6B, and the right side stereo camera 6R. And one output distance image (viewpoint converted distance image) may be generated based on three input distance images of the three stereo cameras. In this case, the camera 2 may be omitted. Further, in this case, the left side stereo camera 6L, the rear stereo camera 6B, and the right side stereo camera 6R are respectively installed approximately at the centers of the left side face, the rear face and the right side face of the upper swing body 63 so that the imaging range is not biased. It is desirable to be done. Therefore, a plurality of sets of imaging mechanisms in each stereo camera are juxtaposed at predetermined intervals in the vertical direction. This is because, regardless of the size of the juxtaposed spacing, it is easy to install the stereo camera approximately at the center of each surface. In addition, when a plurality of sets of imaging mechanisms are juxtaposed in the lateral direction, it is difficult to place the stereo camera in the approximate center of each surface depending on the juxtaposition distance.

  Further, in the above-described embodiment, the stereo camera 6 derives the distance between the object shown in each pixel of the basic image and the stereo camera 6 based on the parallax between the images outputted by each of the two sets of imaging mechanisms. Then, the stereo camera 6 substitutes the derived distance into the value of each pixel of the basic image to generate an input distance image as distance information of a two-dimensional array, and transmits the generated input distance image to the control unit 1 Output. Then, the control unit 1 generates one output distance image using a plurality of input distance images from the plurality of stereo cameras 6. However, the present invention is not limited to this configuration. For example, the image generating apparatus 100 generates a first output image (first viewpoint conversion image) based on three images output by the first imaging mechanism in each of three stereo cameras, and the three stereo cameras The second output image (second viewpoint conversion image) is generated based on the three images output by the second imaging mechanism in each of the above. In addition, the image generation apparatus 100 determines a distance between an object shown in each pixel of the first viewpoint converted image and a predetermined reference point based on the parallax between the first viewpoint converted image and the second viewpoint converted image. The output distance image may be generated without using the input distance image by deriving.

  DESCRIPTION OF SYMBOLS 1 ... Control part 2 ... Camera 2L ... Left-hand side camera 2 R .. Right-hand direction camera 2B. Stereo camera 6 L: Left-side stereo camera 6 R: Right-side stereo camera 6 B: Rear stereo camera 10: Coordinate matching means 11: Image generation means 12: Pixel extraction means 13 · · · Distance correction means 14 · · · distance image synthesis means 15 · · · driving assistance means 40 · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · Output image correspondence map 60 · · · Shovel 61 · · · lower traveling body 62 · · · turning mechanism 63 · · · upper revolving body 64 · · · cab 100 · · · image generation device

Claims (7)

A lower traveling body that performs traveling operation,
An upper revolving unit rotatably mounted on the lower traveling unit;
A boom attached to the upper swing body and included in an attachment;
An arm attached to the boom and included in the attachment;
A sensor mounted on the upper revolving superstructure and acquiring distance information for each pixel;
A shovel having
A camera is disposed on the upper swing body separately from the sensor that acquires distance information for each pixel.
Excavator. At least three sensors for acquiring distance information for each pixel are provided on the upper swing body,
The shovel according to claim 1. The shovel is decelerated or stopped based on an output of a sensor that acquires distance information for each pixel.
The shovel according to claim 1 or 2. The pixel value of each pixel is compared to determine the presence or absence of a predetermined feature,
The shovel according to any one of claims 1 to 3. Make the distance image of a predetermined feature stand out from the distance images of other features, and judge the presence or absence of the predetermined feature,
The shovel according to any one of claims 1 to 4. The change in the pixel value is used to determine the presence or absence of a feature having a predetermined height or more.
The shovel according to claim 4. The sensor for acquiring distance information for each pixel is a stereo camera.
The shovel according to any one of claims 1 to 6.